Senior Data Engineer

Avance Consulting; London; On-site; GBP 50,000 - 90,000; Full time

Job description

As a Senior Data Engineer, you will play a crucial role in designing, developing, and maintaining scalable data pipelines, ensuring high-quality data is available for decision-making across the business.

You will collaborate closely with data architects, product teams, analysts, and data scientists to implement data solutions that power analytics and reporting.

The ideal candidate will have strong experience in data engineering, cloud platforms like DBT, Azure/AWS, data modeling, and ETL/ELT processes.

You will be part of a dynamic and innovative team, working on cutting-edge technologies to deliver impactful data products for our retail chain.

Your responsibilities:

1. ETL Pipeline Development: Develop, optimize, and maintain ETL pipelines to efficiently extract, transform, and load data from various sources, ensuring high data quality.
2. Monitor and troubleshoot production data pipelines, ensuring their performance and reliability.
3. Mentor junior engineers and lead technical discussions to drive best practices and innovation within the team.
4. Stay up to date with the latest trends and technologies in data engineering and recommend solutions to improve data processing capabilities.
5. Query Optimization & Data Transformation: Write and optimize SQL queries, ensuring data integrity, performance, and scalability, using best practices and techniques.
6. Data Vault Model Implementation: Implement flexible Data Vault model in Snowflake to support large-scale analytics and business intelligence.
7. Cross-Team Collaboration: Collaborate with Data Engineers, Product Managers, and Data Scientists to deliver solutions that support data-driven insights and innovation.
8. Stakeholder Engagement: Engage with business stakeholders to understand requirements and translate them into technical solutions that add value.
9. Data Quality & Governance: Implement and enforce data governance and quality processes, ensuring accurate and consistent data flows across all systems.
10. Cloud & Infrastructure Support: Work with cloud platforms such as AWS/Azure and DBT with Snowflake to build and maintain scalable data solutions.
11. Continuous Improvement: Proactively look for ways to improve data systems, processes, and tools, ensuring efficiency and scalability.

Essential skills/knowledge/experience:

1. ETL/ELT & Data Pipelines: Solid understanding of ETL/ELT processes, along with hands-on experience building and maintaining data pipelines using DBT, Snowflake, Python, SQL, Terraform, and Airflow.
2. Experience in designing and implementing data products and solutions on cloud-based architectures.
3. Cloud Platforms: Experience working with cloud data warehouses and analytics platforms, such as Snowflake, and AWS/Azure.
4. GitHub Skills and Experience: Proficiency in using GitHub for version control, code collaboration, and managing data engineering projects.
5. Data Governance and Compliance: Expertise in implementing data governance frameworks in Alation, including data quality management and compliance with industry regulations.
6. Effective Communication and Collaboration: Excellent communication skills for interacting with stakeholders, presenting technical concepts, and collaborating with cross-functional teams.
7. Collaboration & Communication: Strong interpersonal skills with the ability to work cross-functionally with stakeholders, engineers, and analysts.
8. SQL Proficiency: Expertise in writing complex SQL queries, query optimization, and database design for analytics.
9. Problem-Solving & Analytical Thinking: Ability to think critically and solve complex problems, translating business requirements into actionable insights.

Desirable skills/knowledge/experience:

1. Knowledge of Data Visualization Tools: Experience with tools such as MicroStrategy / Power BI.
2. Terraform: Experience with Terraform and Terragrunt for infrastructure as code.
3. GenAI: Experience with Generative AI technologies.