Data Engineering Lead

Job Description

Lead Data Engineer

Location: London

A Lead Data Engineer is sought after by a leading workplace pensions provider to run their expanding data engineering team in their London City office. The team has recently built out a next-gen data lake hosted in AWS which makes use of Dremio, Data Mesh architecture, and a wide range of AWS tools (S3, Glue, Step Functions, etc.).

With the infrastructure of this new Data Lake in place, there is now a focus on enhancing the data stored within (via monitoring, cleaning) and a requirement to design, build, and implement robust ETL pipelines for effective integration with the wider business.

As a Lead Engineer, you will also be responsible for overseeing a small team of Data Engineers, setting and driving best practices and standards, ensuring project delivery is kept on track, and regularly engaging with stakeholders up to C-suite for feedback/updates. This will also involve working directly alongside the Head of Data Platforms to ensure ongoing engineering work is aligned to overall architecture and roadmap, as well as future project planning.

Required Skills

1. 7/8 years’ experience in Python programming, particularly around data analysis, automation, and the building of data pipelines (as well as an understanding of data warehousing concepts)
2. Strong SQL skills, especially in relation to ETL processes and data management (querying, cleaning, storing)
3. Experience working with and deploying to AWS cloud (having worked with any/all of S3, Glue, Lambda, Step Functions, etc.)
4. Usage of PowerBI for data visualization
5. Experience of both Dremio and Data Mesh Architecture
6. Prior experience of leading software engineering teams (including line management)
7. Clear, confident communications with previous experience of working directly with business users