Lead Data Engineer

We are on the lookout for an experienced Lead Data Engineer to take on a leadership role within our client’s Data Engineering team, driving technical excellence and transforming data ingestion processes. You will be responsible for implementing the Data Engineering strategy, working closely with the Head of Data Engineering to define best practices, and collaborating with Project teams to ensure technical specifications and tasks are clear and achievable for all team members.

Responsibilities

1. Lead data engineering projects and mentor junior team members
2. Implement the data engineering strategy in collaboration with the Head of Data Engineering
3. Define testing frameworks for validating and transforming synthetic data
4. Set best practices for containerisation and deployment to Kubernetes clusters
5. Conduct monthly AWS cost reviews with DevOps to identify savings and optimise resources

Qualifications

1. Experience in lead data engineering or similar senior role, with strong leadership and architecture design experience
2. Advanced Python and SQL skills
3. Hands-on experience with Apache Kafka, Apache Spark, and Delta tables
4. Proficient in Airflow and message brokers (e.g. RabbitMQ, AWS SQS)
5. Expert-level knowledge of Docker and Kubernetes