Data Engineer / Scientist

Austin Fraser; London; On-site; GBP 50,000 - 90,000; Full time

Job description

Overview

We are seeking an experienced Data Engineer, preferably at Manager level, to join our growing Data & Analytics function. You will play a key role in designing, building, and optimising scalable data pipelines and data solutions on Azure. The ideal candidate will have deep expertise in Spark, Python, and modern data engineering practices, with a strong understanding of data governance and security.

Key Responsibilities

• Design, develop, and optimise robust ETL/ELT pipelines using Azure Data Factory (ADF) and Spark.
• Work with Databricks to build and maintain scalable data processing solutions.
• Implement best practices for data governance using Unity Catalog.
• Manage data storage and access within Azure Data Lake Storage (ADLS) and blob storage.
• Write clean, efficient, and scalable code using Python and SQL.
• Integrate and secure access to sensitive data using Azure Key Vault.
• Collaborate with cross-functional teams including Data Scientists, Analysts, and Business stakeholders to understand data needs and deliver reliable data solutions.
• Ensure data quality, consistency, and reliability across all data systems and pipelines.
• Contribute to the continuous improvement of data engineering processes, standards, and tools.

Required Skills & Experience

• Strong hands-on experience with Apache Spark; experience with Databricks is highly desirable.
• Proficiency in Python for data engineering and scripting.
• Solid understanding of ETL/ELT concepts, with hands-on experience using Azure Data Factory (ADF).
• Advanced SQL skills for data transformation, manipulation, and performance tuning.
• Experience with Azure Data Lake Storage (ADLS) and Blob Storage.
• Knowledge of Unity Catalog for managing data access, lineage, and compliance.
• Familiarity with Azure Key Vault for managing secrets and secure configuration.
• Proven experience working in a modern cloud-based data platform environment.
• Ability to work independently as well as part of a team in a fast-paced, agile environment.
• Strong problem-solving skills, attention to detail, and communication abilities.