Azure Data Engineer

Apply for Vacancy

Azure Data Engineer

If you have a CV attach here

Find your CV

I have read and agree with the Privacy Policy including GDPR guidelines.

Location: London (Remote)

Type: Permanent

Job Description

An exciting opportunity for an experienced business-focused Data Engineer with strong SQL skills and expertise in Azure (in particular Synapse) to develop, manage and optimise the reporting data infrastructure (data pipelines, ETL, data lake/warehouse) which underpins Power BI data models and fuels the insight and analytics.

This is a great opportunity for a self-starter to establish themselves as the lead expert and solo data engineer in the Insight and Data Management team to make a real difference as the business transforms its data landscape and incorporates new sources of data to drive value from its data in new and exciting ways.

• Maintain and optimise data pipelines that extract data from source systems covering retail, membership, fundraising and other business areas in D365 systems as well as budget/target data in Sharepoint and other source systems into our Azure data lake and data warehouses ensuring that the data is cleaned, transformed and validated to ensure consistency and quality.
• Proactively monitor and set up alerts for data pipeline failures or issues with performance. When issues occur, identify and resolve these issues, communicating the issue to stakeholders with time estimates to fix the data errors.
• Monitor, analyse and optimise data pipelines to improve performance and ensure reliability.
• Maintain and develop staging tables and first-level views transforming and manipulating data as needed to enable the Insight team to create the final layer used for analysis and Power BI models.
• Manage the pipelines and updates of SQL schemas between UAT and Prod environments.
• Work together with the Insight team to improve overall performance of the data lake/warehouse/Power BI model. For example, improve the efficiency of SQL queries.
• Ensure data quality by troubleshooting and proactively identifying and resolving discrepancies in data and SQL scripts.
• Work together with IT to ensure data solutions comply with relevant data compliance regulations and implement best practices for securing data at rest and in transit.
• Roll out improvements, including automation of the update of SQL schemas between UAT and Prod (CI/CD) and implement snapshot data process for key metrics.
• Scope out and implement new data requirements.
• Collaborate and communicate with Insight team, IT department and business stakeholders across retail, membership, fundraising and ticketing to understand their requirements for new data.
• Develop and implement solutions to extract, transform and include new data into our data lake and data warehouses for these new data requirements considering factors such as update frequency, efficiency, usage and best architecture approach.
• Further develop and maintain data pipeline architecture documentation making it easier to maintain and modify in the future. Includes: overview of data pipeline, data sources, data transformation, data storage.

Education and Experience:

• Educated to degree level or equivalent experience.
• Technically sound with extensive experience of writing complex code in SQL and Python.
• Proficiency in designing, managing, optimising data pipelines and ETL processes ensuring data quality, accuracy and timeliness in data delivery.
• Experience in building, managing and optimising data lakes/warehouses.
• Experience in troubleshooting and optimising existing data processes to improve performance and scalability.
• Proven experience as a data engineer with a focus on Microsoft Azure including Azure Synapse Analytics, Azure Datalake, Azure SQL, Power BI.
• Experience of working in any of the following industries: retail, membership/subscriptions, HNWI relationship management, fundraising (charity), ticketing.
• Experience of IT transformation projects or CRM implementations or upgrades.
• Experience of working with Microsoft Dynamics D365 CE and F&O.
• Experience and understanding of Power BI.

Personal Skills and Abilities:

• Strong communicator with excellent interpersonal skills and stakeholder relationship management - be willing to reach out to the right people and ask the right questions to understand stakeholder data needs.
• Ability to think strategically and beyond the technical by focusing on business needs, the value that data provides and how it will be used.
• Curious and passionate about exploring data - be a trouble-shooter and decipher complex data processes by following data from source systems to end and transforming and integrating data from different sources.
• Team player who thrives on working together with colleagues, sharing knowledge and working towards the same goals.
• Handles sensitive and confidential information with tact and discretion and in line with Data Protection regulations.
• A strong focus on delivery with the ability to effectively prioritise and manage workload and meet tight deadlines.
• An interest in visitor attractions and/or environmental causes.

Ready to work with us? Let's talk!

I give my consent for Boston Hale to contact me.