Senior Data Engineer

Overview

The Senior Data Engineer role will plan and drive forward the design and implementation of data pipelines, data stores, and APIs to acquire and prepare data for downstream applications, such as analytics, data insights, and data-driven tools. You will be a hands-on data engineer, who will contribute to the wider teams’ outputs by ensuring data standards and architectures are applied, and by coaching colleagues on approaches to problem-solving and quality of deliverables. Additionally, you will be responsible for integrating data with various applications to ensure seamless data flow and accessibility.

The data domain will cover RSSB, rail industry, and national data sources in the areas of health, safety, quality, operational performance, and sustainability. It will also cover internal business datasets, ensuring that data is effectively integrated with applications to support business processes and decision-making.

Responsibilities

• Ensure data made available for downstream applications is reliable, accessible, and in the appropriate format through active engagement with Product Management and key stakeholders.
• Plan and drive forward the development of data solutions, ensuring a balance between functional and non-functional requirements.
• Advise on best approaches to solving novel data problems, leveraging Python-based solutions.
• Ensure that data standards and architectures are applied to achieve well-engineered outcomes, and that solutions are reliable and perform optimally.
• Conduct complex data quality checks and monitoring, applying necessary remediation to maintain data integrity.
• Undertake necessary maintenance and optimisation activities to ensure the performance and reliability of data pipelines, APIs, and applications.
• Develop both quick prototypes and high-quality products, judging which approach is needed based on the context.
• Design, code, test, and document complex data engineering scripts and applications, applying software engineering best practices with a focus on SQL and Python.
• Collaborate effectively with cross-functional teams, including Product Management and key stakeholders, to understand business requirements and translate them into technical solutions.
• Communicate technical concepts and solutions effectively to both technical and non-technical audiences.
• Mentor and coach colleagues on approaches to problem-solving and quality of deliverables, fostering a collaborative and innovative team environment.

Qualifications

Essential

• BSc in a scientific, computer science, or related discipline.
• Extensive experience with data analysis, data modelling, and quality assurance techniques.
• Proficiency in using SQL and Python to solve a wide array of data challenges.
• Experience developing data pipelines in Python using orchestrators such as Airflow.
• Experience making data available in data science and data visualization tools such as Databricks and Power BI/Fabric.
• Experience with data extraction from a variety of systems, including familiarity with web-oriented architecture.
• Ability to analyse and document technical solutions effectively.
• Communicate effectively and share thoughts and ideas through methods appropriate to the audience.
• Adapt and respond effectively when embracing new opportunities, change and in navigating uncertainty.
• Actively contribute as part of a team and work towards achieving team goals and outcomes.
• Take responsibility and demonstrate accountability in completing tasks and achieving objectives, actively seeking to resolve problems and identify opportunities.
• Committed to customer service and placing customer satisfaction at the heart of our success to ensure we deliver against our shared goals.
• Can work collaboratively within an evolving industry, gaining stakeholder confidence through understanding their goals and motivations and demonstrating credibility as an expert.
• Make timely, informed decisions taking account of the benefits and constraints involved.

Desirable

• Experience developing and deploying APIs in Python using frameworks such as FastAPI.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Experience with version control systems such as Git.
• Understanding of Agile methodologies and experience working in Agile teams.