NPIC Data Engineer (XN07)

Expected Shortlisting Date:
17/01/2025

Planned Interview Date:
27/01/2025

Due to recent and continued expansion, an exciting opportunity has arisen at NPIC (National Pathology Imaging Cooperative) based at LTHT for a Data Engineer. The candidate will work closely with the NPIC Senior Data Engineer to ensure that data architecture is effectively implemented, maintained, and developed - liaising closely with clinical, technical, and system development staff and system suppliers. The role is crucial for undertaking world-leading research, enabling data and AI scientists to create new insights from digital pathology data.

NPIC works collaboratively with Hospital Trusts, Universities, and commercial companies ranging from spinouts and SMEs to large multinationals to expand the use and impact of digital pathology and artificial intelligence, as well as the development of quality assurance tools for digital pathology. Our research team plays a vital role in realizing our ambitions by helping to develop project opportunities and support projects through their lifecycle.

Main duties of the job

We are seeking to appoint an experienced individual who is driven, ambitious, and passionate about improving patient care through the use of cutting-edge technology. The post-holder will be responsible for building, testing, maintaining data architecture, and undertaking new developments in relation to NPIC's research activity, building cohort creation pipelines for digital pathology and associated healthcare data. The Data Engineer will be responsible for the design, implementation, and delivery of platform enhancements, upstream data linkages, and data acquisition and wrangling. The role will work in a data-rich environment together with data analysts and scientists.

Data engineers will work with NPIC's clinical and research platforms, performing integrations and implementing workflows and models for key use cases. Previous experience with SQL and Python is essential for this role, and experience with NHS information governance procedures will be seen as advantageous.

Job description

Job responsibilities

JOB DETAILS
Job Title: NPIC Data Engineer
Reports to: Digital Pathology Systems Lead
Band: 7
Department: Research and Innovation
Location: SJH
AfC Job No: 4059

JOB PURPOSE/SUMMARY
The National Pathology Imaging Co-operative is a £37m programme funded by the UK Industry Strategy Challenge Fund to accelerate the development of opportunities in Digital Pathology and Artificial Intelligence. We provide a welcoming, professional approach to the facility, a world-class centre for training and knowledge exchange in digital pathology and artificial intelligence.

The role is crucial in laying the foundation, enabling data scientists and data analysts to create new insights from data. The Data Engineer will be responsible for the design, implementation, and delivery of platform enhancements, upstream data lineage, and data acquisition and wrangling. The role will work in a data-rich environment together with data analysts and data scientists.

The role will be responsible for ensuring compounding value is created through shared data, common ontologies, and cross-organisation projects.

Previous experience in SQL data warehouse and ETL is essential for this role, and experience in Power BI Data Models including use of DAX, Python, and R will be seen as advantageous. You will be given ample growth opportunity to learn and gain experience in Power BI and data modelling.

The post holder will liaise with senior managers, clinicians, and system users across a national footprint with directorates and partner organisations. The post holder will work with partners and the local Informatics Departments to assist in the development of any necessary interfaces.

The post holder will work closely with the Digital Pathology Systems Lead to ensure that the architecture is effectively implemented, maintained, and developed, liaising closely with clinical, technical, and system development staff and system suppliers.

The post holder will manage a set of projects to professional standards and will line manage a small team of specialists while providing matrix-management of project staff with management of project budgets, which will vary by project.

PRINCIPAL DUTIES & AREAS OF RESPONSIBILITY

As a Data Engineer you will:

1. Inspire best practice for data products and services within the team.
2. Build data engineering capability by providing technical leadership.
3. Work with other senior team members to identify, plan, develop, and deliver data services.

The day-to-day role involves working with NPIC Teams and the corporate data assets. The role will involve driving forward improvements in data services and supporting the organisation with ongoing projects from a data perspective.

The responsibilities of the role will include:

1. Design, develop, construct, test, and maintain architectures and processing workflows.
2. Build robust, efficient, and reliable data pipelines.
3. Optimisation of information management approaches through an understanding of evolving business needs and technology capabilities.
4. Provide technical support and expert advice on best practices in Power BI, performance analysis of Power BI reports with recommendations for improvement to the other teams in healthcare analytics.
5. Integrating external or new datasets into existing data pipelines.
6. Develop solutions for data acquisition.
7. Design, develop, and maintain data acquisition API for central data Lake.
8. Ensure architecture supports business requirements.
9. Develop dataset processes for data modelling, mining, and production.
10. Drive the collection of new data and refinement of existing data sources.
11. Recommend ways to improve data reliability, efficiency, and quality.
12. Applying feature transformations for machine learning models on new data.
13. Continuously monitor and test the system to ensure optimized performance.
14. Use SQL queries, programming languages like DAX, M, Python & R and tools like DAX studio and tabular model in building data models and other data frames.
15. Identify ways to improve data reliability, efficiency, and quality.
16. Conduct research for advancements and incorporation of new functionalities to meet the reporting requirements of the organisation.
17. Develop tools and techniques to manage large datasets with disparate sources for consolidated reporting.
18. Build and deploy sophisticated analytics programs, machine learning, and statistical methods in data models and data frames for predictive and prescriptive modelling including finding hidden patterns using data and undertake tasks to automate data processing and data management.
19. Ensure new solutions are produced in accordance with local policies and guidelines for all stages of development including design, build, testing, and deployment.
20. Develop and maintain a detailed working knowledge of the technical content of source systems to inform the design and ongoing development of reporting.
21. Undertake a rolling maintenance programme for published data models and reports built by the NPIC Team to provide quality assurance and ensure products are fit for purpose and meet the needs of users.
22. Develop and maintain an in-depth knowledge of NHS data flows, datasets, information availability, scope, reliability, and data quality issues.
23. Develop and maintain effective working relationships with colleagues across the Trust to enable the continuous monitoring, development, and improvement of reporting solutions.
24. Represent the NPIC Deployment Team at internal and external meetings.
25. As a subject matter expert for technical solutions and data management act as a reference point and provide technical guidance and advice as required.
26. Build validation checks and processes into the creation of data models, data frames, reports, and dashboards to ensure data integrity at all stages of the development and implementation of reporting.
27. Contribute to the development of the NPIC Power BI reporting, NPIC data warehouse.
28. Make decisions where appropriate to escalate risks and issues that may impact on the completeness and accuracy of reporting.
29. Ensure all handling of patient-related data is in accordance with information governance, Caldecott, and data protection policies.
30. Deal with highly complex data management issues and identify and use knowledge of systems and data flows to resolve areas of dispute.
31. Interpret local and national guidance and develop, produce, and recommend changes to policies and procedures as required to ensure compliance.
32. Continually review the work undertaken previously and implement changes and improvements as dictated by customer feedback, changes to guidance, and local changes such as ward reconfigurations.
33. Other duties as commensurate with the grade and role as agreed with line manager.

Person Specification

Experience

Essential

• Experience of working with remote and multi-disciplinary teams and driving forward complex work programmes.
• Expert user of Microsoft office products including advanced functions of MS Excel.
• Experience of working in an NHS information environment in an information management role.
• Significant experience of designing, developing, and publishing reports and dashboards across a broad range of areas using different reporting tools such as SSRS.
• Good project management skills, including the ability to juggle competing priorities and tight deadlines for self and team, and work autonomously to drive own work programme and that of team members.

Qualifications

Essential

• Educated to degree level or equivalent level of knowledge gained through relevant work experience.

Desirable

• Educated to masters level.
• A relevant qualification in information management.

Other Criteria

Essential

• SQL Fluent in SQL-based systems like MySQL, PostgreSQL Microsoft SQL Server.
• Significant experience of extracting and processing data from multiple source systems using SQL.
• ETL solutions Must have ETL skills to build processes to move data between systems e.g., Microsoft ADF.
• API design and implementation Understanding of best practice when setting up APIs and the coding ability to develop and deliver the design.
• Big Data Tools Strong understanding and knowledge of big data tools.
• Coding ability Proficiency in Python.

Skills & behaviours

Essential

• Can work autonomously to achieve agreed objectives without reference to a manager.
• Excellent problem-solving skills.
• Capable of working with clinical & non-clinical staff at all levels.
• Strong written & verbal communication skills.

Employer details

Employer name
Leeds Teaching Hospitals

Address
St. James's University Hospital
Beckett Street
Leeds
LS9 7TF

Any attachments will be accessible after you click to apply.
C9298-DRI-0184