Data Engineer

N Consulting Ltd; London; Hybrid; GBP 100,000 - 125,000

Job description

Role : Data Engineer
Work Mode : Hybrid
Contract Role
Location : London OR Edinburgh
Experience : 10 -12 Years

Job Description:

1. Design, develop, and maintain ETL processes to move data from source systems to Snowflake, Teradata and SQL Server environments.
2. Build scalable data pipelines for processing and storing large datasets.
3. Collaborate with analysts and business stakeholders to define and implement data strategies.
4. Optimize database performance, including query optimization, indexing, and partitioning across all enterprise data platforms.
5. Work with cross-functional teams to gather requirements and develop effective data solutions.
6. Ensure data quality, consistency, and integrity throughout the data lifecycle.
7. Maintain and improve existing data architectures and workflows to meet business requirements.
8. Create and maintain documentation for data systems, pipelines, and processes.
9. Monitor and troubleshoot data pipeline performance issues and resolve them promptly.
10. Assist in the migration and integration of data from legacy systems into Snowflake and other strategic data stores.
11. Perform data transformation tasks using SQL, Snowflake SQL, and relevant data processing tools.

Required Skills and Qualifications:

1. Proven experience working with SQL Server (e.g., T-SQL, Stored Procedures, Indexing, Query Optimization, System Catalog Views).
2. Strong experience in Snowflake architecture, including data loading, transformation, and performance tuning.
3. Proficient in ETL processes using tools such as Informatica PowerCenter and BDM, AutoSys, Airflow, and SQL Server Agent.
4. Experience with cloud platforms, preferably AWS.
5. Strong knowledge of AWS cloud services, including EMR, RDS Postgres, Redshift Athena, S3, and IAM.
6. Solid understanding of data warehousing principles and best practices.
7. Strong proficiency in SQL for data manipulation, reporting, and optimization.
8. Knowledge of data modeling and schema design.
9. Experience working with large, complex datasets and implementing scalable data pipelines.
10. Familiarity with version control tools such as GitLab.
11. Experience with data integration, data governance, and security best practices.