Data Engineering Manager

We are seeking a Data Engineering Manager to lead the development of scalable data pipelines, ETL workflows, and analytics solutions while driving the migration of on-premises Microsoft SQL Server databases to Azure. This role is responsible for ensuring efficient data ingestion, transformation, and storage while leveraging Apache Kafka for real-time data streaming and event-driven architectures.

The ideal candidate has strong expertise in Azure cloud data services, Microsoft SQL Server, and data engineering best practices, with experience leading high-performing teams and managing cloud migration projects.

Role And Responsibilities

Data Engineering & Pipeline Development

1. Design, build, and maintain scalable ETL and data pipelines using Azure Data Factory and Databricks.
2. Develop real-time and batch data processing workflows for analytics and reporting.
3. Ensure data integrity, security, and governance across all pipelines.
4. Implement event-driven architectures with Apache Kafka and Azure Event Hubs.

On-Premises to Azure Migration

1. Lead the migration of Microsoft SQL Server databases to Azure SQL Database and Synapse Analytics.
2. Define and implement Azure-based data architecture for performance, scalability, and cost efficiency.
3. Optimize data storage, indexing, and partitioning strategies in Azure.
4. Work closely with DBA, DevOps, and security teams to ensure a seamless migration with minimal downtime.

Data Architecture & Infrastructure

1. Oversee the development of a modern Azure data platform (data lake, data warehouse, and real-time processing).
2. Manage and optimize Kafka-based streaming architectures using Azure Event Hubs.
3. Implement data governance, security policies, and compliance frameworks (BNM RMiT, GDPR).
4. Monitor and enhance data performance, availability, and reliability.

Data Analytics & Business Intelligence Enablement

1. Work with data analysts and scientists to develop self-service analytics and reporting.
2. Design efficient data models to support business intelligence (BI) using Azure Synapse Analytics and Power BI.
3. Enable seamless integration between operational and analytical data sources.

Team Leadership & Collaboration

1. Lead and mentor a team of data engineers, fostering best practices in Azure data engineering.
2. Promote Agile methodologies, DevOps, and CI/CD pipelines for data workflows.
3. Collaborate with engineering, product, and business teams to define data objectives.

Performance Monitoring & Continuous Improvement

1. Monitor and troubleshoot Azure data pipelines, ETL processes, and Kafka streams.
2. Identify opportunities to automate and improve efficiency.
3. Stay up to date with Azure cloud, big data, and streaming technologies.

Qualifications & Requirements

Technical Skills:

1. Strong experience with Microsoft SQL Server, including query optimization and indexing.
2. Expertise in Azure Data Services (Azure SQL Database, Synapse Analytics, Data Factory, Data Lake).
3. Hands-on experience with Apache Kafka & Azure Event Hubs for real-time streaming.
4. Proficiency in Python, SQL, or Scala for data transformation and processing.
5. Experience with ETL tools (Azure Data Factory, dbt, Apache Airflow).
6. Strong understanding of data security, governance, and compliance (BNM RMiT, GDPR).
7. Familiarity with CI/CD and DevOps practices for data engineering in Azure.
8. Experience with big data frameworks (Databricks, Apache Spark).

Leadership & Soft Skills

1. Proven experience leading data engineering teams and projects.
2. Strong collaboration skills with engineering, product, and business teams.
3. Excellent problem-solving and decision-making skills.
4. Strong communication skills for both technical and non-technical stakeholders.

Preferred Qualifications

1. Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
2. 5+ years of experience in data engineering, analytics, or cloud migration, with at least 2+ years in a managerial role.
3. Experience in enterprise database migrations from on-prem SQL Server to Azure.
4. Previous experience in high-scale, real-time data streaming environments (e.g., fintech, payments, remittance).