Azure Data Engineer (Databricks Expert)

Role : Azure Data Engineer (Databricks Expert)

Location : Toronto, ON

Hire Type : Full-Time with Eviden

Job Description:

About the Role:

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join our team and tackle critical performance challenges within the Azure data platform. You will play a pivotal role in optimizing and re-architecting key components of data infrastructure, directly impacting the performance and scalability of core data marts. This is not a maintenance role; this is a chance to make a significant impact by redesigning and implementing solutions for a high-visibility project.

Responsibilities:

1. Performance Optimization: Lead the performance tuning and optimization of Azure-based data pipelines, specifically addressing the current bottleneck where processing 1000 records takes 40 minutes. This requires a deep understanding of Azure Data Factory (ADF), Databricks (Spark), and data processing best practices.
2. Architecture Review and Redesign: Evaluate the existing Azure data platform architecture, identify bottlenecks and design flaws, and propose and implement solutions for a more efficient and scalable system. This will involve working with Delta tables and optimizing data storage and retrieval.
3. Databricks Expertise: Utilize your advanced PySpark skills, particularly with structured streaming, to re-engineer data transformation and processing logic within Databricks. The ideal candidate will have a proven track record of optimizing Spark jobs for performance.
4. Data Integration: Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Database (DB), and file exchange processes (batch mode) to ensure seamless data flow and integration.
5. MDM Integration (Desirable): Contribute to the ongoing development and optimization of the Master Data Management (MDM) system, which follows a hub-and-spoke architecture. Experience with large-scale MDM implementations is highly desirable.
6. Collaboration: Work closely with other engineers, business stakeholders, and vendors to understand requirements, communicate progress, and ensure successful project delivery. You will be expected to mentor and guide other team members.

Qualifications:

1. Minimum experience of implementing 2 projects on Azure data platform.
2. Minimum experience of around 8-12 years.
3. Ninja Level Expertise: A proven track record of success in optimizing and scaling Azure data platforms, particularly with Databricks and Spark. We are looking for an expert who can quickly diagnose and resolve complex performance issues.
4. Deep Azure Knowledge: Extensive experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, Event hub, and other relevant Azure services. Experience with Azure SaaS offerings is a plus.
5. Databricks Mastery: Advanced proficiency in PySpark and structured streaming. Demonstrated ability to write and optimize complex Spark queries for performance.
6. Data Warehousing and MDM: Solid understanding of data warehousing principles and experience with Master Data Management (MDM) implementations, preferably with a hub-and-spoke architecture.
7. Problem-Solving Skills: Exceptional analytical and problem-solving skills, with the ability to identify root causes of performance bottlenecks and design effective solutions.
8. Communication Skills: Excellent communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.

Desirable:

1. Experience with AI Search and Cosmos DB.
2. Experience with Informatica MDM in an Azure environment.