Specialized IT Consultant - Senior

Position: Specialized IT Consultant

Location: Toronto, ON (Hybrid)

Duration: 319 Days Contract + Possible Extension

Experience:

1. Experience of 7+ years of working with SQL Server, T-SQL, Oracle, PL/SQL development or similar relational databases
2. Experience of 2+ years of working with Azure Data Factory, Databricks and Python development
3. Experience building data ingestion and change data capture using Oracle Golden Gate
4. Experience in designing, developing, and implementing ETL pipelines using Databricks and related tools to ingest, transform, and store large-scale datasets
5. Experience in leveraging Databricks, Delta Lake, Delta Live Tables, and Spark to process structured and unstructured data
6. Experience working with building databases, data warehouses and working with delta and full loads
7. Experience on Data modeling, and tools – e.g. SAP Power Designer, Visio, or similar
8. Experience working with SQL Server SSIS or other ETL tools, solid knowledge and experience with SQL scripting
9. Experience developing in an Agile environment
10. Understanding data warehouse architecture with a delta lake
11. Ability to analyze, design, develop, test and document ETL pipelines from detailed and high-level specifications, and assist in troubleshooting
12. Ability to utilize SQL to perform DDL tasks and complex queries
13. Good knowledge of database performance optimization techniques
14. Ability to assist in the requirements analysis and subsequent developments
15. Ability to conduct unit testing and assist in test preparations to ensure data integrity
16. Work closely with Designers, Business Analysts and other Developers
17. Liaise with Project Managers, Quality Assurance Analysts and Business Intelligence Consultants
18. Design and implement technical enhancements of Data Warehouse as required

Technical Skills (70 points):

1. Experience in developing and managing ETL pipelines, jobs, and workflows in Databricks
2. Deep understanding of Delta Lake for building data lakes and managing ACID transactions, schema evolution, and data versioning
3. Experience automating ETL pipelines using Delta Live Tables, including handling Change Data Capture (CDC) for incremental data loads
4. Proficient in structuring data pipelines with the Medallion Architecture to scale data pipelines and ensure data quality
5. Hands-on experience developing streaming tables in Databricks using Structured Streaming and readStream to handle real-time data
6. Expertise in integrating CDC tools like GoldenGate or Debezium for processing incremental updates and managing real-time data ingestion
7. Experience using Unity Catalog to manage data governance, access control, and ensure compliance
8. Skilled in managing clusters, jobs, autoscaling, monitoring, and performance optimization in Databricks environments
9. Knowledge of using Databricks Autoloader for efficient batch and real-time data ingestion
10. Experience with data governance best practices, including implementing security policies, access control, and auditing with Unity Catalog
11. Proficient in creating and managing Databricks Workflows to orchestrate job dependencies and schedule tasks
12. Strong knowledge of Python, PySpark, and SQL for data manipulation and transformation
13. Experience integrating Databricks with cloud storage solutions such as Azure Blob Storage, AWS S3, or Google Cloud Storage
14. Familiarity with external orchestration tools like Azure Data Factory
15. Implementing logical and physical data models
16. Knowledge of FHIR is an asset

Design Documentation and Analysis Skills (20 points):

1. Demonstrated experience in creating design documentation such as:
2. Schema definitions
3. Error handling and logging
4. ETL Process Documentation
5. Job Scheduling and Dependency Management
6. Data Quality and Validation Checks
7. Performance Optimization and Scalability Plans
8. Troubleshooting Guides
9. Data Lineage
10. Security and Access Control Policies applied within ETL
11. Experience in Fit-Gap analysis, system use case reviews, requirements reviews, coding exercises and reviews
12. Participate in defect fixing, testing support and development activities for ETL
13. Analyze and document solution complexity and interdependencies including providing support for data validation
14. Strong analytical skills for troubleshooting, problem-solving, and ensuring data quality

Communication and Leadership Skills (10 points):

1. Ability to collaborate effectively with cross-functional teams and communicate complex technical concepts to non-technical stakeholders
2. Strong problem-solving skills and experience working in an Agile or Scrum environment
3. Ability to provide technical guidance and support to other team members on Databricks best practices
4. Must have previous work experience in conducting Knowledge Transfer sessions, ensuring the resources will receive the required knowledge to support the system
5. Must develop documentation and materials as part of a review and knowledge transfer to other members

Must Have Skills:

1. 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL
2. 2+ Delta Lake, Databricks and Azure Databricks pipelines
3. Strong knowledge of Delta Lake for data management and optimization
4. Familiarity with Databricks Workflows for scheduling and orchestrating tasks
5. 2+ years Python and PySpark
6. Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments
7. Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data
8. SQL Server, Oracle