Senior Data Engineer - Contango

Our Team

Our team comprises highly skilled professionals who bring a wealth of international experience in technology entrepreneurship. We thrive on ambition and recognize that fun and performance go hand in hand. Together we create a dynamic environment where excellence is the currency of success.

Position Overview

As a Data Engineer, you will play a crucial role in designing, building, and maintaining our data infrastructure. You will be responsible for ensuring the availability, integrity, and efficiency of our data pipelines, enabling our organization to make informed data-driven decisions. The value you bring will be instrumental in delivering measurable value to Contango and beyond using data and AI.

Responsibilities

• Leadership and ownership: Own overall data lake and data warehouse architecture with the responsibilities to design, architect, and maintain the data warehouse infrastructure to support data management, data governance, analytics, ML & AI needs, and address future challenges.
• Data pipeline development: Design, build, and maintain scalable data pipelines for ingesting, processing, and transforming data from various sources, ensuring data quality, scalability, and reliability.
• Data modelling: Develop and maintain data models, architecture patterns, schemas, and structures that support the needs of data analysts, data scientists, and other stakeholders.
• ETL (Extract, Transform, Load): Create and optimize ETL processes to extract data from diverse sources, transform it into usable formats, and load it into data warehouses or other storage solutions.
• Data governance, quality, and validation: Establish and enforce data governance policies, standards, procedures, and best practices, and implement data quality checks, validation processes, and error handling to ensure the accuracy and consistency of data.
• Performance tuning: Continuously monitor and optimize the performance of data pipelines to meet business requirements and scalability needs.
• Data security: Implement and maintain data security measures to protect sensitive information and ensure compliance with data privacy regulations.
• Collaboration: Work closely with data analysts, data scientists, and other stakeholders to understand their data requirements and provide support in data access and availability.
• Documentation: Maintain thorough documentation of data engineering processes, data models, and data dictionaries.
• Stay informed: Keep up to date with emerging trends and technologies in data engineering to ensure our data infrastructure remains cutting-edge.

Qualifications

• Bachelor's degree in computer science, information technology, or a related field (master's degree is a plus).
• Demonstrable experience of minimum 8 years, with at least 5 years in a leadership position as a Senior Data Engineer or similar, with a portfolio of projects; 10 years of experience in data engineering with a focus on designing and implementing data lakes, data warehouses, data modelling, ETL processes, and data transformation pipelines.
• Experience in cloud-based data platforms (e.g., Spark, Azure Synapse Analytics, etc.).
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong knowledge of database systems (SQL and NoSQL), data warehousing, and big data technologies.
• Knowledge of data governance principles, data quality management, and regulatory compliance.
• Experience with orchestration services (i.e., Azure Data Factory (ADF), Azure Logic Apps, Azure Functions, Databricks Jobs, Airflow, etc.).
• Familiarity with data engineering integration tools and frameworks and ETL processes.
• Experience with cloud platforms like AWS, GCP, or Azure.
• Experience with API development and integration with data services.
• Excellent communication, leadership, and collaboration skills.
• Ability to work in a fast-paced dynamic environment and lead multiple projects and teams to deliver high-quality results.
• Technical Expertise:
  • Programming languages: Python, Java, Scala, and R.
  • Data management & databases: Oracle, SAP, SQL, NoSQL, Data Warehousing.
  • Big data technologies: Apache Hadoop, Spark, Kafka, etc.
  • Cloud platforms: Experience with Microsoft Fabric, Azure (Synapse Analytics, Databricks, Machine Learning, AI, Search, Functions, etc.), Databricks on Azure, AWS (S3, Redshift, SageMaker, etc.), Google Cloud Platform (BigQuery, Dataflow, etc.).
  • Data governance: Experience with Ab Initio, Informatica, Collibra, Purview, IBM InfoSphere, Great Expectations, Deepchecks, Databricks Unity Catalog, Databricks Delta Sharing, Databricks Catalog Explorer, Databricks Audit Logging, and Databricks Identity Management.
  • Data visualization: Familiarity with Matplotlib, Seaborn, Tableau, Power BI.
  • DevOps & MLOps: CI/CD principles, Docker, MLFlow, Kubernetes.

Benefits

• Competitive salary and benefits package.
• Opportunity to work on cutting-edge technology projects.
• Collaborative and innovative work environment.
• Professional growth and training opportunities.

If you possess the requisite experience, qualifications, and leadership acumen to drive our success and shape the future of Data and AI innovation within the UAE, we invite you to submit your application.