Data Engineer

Roles and Responsibilities

Data Engineer

Location: Dubai

We are looking for Experience up to: 5-8 Years

Summary: Experienced Informatica PC and IDMC Developer with 5+ years in data engineering. Skilled in designing end-to-end data pipelines using PC/IDMC for effective migration and transformation across cloud platforms. The ideal candidate will have in-depth knowledge of Informatica PowerCenter and IDMC. A strong foundation in Data Warehousing concepts and proficiency in Snowflake and SQL is essential. Strong team player with agile experience, delivering timely, high-impact data solutions.

Technical Skills

• Tools: Informatica Cloud Data Integration, Informatica PowerCenter
• Data Warehousing: Snowflake, DataLake
• Programming: SQL, Python, Shell Scripting
• Data Management: Storage management, quality monitoring, governance
• Modeling: Dimensional modeling, star/snowflake schema

Core Competencies

• Design, develop, and optimize ETL workflows using Informatica PowerCenter (PC) and Informatica IDMC.
• Manage data ingestion processes from diverse data sources such as Salesforce, Oracle databases, PostgreSQL, and MySQL.
• Implement and maintain ETL processes and data pipelines to ensure efficient data extraction, transformation, and loading.
• Utilize Snowflake as the data warehouse solution for managing large volumes of structured and unstructured data.
• Maintain and optimize ETL jobs for performance and reliability, ensuring timely data availability for business users.
• Support data migration, data integration, and data consolidation efforts.
• Write and maintain basic Python scripts for data processing and automation tasks.
• Utilize Unix shell commands for data-related tasks and system management.
• Troubleshoot and resolve ETL-related issues, ensuring data integrity and availability.
• Ensure adherence to best practices for data governance and security.

Professional Experience

• Informatica Developer
• Developed ELT processes using PC/IDMC to integrate data into Snowflake.
• Implemented storage management for Azure Blob and Snowflake, enhancing data security.
• Worked on basic Python and shell scripting languages.

Desired Candidate Profile

• Data Architecture Design:

  • Design and develop scalable, efficient, and reliable data architectures that support data storage, processing, and analytics needs.
  • Build and maintain data warehouses, databases, and other storage solutions.
  • Create and optimize data models that meet the organization’s analytical and business requirements.

• Data Pipeline Development:

  • Build and maintain ETL (Extract, Transform, Load) pipelines to move data from various sources to data warehouses, databases, or cloud storage systems.
  • Ensure pipelines can handle high volumes of data, perform transformations, and ensure data integrity and accuracy.
  • Automate data workflows and ensure they are scalable and robust for ongoing data processing tasks.

• Data Integration:

  • Integrate data from diverse sources (databases, APIs, third-party services, etc.) into unified systems.
  • Collaborate with external teams to gather data requirements and develop integration solutions.
  • Manage batch and real-time data integration processes, ensuring that data flows continuously and efficiently.

• Data Quality Management:

  • Ensure the accuracy, consistency, and reliability of data by implementing quality checks and monitoring.
  • Identify and resolve issues related to missing, incomplete, or inconsistent data.
  • Establish and maintain data governance standards to ensure compliance with regulatory and internal data policies.

• Data Warehousing:

  • Develop and optimize data warehouses, ensuring they are structured efficiently for query performance and analysis.
  • Ensure data storage is cost-effective while providing fast access and scalability.
  • Support the implementation of OLAP (Online Analytical Processing) and OLTP (Online Transaction Processing) solutions.

• Performance Optimization:

  • Optimize query performance to ensure fast data retrieval, using techniques like indexing, partitioning, and caching.
  • Monitor and tune data systems for better performance, minimizing bottlenecks and downtime.
  • Scale data infrastructure based on increasing data volumes and processing demands.

• Cloud Data Engineering:

  • Work with cloud platforms like AWS, Google Cloud Platform (GCP), or Microsoft Azure to deploy, manage, and scale data storage and processing systems.
  • Utilize cloud-native tools (e.g., AWS Redshift, BigQuery, Azure Synapse Analytics) to manage and process data.

• Collaboration with Data Teams:

  • Work closely with data scientists and data analysts to understand their data requirements and provide clean, reliable, and accessible data.
  • Support the development of machine learning models by providing clean, labeled, and pre-processed data.
  • Collaborate with business intelligence teams to ensure data is structured in a way that supports reporting and analytics.