Python Data Engineer

Location: Dubai

Summary: Experienced Python Data Engineer with 5 years in data engineering. Strong foundation in Python, SQL databases, and Snowflake with expertise in data engineering practices and proficiency in tools like Apache Airflow and Shell scripting. This role requires a hands-on approach to designing and maintaining data pipelines and ensuring efficient data integration, storage, and processing.

Technical Skills:

1. Data Warehousing: Snowflake, Data Lake
2. Programming: SQL, Python, Shell Scripting

Core Competencies:

1. Design, develop, and maintain robust, scalable data pipelines to support business operations and analytics using Python.
2. Integrate APIs using Python to fetch, process, and transform data for various use cases.
3. Work with a variety of SQL databases including PostgreSQL, MySQL, MS SQL Server, and Oracle to perform data transformations, migrations, and optimizations.
4. Utilize Snowflake to build and manage data warehouses ensuring high performance and scalability.
5. Implement and manage workflows using Apache Airflow to automate data pipeline orchestration.
6. Develop and maintain Shell scripts to automate tasks and streamline processes.
7. Ensure data quality, consistency, and security across all platforms.
8. Collaborate with data analysts and other stakeholders to understand data requirements and deliver solutions.
9. Monitor, troubleshoot, and optimize data pipelines for efficiency and reliability.

Qualifications and Skills:

1. Programming: Strong proficiency in Python with experience in API integration and data processing.
2. Data Warehousing: Hands-on experience with Snowflake including schema design, data modeling, and performance optimization.
3. Workflow Automation: Proficiency in Apache Airflow for managing and orchestrating workflows.
4. Scripting: Strong skills in Shell scripting for automation and system management.
5. ETL/ELT Processes: Solid understanding of designing and maintaining ETL/ELT pipelines.
6. Problem Solving: Strong analytical and problem-solving skills to handle complex data challenges.
7. Communication: Excellent communication skills to work effectively with cross-functional teams.