Data Engineer

The Data Engineer is responsible for designing, developing, and managing data pipelines and architectures, ensuring data is clean, reliable, and accessible for analysis. The ideal candidate will work closely with data scientists, analysts, and other stakeholders to support the development of data-driven insights and decision-making within the organization.

Key Responsibilities:

1. Design and Build Data Pipelines: Develop, construct, test, and maintain data pipelines that facilitate the transfer and processing of data across different systems and platforms.
2. Data Integration: Integrate data from various sources, including internal databases, third-party APIs, and cloud platforms, into centralized data storage solutions such as data lakes or data warehouses.
3. Data Storage Management: Implement and maintain scalable data storage solutions (e.g., SQL/NoSQL databases, data lakes, or data warehouses).
4. Data Transformation and ETL: Create and manage ETL (Extract, Transform, Load) processes to ensure data is clean, transformed, and structured for analysis.
5. Optimization and Performance Tuning: Monitor, optimize, and troubleshoot the performance of data systems and pipelines to ensure efficiency, reliability, and scalability.
6. Collaboration: Work with data scientists, business analysts, and other teams to ensure data accessibility and help translate business requirements into technical solutions.
7. Data Quality Assurance: Implement data quality checks, monitoring, and validation processes to ensure high-quality data is available for decision-making.
8. Documentation and Reporting: Create detailed documentation for data processes, pipelines, and systems for future reference and troubleshooting.
9. Security and Compliance: Ensure data security and compliance with relevant regulations (e.g., GDPR, HIPAA) by implementing best practices for data governance.

Required Skills and Qualifications:

1. A minimum of Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience).
2. Strong knowledge of SQL and database systems (e.g., PostgreSQL, MySQL, etc.).
3. Experience with data platform configuration and tools (e.g., Microsoft Azure data ADLS, data Lakehouse, Synapse).
4. Proficiency in programming languages such as Python, Java, or Scala.
5. Familiarity with cloud platforms (e.g., AWS, Azure, Google Cloud).
6. Experience with data modeling and data warehousing concepts.
7. Data Pipeline Development: Experience in building and maintaining ETL pipelines and data workflows.
8. Familiarity with data tools like Airflow, Talend, or dbt for orchestration and automation.
9. Data Storage: Knowledge of NoSQL databases (e.g., MongoDB, Cassandra) and cloud storage solutions (e.g., Azure, AWS).
10. Strong troubleshooting and debugging skills.
11. Ability to effectively communicate with both technical and non-technical stakeholders.

Preferred Qualifications:

1. Experience with machine learning pipelines or data science workflows.
2. Experience with data visualization tools (e.g. Power BI, Tableau) or analytical tools.

Soft Skills:

1. Strong attention to detail and a commitment to data quality.
2. Excellent communication skills, both written and verbal.
3. Ability to work independently and as part of a team in a fast-paced, dynamic environment.
4. Strong time management skills and the ability to prioritize tasks effectively.