Lead Data Engineer

About the Company

We are seeking a skilled Data Engineer to join our team. The ideal candidate will have a strong technical background in designing and developing data pipelines, integrating and managing data systems, and collaborating with various stakeholders to deliver high-quality data solutions. This role is vital to ensuring our data infrastructure is robust, scalable, and optimized to meet the business’s evolving needs.

About the Role

Key Responsibilities:

Design and Develop Data Pipelines:

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional and non-functional business requirements.

Data Integration and ETL:

• Build the infrastructure required for the optimal extraction, transformation, and loading (ETL) of data from various data sources.
• Develop processes and tools to monitor and analyze data pipeline performance.

Database Management:

• Work with data and analytics experts to strive for greater functionality in our data systems.
• Implement, maintain, and optimize database solutions, including both SQL and NoSQL databases.

Collaboration and Communication:

• Collaborate with data scientists, business analysts, and other stakeholders to understand data needs and provide technical solutions.
• Communicate project progress, issues, and solutions effectively to the team and management.

Data Quality and Governance:

• Ensure data quality and data governance standards are maintained.
• Implement data security measures and compliance policies.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• Proven experience as a Data Engineer or in a similar role.

Technical Skills:

• Proficient in SQL and experience with relational databases.
• Experience with big data tools such as Hadoop, Spark, or Kafka.
• Knowledge of data pipeline and workflow management tools like Airflow or Luigi.
• Experience with cloud services (AWS, Azure, GCP) and containerization technologies (Docker, Kubernetes).

Programming Skills:

• Strong programming skills in Python, Java, Scala, or similar languages.

Soft Skills:

• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.
• Ability to work in a fast-paced, dynamic environment.

Preferred Skills

• Experience with machine learning and data mining techniques.
• Familiarity with data visualization tools such as Tableau or Power BI.
• Knowledge of data privacy regulations and best practices.