Data Engineer

As a Data Engineer, you will be required to:

1. Perform data extraction, cleaning, transformation, and flow.
2. Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
3. Integrate and collate data silos in a manner that is both scalable and compliant.
4. Collaborate with Project Manager, Frontend Developers, UX Designers, and Data Analysts to build scalable data-driven products.
5. Be responsible for developing backend APIs and working on databases to support the applications.
6. Bridge the gap between engineering and analytics.
7. Work in an Agile Environment that practices Continuous Integration and Delivery.
8. Work closely with fellow developers through pair programming and code review processes.

EXPERIENCE AND SKILLS NEEDED

As a Data Engineer, you need to bring to the team:

1. Experience with building production-grade data pipelines, ETL/ELT data integration with proper documentation.
2. Proficient in general data cleaning and transformation (e.g., pandas, pyspark, SQL).
3. Knowledge in database design and various databases (e.g., structured databases like Postgres, MySQL, and unstructured databases like MongoDB).
4. Knowledge about system design, data structures, and algorithms.
5. Knowledge with REST APIs and web requests/protocols in general.
6. Comfortable coding in at least 2 scripting languages (e.g., Python, SQL).
7. Comfortable working in both Windows and Linux development environments.
8. Interest in data engineering in a big data environment using Cloud platforms (i.e., AWS, Azure, Google Cloud).
9. Familiar with data modeling, data access, and data storage infrastructure like Data Mart, Data Lake, and Data Warehouse.
10. Proficient in creating comprehensive data dictionaries to ensure clear and consistent data definitions.
11. Skilled in designing Entity-Relationship Diagrams (ERDs) to visually represent data models and relationships.
12. Capable of collaborating with stakeholders to understand business processes and develop effective data solutions.
13. Familiar with BI Tools such as Tableau.
14. Familiar with Docker containers.