Data Engineer

About the Role:

As part of a cross-functional team of engineers, data scientists, and product owners you will be responsible for designing, implementing, optimizing, and evaluating our product (specific product language here) algorithms. If you love seeing your ideas blossom through their whole life cycle from concept to governing the interaction with millions of users - then Avrioc is the place to be for you!

Responsibilities

• Collaborate with various stakeholders to deeply understand the needs of data practitioners and deliver at scale.
• Help define, build, and maintain the Data Platform.
• Work with emerging technologies to build distributed applications.
• Manage end-to-end development efforts for timely delivery of high-quality solutions that meet requirements, align with the architectural vision, and comply with applicable standards.
• Present technical solutions, capabilities, considerations, and features in business terms.
• Contribute to critical initiatives such as Data Discovery, Data Lineage, and Data Quality.
• Build data systems, pipelines, analytical tools, and programs.
• Conduct complex data analyses and report on results.

Skills and Attributes

• Strong technical expertise in data modeling, data ingestion, distributed processing, data warehousing, and ETL processes.
• Experience with Apache Kafka and Spark Streaming.
• Exposure to connecting and querying different data sources, including NoSQL databases (e.g., MongoDB), MySQL, Elasticsearch, and Kafka topics.
• Proficiency in SQL, Python, PySpark, Spark SQL, and a solid understanding of the Databricks platform.
• Ability to build large-scale batch and real-time data pipelines.
• Knowledge of best practices for optimizing Databricks cluster configurations (size, type, memory) based on data requirements and delivering optimized code.
• Experience in maintaining, monitoring, and handling enhancement requests for data pipelines.
• Familiarity with visualization tools such as Kibana and QlikView.
• Exposure to Docker, Kubernetes, and GitLab.
• Experience with JIRA processes and Confluence for documentation and project management.

Qualification & Requirements

• Degree in Computer Science, Data Science, Mathematics, IT, or a related field.
• 5 years of Experience as a data engineer or similar role.