Senior Data Engineer

Aculocity Data Team: Senior Data Engineer, Job description

Premise:

The Senior Data Engineer is to be a lead resource in building and developing the Aculocity Data capabilities.

As Senior Data Engineer, you will play a pivotal role in our agile Data Engineering team, leading the design, development, and optimization of complex data infrastructure and pipelines. Your expertise will be crucial in delivering high-impact reporting, analytics, and machine learning solutions that drive business success. You will leverage your extensive experience to not only build and maintain critical data systems but also to mentor and guide more junior engineers, ensuring the seamless translation of intricate business requirements into robust and scalable data solutions.

Job Title: Senior Data Engineer

Reports To: Data Engineering Manager

Data Engineer Job Responsibilities:

1. Serve as the subject matter expert for data and systems.
2. Develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexity.
3. Collaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
4. Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
5. Develop end-to-end ML pipelines encompassing the ML lifecycle from data ingestion, data transformation, model training, model validation, model serving, and model evaluation over time.
6. Collaborate closely with AI scientists to accelerate deployments of ML algorithms to production.
7. Setup CI/CD/CT pipelines, model repository for ML algorithms.
8. Deploy models as a service.
9. Contribute to engineering wiki, and document work.
10. Perform data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
11. Work closely with cross-functional teams of frontend and backend engineers, product managers, and analysts to enhance data models and support advanced BI and analytics.
12. Define company data assets (data models), ETL jobs to populate data models.
13. Design data integrations and data quality framework.
14. Design and evaluate open source and vendor tools for data lineage.
15. Work closely with all business units and engineering teams to develop strategy for long term data platform architecture.
16. Mentor junior data engineers, lead code reviews, and promote best practices and skill development.

Data Engineer Qualifications / Skills:

1. Knowledge of best practices and IT operations in an always-up, always-available service.
2. Establish and promote best practices for data pipeline and model development.
3. Experience with or knowledge of Agile Software Development methodologies.
4. Excellent problem solving, creativity, attention to detail and troubleshooting skills.
5. Process oriented with great documentation skills.
6. Excellent oral and written communication skills with a keen sense of customer service.
7. Eagerness to learn and upskill to stay at the forefront of technology offerings in the market.
8. Understanding of ML Algorithms, experience creating & executing efficient MLOps pipelines, and tuning ML models.
9. Team player mindset with an enthusiasm for collaboration.

Education, Experience, and Licensing Requirements:

Must Have:

1. BSc or MSc degree in Computer Science or a related technical field.
2. 5+ years of Python or R development experience.
3. 5+ years of MS SQL experience (PostgreSQL experience is a plus).
4. 5+ years of experience with Warehouse Architecture, schema design and dimensional data modeling.
5. 5+ years of experience in Data Analytics and Business Intelligence tools such as Power BI.
6. Ability in managing and communicating data warehouse plans to internal clients.
7. Experience designing, building, and maintaining data processing systems on multiple platforms both Cloud (Azure, AWS, MS Fabric is a plus) and On-Premises (MS SQL Server, SSIS).
8. Experience in ML Model deployment, ML frameworks and libraries.
9. Good experience in Apache Spark.
10. Experience debugging and reasoning about production issues is desirable.
11. Experience presenting demos and training of technical, non-technical and analytical resources.

Advantages to have:

1. Experience in data streaming is advantageous i.e. Kafka and/or AWS Kinesis.
2. IoT device and systems integration.