Data Engineer

Explore exciting open positions in the IT and Telecom Networking Industry today.

Job Description

Description:

As an ML Data Engineer, you will be responsible for supporting and implementing code in order to build up the ML Development and Production process. You will be working with a team of data scientists and data engineers to provide an advanced security monitoring solution for Bell. For this reason, you will be asked to assess the current Big Data technology in use and to test new ones to evolve the system. We are not looking for a Data Scientist, but for a Big Data Engineer that understands the ML process and has experience working with Data Scientists. Your responsibilities will be to support the architecture team to realize this ambitious new program of work. You will need to be comfortable building both new systems and implementing and integrating with existing open-source and commercial products. Part of your daily work will be to challenge certain concepts and ideas. The goal is to help your peers' knowledge to evolve and to open their minds to new approaches.

Responsibilities:

• Integrating machine learning models onto a Hadoop based big data environment using streaming technology.
• Working with data scientists to develop part of their code to interface with other components.
• Liaise with the DevOps team maintaining the underlying hardware and supporting technologies.
• Implement automation and monitoring tools with a DevSecOps mindset.
• Work with the Architecture Team to test new technology and plan the evolution of the Big Data System.
• Developing a log enrichment pipeline capable of functioning at a massive scale.
• Write code for data enrichment and normalization and deploying it to Openshift.
• Configure Kafka and other components for handling data pipeline.
• Maintain up-to-date documentation on the systems and procedures.

Required skills:

• University degree in Computer Science or five years' equivalent experience in Information Technology.
• Previous experience writing code for use at scale i.e. 'big data' - Information security-related applications e.g. SIEM a benefit.
• Proven track record of developing effective and efficient real-time big data processing pipelines.
• Experience with Hadoop, Cloudera or Hortonworks.
• Proven experience with OpenShift or Kubernetes.
• Proven Experience with Apache Spark and Kafka.
• Strong ability to develop in Python. Java and/or Scala.
• Experience with CI/CD.
• Skilled thinker, proactive, team-player, highly resourceful and detail-oriented.
• Able to track the work by following agile methodologies.
• Security certification: CISSP, CEH, GSEC, OSCP highly desirable.