Data Engineer

JOB DESCRIPTION

Execution of the BCT Divisional plans and monitor performance thereof based on SLA.

Identify technology strategic opportunities to meet business needs.

Data Architecture & Infrastructure Development

1. Design, develop, and maintain data pipelines for processing large-scale datasets.
2. Build and optimize data warehouses, data lakes, and ETL (Extract, Transform, Load) processes.
3. Implement data models and database schemas to improve data efficiency.
4. Ensure data systems are scalable, secure, and cost-effective.
5. Data integration: develop ETL / ELT workflows to extract data from a variety of sources, including databases, APIs, and streaming services.
6. Automate the processes of data ingestion, transformation, and storage.
7. Troubleshoot and monitor pipeline performance and data quality issues.
8. Optimise and manage SQL and NoSQL databases, MySQL.
9. Manage cloud-based storage solutions, including MS Azure and AWS.
10. Guarantee the integrity, consistency, and accessibility of data across various platforms.
11. Ensure privacy and security by implementing data stewardship policies (POPIA).
12. Manage and supervise encryption, authentication, and access controls.
13. Formulate strategies for data retention, recovery, and backup.
14. Collaborate closely with data scientists to guarantee that the data is appropriately organized for machine learning models.
15. Provide data analysts with well-organised and clean datasets for the purpose of reporting and visualization.
16. Collaborate with software engineers to incorporate data solutions into applications.
17. Enhance data processing speed and efficiency by utilising big data technologies (e.g., Hadoop, Spark, etc).
18. For cloud environments, optimise storage costs and query performance.
19. Employ relevant scripting, SQL, or Python to automate repetitive duties.

Contribute to budget planning where required.

Respond to all data related loss notifications impacting business.

JOB REQUIREMENTS

Qualifications:

• Bachelors in computer science, Information Technology, Data Engineering, or any related field.
• Honours degree advantageous.
• Certified in Data Analytics.
• Advantageous - Cloud Professional Data Engineer, or Microsoft Azure Data Engineer.

Experience:

• Years of experience in data engineering, database management, or software development.