Software Engineer (SC Cleared)- Azure Data & DevOps

Scrumconnect Consulting; Greater London; On-site; GBP 45,000 - 75,000

Job description

Scrumconnect Consulting is looking for a Software Engineer to work on a Strategic Data Platform with expertise in Azure Data Factory (ADF), Python, PySpark, Java, Terraform, and Azure DevOps. The ideal candidate will have experience in cloud-based data engineering, automation, and infrastructure provisioning while working in an Azure environment.

Key Responsibilities:

1. Develop, test, and deploy scalable data pipelines using Azure Data Factory (ADF), Python, PySpark, and Java.
2. Implement data transformation, ETL/ELT workflows, and data integration solutions.
3. Optimize data flow and performance for cloud-based data processing.
4. Use Terraform to provision and manage Azure resources.
5. Implement infrastructure as code (IaC) best practices for automated cloud deployments.
6. Ensure efficient resource scaling and cost optimization.

DevOps & CI/CD Automation:

1. Collaborate with DevOps teams to build automated CI/CD pipelines in Azure DevOps.
2. Deploy and manage containerized applications using Docker & Kubernetes.
3. Monitor and troubleshoot build, deployment, and infrastructure issues.

Performance Optimization & Security:

1. Optimize data pipeline performance in Azure.
2. Implement cloud security best practices and ensure compliance with data governance policies.
3. Troubleshoot data and infrastructure-related performance issues.
4. Work closely with data engineers, cloud architects, and DevOps teams to design solutions.
5. Participate in agile ceremonies, sprint planning, and technical discussions.
6. Maintain technical documentation and best practices.

Required Skills & Experience:

1. Strong experience in: Python, PySpark, Java, Azure Data Factory (ADF).
2. Hands-on experience with Azure Data Lake, Synapse, and Azure Functions.
3. Experience in cloud infrastructure automation and DevOps workflows.
4. Experience with Azure DevOps, Git, YAML pipelines.
5. Ability to work with Docker, Kubernetes, and containerized applications.
6. Strong problem-solving and debugging skills.
7. Experience working in Agile/Scrum environments.
8. Excellent communication and collaboration skills.

Nice to Have:

1. Experience with Databricks, Apache Spark, or ML workloads.
2. Knowledge of security best practices in cloud environments.
3. Azure, Terraform, or DevOps-related certifications.

Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Software Development