Big Data Architect

We are headhunting for an AiOps Architect and I would like to run the role past you to see if you have any interest or could recommend someone to me.

Our client is a mid-sized leader in AiOps Telecoms software products related to network management/service assurance.

The role involves designing and supporting the implementation of our client's Cloud Native Big Data Analytics infrastructure layer. This infrastructure serves as the backbone for supporting AI/ML workflows, enabling both real-time and batch processing for training and inference purposes. The typical data throughput to support is in the range of continuous streams of Terabytes of data per hour.

We are looking for someone with the following background:

1. Software development background, with major experience in:
1. Back-end data processing
2. Data lakehouse
2. Hands-on experience of Big Data open-source technologies such as:
1. Apache Airflow
2. Apache Kafka
3. Apache Pekko
4. Apache Spark & Spark Structured Streaming
5. Delta Lake
6. AWS Athena
7. Trino
8. MongoDB
9. AWS S3
10. MinIO S3
3. Proven successful hands-on experience of:
1. Setting up data governance tooling and processes (schema registry, data lineage control) and data access control
2. Setting up data pipelines for model training and inference
3. Kubernetes or Openshift in the context of Big Data analytics
4. AWS services

In the context of both on-premise and SaaS Telco Service Assurance product development, your role is to:

1. Lead the general vision for a coherent and future-proof Big Data Analytics for AI/ML framework.
2. Be an innovation force to leverage GenAI technologies to facilitate and accelerate the delivery of the big data jobs.
3. Build and validate the high-level design of the necessary components supporting the Roadmap requirements, including Test Strategy and Performance and Scalability validation.
4. Build the Project Design (work packages, skillsets, efforts, dependencies, resourcing) accompanying the architecture for every project iteration.
5. Communicate the architecture to the development teams and accompany the development process through periodic detailed reviews, ensuring architecture is understood and respected.
6. Produce quarterly Technology Watch reports on the latest Big Data Analytics landscape.
7. Setup rigorous selection process and support selection of appropriate 3rd-party components, supporting the Big Data Analytics stack.
8. Educate/evangelize the development teams on Analytics design best practices.
9. Review low-level designs and critical code changes produced by the development teams.
10. Support all Big Data Analytics-related technical questions in the context of RFI/RFQ responses.
11. Provide feasibility study / high-level costings in response to 3 to 5 year roadmap high-level strategy.