Senior Data Engineer

Role: Senior Data Engineer

Location: Abu Dhabi, UAE

Company: AI71

About Us:

AI71, a pioneering AI company launched by Abu Dhabi's Advanced Technology Research Council (ATRC) and VentureOne, stands as a pivotal movement in the realm of AI innovation. Leveraging the globally top-ranked Falcon AI models from the Technology Innovation Institute, AI71's focus spans across multi-domain advancements, initially targeting the medical, education, and legal sectors. We specialize in turning innovation into impact. If you're ready to lead a world-class software engineering team at the intersection of AI and venture building, then AI71 is the place for you.

About the Role:

As a Data Engineer in our organization, you will be instrumental in developing scalable, robust, and efficient data pipelines to support the development of enterprise-grade generative AI products and intelligent agents. You will work closely with cross-functional teams to process and structure data that fuels our AI models, ensuring seamless integration and delivery of insights to enterprise clients.

Key Responsibilities:

1. Data Infrastructure Development:
   • Design, implement, and maintain scalable data pipelines tailored for training and fine-tuning generative AI models.
   • Build and optimize data architectures for extracting, transforming, and loading (ETL/ELT) processes, specifically for high-dimensional and unstructured datasets.
2. Data Integration and Management:
   • Integrate enterprise data from diverse sources, including APIs, cloud-based storage, and proprietary databases, ensuring compliance with security and privacy standards.
   • Develop and maintain large-scale data warehouses and data lakes to support model training and analytics.
3. Collaboration:
   • Partner with data scientists and AI researchers to understand data requirements for model development and evaluation.
   • Work closely with product managers and engineers to ensure data solutions align with business and product goals.
4. Performance and Optimization:
   • Monitor and optimize data workflows to handle large-scale, real-time data streams used by AI agents.
   • Ensure data systems are scalable, secure, and cost-effective for enterprise-grade applications.
5. Data Quality and Compliance:
   • Implement systems to ensure the quality, accuracy, and consistency of training data.
   • Ensure compliance with enterprise data governance policies, privacy regulations (e.g., GDPR, CCPA), and AI ethics guidelines, based on provided guidelines.

Required Qualifications:

• Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.
• Proven experience in building and maintaining large-scale data pipelines and architectures.
• Proficiency with programming languages such as Python, Java, or Scala.
• Expertise in SQL and experience with distributed data processing tools like Apache Spark, Hadoop, or Kafka.
• Familiarity with cloud platforms (AWS, GCP, or Azure) and their data-related services (e.g., S3, BigQuery, or Azure Data Lake).
• Experience working with unstructured data such as text, images, or videos.

Preferred Qualifications:

• Knowledge of MLOps and AI lifecycle workflows is a plus.
• Experience with generative AI technologies and frameworks (e.g., Hugging Face, OpenAI API).
• Familiarity with enterprise data tools like Snowflake or Databricks.
• Understanding of natural language processing (NLP) concepts and techniques.

Why AI71:

• Proven performance of our large language models (https://falconllm.tii.ae/)
• Strong traction and adoption from the open-source community
• Secured proprietary data to build specialized distinctive models.
• Locked large compute power to support our roadmap.
• Signed anchor clients, to develop POCs and demonstrate our solutions.