Data Engineer - AI Chatbots

We're seeking a talented and passionate Data Engineer to join our team and revolutionize how parents and students interact with products, offerings, events, academic performance, and recommendations. In this role, you'll leverage your expertise in data pipelines, Azure cloud, and data management to ensure our chatbot has the data it needs to learn, adapt, and deliver exceptional user experiences.

What You'll Do:

• Design, develop, and maintain data pipelines for ingesting structured and unstructured data required for AI chatbot development and functionality.
• Work with stakeholders to identify and source data from various sources (internal systems, external APIs).
• Clean, pre-process, and transform data to enable chatbot functionality.
• Develop and implement data quality checks and monitoring pipelines to ensure data integrity.
• Collaborate with Cloud and Digital teams, Gen AI Engineers, Project Managers, and Business stakeholders to understand AI-based chatbot data requirements.
• Utilize Azure cloud platform for data storage, processing, and analytics.
• Document data pipelines and processes for maintainability and knowledge sharing.

You're a Perfect Fit If You Have:

• A Master's degree in Computer Science, Information Systems, or a related engineering field.
• 5+ years of experience as a Data Engineer or similar role.
• Strong experience in developing and managing data pipelines for large-scale data processing enabling AI development.
• Experience with data modeling techniques and NoSQL DB (Cosmos DB for MongoDB, MongoDB, etc.).
• Familiarity with Large Language Models (LLMs) and their training data requirements.
• Proficiency in programming languages like Python, SQL, and Spark.
• Experience working with Azure Cloud for data storage and processing (Databricks, ADF, Azure Functions, etc.).
• Experience working with Developer tools such as Azure DevOps, PyCharm/VS Code, etc.
• Solid understanding of data quality principles and best practices.
• Experience with DevOps tools and methodologies (e.g., Git, CI/CD).
• Experience working with structured and unstructured data processing methods.
• Excellent communication and collaboration skills to work effectively with cross-functional teams.

Nice-to-Have Skills:

• Experience working on AI-based chatbots or conversational AI systems data requirements and pipelines.
• Familiarity with Large Language Models (LLMs) and their data requirements.
• Experience with data governance frameworks and data security best practices.
• Experience with containerization technologies (e.g., Docker, Kubernetes) and serverless architectures.
• Knowledge on Data Security, Data Access Management including Data Encryption.
• Experience with RESTful APIs, message brokers, and event-driven architectures.