Large Language Model Evaluation Engineer

About PatSnap

Patsnap empowers IP and R&D teams by providing better answers, so they can make faster decisions with more confidence. Founded in 2007, Patsnap is the global leader in AI-powered IP and R&D intelligence. Our domain-specific LLM, trained on our extensive proprietary innovation data, coupled with Hiro, our AI assistant, delivers actionable insights that increase productivity for IP tasks by 75% and reduce R&D wastage by 25%. IP and R&D teams collaborate better with a user-friendly platform across the entire innovation lifecycle. Over 15,000 companies trust Patsnap to innovate faster with AI, including NASA, Tesla, PayPal, Sanofi, Dow Chemical, and Wilson Sonsini.

About the Role

We are seeking a highly skilled and innovative Large Model Evaluation and Development Engineer to join our cutting-edge AI team. In this role, you will be responsible for designing and implementing evaluation frameworks and methodologies for large-scale models, ensuring they meet the highest quality standards. You will collaborate closely with AI researchers, data scientists, and product managers to drive advancements in AI technologies. This position offers the opportunity to work on groundbreaking projects that push the boundaries of AI and machine learning.

Key Responsibilities

• Design and implement comprehensive evaluation frameworks for large-scale AI models, including language models and vision models.
• Develop and maintain high-quality datasets for model evaluation and benchmarking.
• Create and apply metrics to assess model performance, robustness, and quality.
• Collaborate with cross-functional teams to integrate evaluation results into the development cycle, providing actionable insights for model improvement.
• Develop tools and platforms for automated model evaluation and reporting.
• Stay updated with the latest research and trends in AI model evaluation and incorporate best practices into our processes.
• Improve the efficiency of the evaluation process, including data collection and annotation strategies.

Desired Qualifications

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, or a related field.
• Proficiency in programming languages such as Python and database languages like SQL, familiarity with Linux operating systems, and strong programming skills.
• Experience with data manipulation and analysis using libraries such as NumPy and Pandas.
• In-depth understanding and application of common language or vision model evaluation metrics such as Cross-entropy, BPC/BPW, Perplexity, and ROUGE.
• Deep understanding of Bert and GPT-related principles, including key technologies like prompt, finetuning, and transformer.
• Familiarity with large model architectures and training processes.
• Strong insight into the development of large model technologies and the ability to learn and practice.
• Excellent problem-solving and communication skills, with the ability to work collaboratively in a team environment.

Why Join Us

• Work at the forefront of AI technology and participate in pioneering projects.
• Participate in the development of cutting-edge large model evaluation platforms.
• Opportunities for professional growth and continuous learning.
• Engage in regular team events and knowledge-sharing sessions.