AI Evaluation Engineer (QA)

📋 Description

• Run evals at scale across large question sets, from small human-UAT batches up to hundreds of
• Statistically measure factual grounding and accuracy lift (before/after), not just human A/B
• Build a metrics framework showing quality improvement (e.g., “answer is X% supported by source
• Design load and quality tests as the corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites; integrate them into CI/CD.

🎯 Requirements

• Bachelor’s Degree in a technical field or equivalent experience.
• 4+ years in QA / test engineering, with exposure to data/ML systems.
• Strong Python and data-science techniques for measuring factual grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.
• Ability to build automated, large-scale eval harnesses plus lighter human-in-the-loop A/B tests.
• Comfort working across multiple LLM providers’ outputs.

🎁 Benefits

• Equal Opportunity Employer
• Accommodations are available upon request throughout the recruitment process.
• Commitment to diversity, equity and inclusion in the workplace.