AI Evaluation Engineer (QA)

📋 Description

• As a QA / AI Evaluation Engineer, join a motivated team to improve answer quality through
• Run evaluations from small human-UAT batches up to millions of automated evaluations.
• Measure factual grounding and accuracy lift; build a metrics framework to show quality improvements
• Design load and quality tests as the corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites; integrate into CI/CD pipelines.

🎯 Requirements

• Bachelor’s degree or equivalent in a technical field.
• 4+ years in QA/test engineering with exposure to data/ML systems.
• Strong Python and data-science techniques for measuring grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.
• Ability to build automated large-scale eval harnesses plus lighter human-in-the-loop A/B tests.
• Comfort across multiple LLM providers’ outputs; test automation frameworks and scripting.

🎁 Benefits

• Inclusive, Equal Opportunity Employer policy; accommodations available upon request during