Job Description
📋 Description Own the quality bar for Harvey’s human evaluations and ensure outputs are decision-grade for Author and maintain evaluation guidelines, benchmarks, and gold references used by contract Standardize rubric and evaluation design into repeatable templates with documented methodology Own onboarding, calibration training, and inter-rater reliability for a distributed pool of raters Define and maintain failure-mode taxonomy to translate eval results into actionable signals Run QA on vendor and attorney deliverables before launch decisions 🎯 Requirements 6+ years in product/ research/ evaluation operations or quality program management Experience owning quality inputs (guidelines, benchmarks, QA procedures) for expert-driven work Experience onboarding and calibrating distributed expert raters and running feedback loops Strong grounding in calibration, inter-rater reliability, sampling, rubric design Comfort interpreting evaluation data with or without AI support Proven ability to scale quality processes under shipping pressure and maintain documentation 🎁 Benefits Compensation: $155,400 - $233,200 USD Equity and bonus opportunities Applicant privacy notices provided by company
