Job Description
📋 Description Define and execute product strategy for Ridgeline's AI Quality Platform Build scalable LLM evaluation frameworks and automated harnesses Establish offline/online evaluation methodologies and benchmarks Roadmap AI observability for monitoring quality, latency, cost, reliability Partner with engineering/researchers for regression testing and experimentation Drive decisions using AI quality metrics and feedback 🎯 Requirements 5+ years PM experience on technical platforms or developer products Proven track record shipping AI products powered by LLMs Experience with LLM eval frameworks, quality metrics, continuous eval Deep knowledge of offline/online eval, benchmarking, regression tests Experience with LLM observability tools (Datadog LLM Observability, Langfuse/Smith, Arize, etc.) Familiar with prompt engineering, eval datasets, LLM-as-a-judge 🎁 Benefits Company Stock Plan Unlimited vacation Educational and wellness reimbursements $0 cost employee insurance plans Careers page for more perks
