San FranciscoFull TimeSoftware Development
Remotely
productpythonsqlllmpartnershipsaidataevaluationbenchmarks
Job Description
📋 Description
- Prioritize the Benchmark Portfolio: Own intake and prioritization of new benchmarks and
- Support Dataset Demand Generation: Run leaderboard evals to drive research brand equity during
- Manage Training and Inference Partners & Budget: Own relationships with inference providers
- Track Customer Delivery: Maintain view across active evals, loss analysis, and post-training
- Shape the Product: Translate delivery friction into product requirements and work with Engineering
🎯 Requirements
- Experience: 3+ years in Product Operations/Management/Technical PM or similar technical
- Technical Fluency: LLM evaluation concepts; post-training and inference economics; proficient in
- Vendor and Capacity Judgment: Plan capacity against demand, manage SLAs, balance cost and
- Ownership: Able to operate in ambiguity and own outcomes from definition to execution.
- Communication: Strong written and verbal skills; able to brief executives.
- Systems Thinking: Strong PM and cross-functional coordination; build repeatable systems.
🎁 Benefits
- Bi-annual performance bonus structure
- Generous equity grant vested over 4 years
- Up to $15k relocation bonus
- $10K housing bonus (0.5 miles from office)
- $1.5K monthly meals stipend
- Free Equinox membership
Back to all jobs