AI SDET (Software Development Engineer in Test)

📋 Description

• Own and manage the AI evaluation harness, including the creation, maintenance, and continuous
• Develop and maintain “Golden Path” scenarios representing expected AI behavior and critical user
• Design adversarial and edge-case test scenarios to identify weaknesses in AI agents and expose
• Validate regulatory quality gates and compliance requirements, including frameworks such as SR 26-2
• Execute shadow-mode validation to compare AI system performance against established human
• Test for data quality degradation, population segmentation errors, inconsistencies, and other risks

🎯 Requirements

• Advanced proficiency in Python 3.11+, Pytest, and modern automated software testing frameworks.
• Hands-on experience with AI evaluation platforms and methodologies, including Langfuse, RAGAS, or
• Practical knowledge of LLM-as-a-judge evaluation patterns and approaches for assessing
• Understanding of model risk management principles and regulatory compliance testing for AI-driven
• Experience with shadow-mode deployment strategies, statistical validation, and comparative model
• Strong understanding of automated testing principles and the ability to design robust, repeatable

🎁 Benefits

• Health and dental insurance.
• Meal and food allowances.
• Childcare assistance.
• Extended paternity leave.
• Access to gyms and health and wellness professionals through partner programs such as Wellhub and
• Profit Sharing and Results Participation (PLR).