Search
Go
This job is no longer accepting applications
This listing was closed on Aug 19, 2026.
Browse Open JobsSenior Machine Learning Engineer, Agent Oversight
Summary by AIEngineer.careers
Senior Machine Learning Engineer, Agent Oversight is a Machine Learning Engineer role (full-time). with Scale AI. in NEW YORK, US. Compensation shown: $216K–$270K. Imported listing (source: zerogtalent.com). Apply on the employer's site (zerogtalent.com).
Imported listing (source: zerogtalent.com) · Apply on zerogtalent.com
NEW YORK · US
$216K–$270K
Full-time
SENIOR
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from zerogtalent.com
Scale AI is hiring a Senior Machine Learning Engineer on the Agent Oversight team within the Applied Intelligence Systems group. This role involves building observability tools, designing evaluation frameworks, and developing improvement loops for production agentic AI systems. You'll work with LLMs, agent architectures, and production ML to ensure reliable performance at scale.
Responsibilities
- Build observability tools for agent behavior in production
- Design evaluation methodologies and metrics for agentic applications
- Develop ML systems for drift, anomaly, and misalignment detection
- Conduct rigorous experiments to validate performance improvements
- Collaborate with product managers, engineers, and customers to translate requirements into platform capabilities
Requirements
- 5+ years as ML engineer or applied scientist on production ML/LLM systems
- Experience in at least two of: evaluation/monitoring infrastructure, agent system design, or novel methods for agent evaluation
- Hands-on experience with LLMs and agent architectures (tool use, planning, multi-agent orchestration)
- Comfortable partnering with software engineers to productionize research
Scale AI
San Francisco · US · 1500+ employees
Scale AI is a leading data platform providing high-quality training data and AI infrastructure for machine learning. The company helps enterprises and governments build, deploy, and monitor reliable AI applications.
