Senior Machine Learning Engineer, Agent Oversight

Scale AI is hiring a Senior Machine Learning Engineer on the Agent Oversight team within the Applied Intelligence Systems group. This role involves building observability tools, designing evaluation frameworks, and developing improvement loops for production agentic AI systems. You'll work with LLMs, agent architectures, and production ML to ensure reliable performance at scale.

• Build observability tools for agent behavior in production
• Design evaluation methodologies and metrics for agentic applications
• Develop ML systems for drift, anomaly, and misalignment detection
• Conduct rigorous experiments to validate performance improvements
• Collaborate with product managers, engineers, and customers to translate requirements into platform capabilities

• 5+ years as ML engineer or applied scientist on production ML/LLM systems
• Experience in at least two of: evaluation/monitoring infrastructure, agent system design, or novel methods for agent evaluation
• Hands-on experience with LLMs and agent architectures (tool use, planning, multi-agent orchestration)
• Comfortable partnering with software engineers to productionize research
• Rigorous approach to experimentation and cross-functional collaboration