Senior Machine Learning Engineer, Agentic Evals and Self-Improvement (Nova)

Sourced from launchyournextchapter.com

Iterable is hiring a Senior Machine Learning Engineer to lead evaluation and automatic self-improvement for Nova, its AI customer engagement intelligence layer. You will build evaluation systems for models and agents, diagnose failures, and turn findings into automated improvements to prompts, context, tools, and agent harnesses. The role spans research through production, with autonomy to shape AI direction at Iterable.

• Build evaluation systems for models and agents: benchmarks, LLM judges, simulations, trajectory analysis, and regression tests
• Create a loop that takes failures from diagnosis to fix to measured improvement
• Turn evaluation results into automatic optimization of prompts, context, tools, rubrics, and agent harnesses
• Build synthetic data pipelines for evaluation and post-training with attention to quality, privacy, and reproducibility
• Apply recent research (LLM-as-judge, reward modeling, test-time search, post-training) to production
• Mentor engineers and help set technical direction for Nova

• 5+ years of hands-on MLE experience putting complex ML systems into production at scale
• Master's or PhD in Computer Science, Machine Learning, AI, or a related field
• Strong Python skills with emphasis on readable, modular, maintainable code and modern ML frameworks
• Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning
• Experience building automated evaluation such as LLM judges, reward models, or benchmark infrastructure
• Experience with at least one of: prompt/context optimization, post-training, preference optimization, or reinforcement learning
• Strong communication and cross-team collaboration

• Experience with LLM APIs from multiple providers (Anthropic, OpenAI, Google), including tool calling and prompt caching
• Experience with automatic prompt optimization and error analysis