Machine Learning Engineer, LLM Evals & Observability

Glean is the Work AI platform. This role is on the team owning evaluation and quality measurement for Glean's AI Assistant and Agents. You will build evaluation pipelines, curate datasets, build LLM-powered judges, and observability infrastructure. The role combines infrastructure engineering, applied ML, and direct product impact.

• Design and curate evaluation datasets with sampling strategies and golden sets.
• Build and maintain large-scale evaluation pipelines measuring assistant quality.
• Build LLM-powered judges scoring correctness, completeness, and response quality.
• Evaluate new models and product changes before shipping.
• Build observability infrastructure for AI agents including trace enrichment and dashboards.
• Close the loop between quality measurement and improvement using eval results and feedback.

• 2+ years of software engineering experience with strong coding skills.
• Strong backend fundamentals in Go and Python; comfortable with distributed data pipelines.
• Experience with LLM evaluation, reinforcement learning from human feedback, NLP, or similar large ML systems.