Search
Go
This job is no longer accepting applications
This listing was closed on Jul 17, 2026.
Browse Open JobsMachine Learning Engineer, LLM Evals & Observability
Summary by AIEngineer.careers
Machine Learning Engineer, LLM Evals & Observability is a AI Engineer role (full-time). with Glean Technologies. in SAN FRANCISCO, US. Imported listing (source: glassdoor.com). Apply on the employer's site (glassdoor.com).
Imported listing (source: glassdoor.com) · Apply on glassdoor.com
SAN FRANCISCO · US
—
Full-time
MID
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from glassdoor.com
Glean is the Work AI platform. This role is on the team owning evaluation and quality measurement for Glean's AI Assistant and Agents. You will build evaluation pipelines, curate datasets, build LLM-powered judges, and observability infrastructure. The role combines infrastructure engineering, applied ML, and direct product impact.
Responsibilities
- Design and curate evaluation datasets with sampling strategies and golden sets.
- Build and maintain large-scale evaluation pipelines measuring assistant quality.
- Build LLM-powered judges scoring correctness, completeness, and response quality.
- Evaluate new models and product changes before shipping.
- Build observability infrastructure for AI agents including trace enrichment and dashboards.
- Close the loop between quality measurement and improvement using eval results and feedback.
Requirements
- 2+ years of software engineering experience with strong coding skills.
- Strong backend fundamentals in Go and Python; comfortable with distributed data pipelines.
- Experience with LLM evaluation, reinforcement learning from human feedback, NLP, or similar large ML systems.
Tech Stack
Glean Technologies
Palo Alto · US · 300+ employees
Glean is an AI platform for enterprises that connects company knowledge to power search, assistants, and agents. It is designed to index data and learn internal processes to automate work securely across teams.
