AI Engineer – Agent Evaluation Specialist

Summary by AIEngineer.careers

AI Engineer – Agent Evaluation Specialist is a LLM Evaluation Engineer role (full-time). with TMS LLC. in LOS ANGELES, US. Imported listing (source: jobs.smartrecruiters.com). Apply on the employer's site (jobs.smartrecruiters.com).

Imported listing (source: jobs.smartrecruiters.com) · Apply on jobs.smartrecruiters.com

How our listings work

LOS ANGELES · US

—

Full-time

STAFF

Apply Now

More AI jobs in United States

The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.

Imported job description

Sourced from jobs.smartrecruiters.com

Hands-on role focused on evaluating LLM-powered agents. The engineer designs LLM-as-a-judge frameworks, maintains golden datasets, and monitors agent performance while troubleshooting failures using Langfuse. Requires 14+ years of experience.

Responsibilities

Design and implement LLM-as-a-judge evaluation frameworks
Maintain and curate golden datasets for agent evaluation
Monitor agent performance and troubleshoot failures using Langfuse
Collaborate with engineering teams to improve agent reliability

Requirements

14+ years of professional software or AI engineering experience
Hands-on experience evaluating LLM-powered agents
Experience with LLM-as-a-judge frameworks and golden datasets
Familiarity with Langfuse or similar observability tooling

Tech Stack

Agent EvaluationLangfuseLLMLLM-as-a-Judge

TMS LLC

Parsippany-Troy Hills · US · 65+ employees

Technology Management Solutions (TMS, LLC), headquartered in Parsippany, New Jersey, is a premier IT company specializing in digital product engineering and innovative technology solutions. Since its inception, TMS has empowered businesses across industries—Financial Services, Retail, Medical, Entertainment, Manufacturing, and Automotive—to achieve their digital transformation goals.