Search
Go
Human Evaluation Researcher
Summary by AIEngineer.careers
Human Evaluation Researcher is a AI Evaluation Researcher role (full-time). with Nuance Labs. in WASHINGTON, US. Compensation shown: $160K–$190K. Imported listing (source: zerogtalent.com). Apply on the employer's site (zerogtalent.com).
Imported listing (source: zerogtalent.com) · Apply on zerogtalent.com
WASHINGTON · US
$160K–$190K
Full-time
SENIOR
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from zerogtalent.com
Nuance Labs is seeking a Senior AI Evaluation Researcher to design and run qualitative and quantitative studies for photorealistic, real-time AI avatars. You will develop robust evaluation frameworks, calibrate automated metrics like LLM-as-judge against human judgment, and build scalable human-evaluation pipelines to guide model training and product decisions. This hands-on IC role combines experimental psychology, UX research, and ML evaluation to solve ambiguous challenges in emotional resonance and conversational naturalness.
Responsibilities
- Design and conduct qualitative/quantitative studies including side-by-side comparisons, interviews, and longitudinal panels
- Develop rubrics, anchored scales, and annotation guidelines to align human raters on ambiguous judgments
- Build and maintain human-evaluation pipelines, participant panels, and rater calibration processes
- Calibrate automated and LLM-as-judge metrics against ground-truth human feedback
- Translate research findings into actionable insights for modeling and training teams
Requirements
- 5+ years designing and running human-subjects research in industry or academia (UX, HCI, psychology, behavioral science)
- Strong grounding in qualitative methods (ethnography, contextual inquiry) and quantitative methods (experimental design, psychometrics)
Nuance Labs
Seattle · US · 14+ employees
Nuance Labs is an AI research company focused on building a human foundation model that incorporates emotional intelligence. The company aims to create visual conversational AI that can understand and respond in real-time to nuances such as tone, facial expressions, and body language.
