Applied Data Scientist, LLM Evaluation

Sourced from flexgen.zya.me

Driver builds systems that turn source code into human language, providing a context layer for software development and AI agents. This role owns the LLM evaluation strategy from first principles to production infrastructure, defining quality metrics, building evaluation datasets, and creating automated benchmarking pipelines.

• Define quality metrics and build curated evaluation datasets across languages and repo archetypes.
• Build automated evaluation pipelines, LLM-as-judge tooling, and CI integration for regression detection.
• Develop automated quality signals and sampling strategies to minimize human annotation effort.
• Run experiments quantifying cost/quality/latency tradeoffs for model and pipeline changes.

• Bachelor's, Master's, or PhD in Statistics, Machine Learning, Data Science, Computational Linguistics, or a related quantitative field.
• Minimum 3-5 years in applied science, ML engineering, or data science roles with a focus on evaluation, NLP, or generative AI.
• Strong statistical foundations: experimental design, hypothesis testing, confidence intervals, effect sizes, power analysis.
• Experience designing and running evaluations for LLM or NLP systems.