ML Research Engineer

Sourced from career.io

Radical AI is hiring an ML Research Engineer to build the evaluation and experimentation systems behind a scientific agent that decides which experiments to run. You will partner with the Principal Scientist to turn hypotheses about agent behavior into reproducible experiments and production improvements. The role is engineering-first, ML-strong, and research-capable, spanning evaluation datasets, replay, trajectory analysis, and control policies.

• Formulate hypotheses about agent behavior and design controlled experiments with the Principal Scientist
• Build reproducible replay of historical agent runs for consistent model, prompt, and policy comparisons
• Create and maintain versioned evaluation datasets, splits, and ground truth
• Develop evaluation harnesses and metrics for decision quality, task success, error recovery, cost, and latency
• Instrument agent runs, characterize failure modes, and convert recurring failures into regression coverage
• Prototype, test, and ship control policies for planning, pivoting, escalation, and termination
• Build experiment infrastructure including data pipelines, artifact/dataset versioning, and experiment tracking
• Carry promising changes from prototype through production with platform engineers

• 4-8 years building production ML systems, research infrastructure, or data-intensive backends
• Excellent Python and strong software engineering discipline (testing, typing, packaging, code review, CI)
• Direct experience with ML evaluation and experimentation, including offline evaluation harnesses and metric design
• Data pipeline skills: schema design, versioning, orchestration (Airflow, Prefect, Dagster, Ray, or equivalent), object storage
• Strong reproducibility instincts: pinned environments, seeded runs, versioned artifacts
• Ability to reduce ambiguous system behavior to controlled experiments and communicate conclusions
• Ability to prototype quickly and harden successful ideas into reliable production systems
• Initiative and ownership in an environment with an evolving roadmap

• Experience with LLM agents: tool calling, planning loops, context management, trajectory logging, agent evaluation
• Background in sequential decision making: bandits, Bayesian optimization, reinforcement learning, or off-policy evaluation
• Experience with experiment tracking, agent observability, and artifact versioning tools (Langfuse, MLflow, Weights and Biases, DVC)
• Scientific computing or experience with materials, chemistry, or instrument data
• Experience operating large-scale inference systems, including GPU scheduling and distributed execution
• Observability experience (OpenTelemetry, Prometheus, Grafana, Datadog)
• Prior work in lab automation, robotics, or a self-driving lab