Search
Go
Senior/Staff AI Engineer, Quality & Evals (m/f/x)
Summary by AIEngineer.careers
Senior/Staff AI Engineer, Quality & Evals (m/f/x) is a LLM Evaluation Engineer role (full-time). with cortea. in BERLIN, DE. Imported listing (source: jobera.com). Apply on the employer's site (jobera.com).
Imported listing (source: jobera.com) · Apply on jobera.com
BERLIN · DE
—
Full-time
STAFF
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from jobera.com
Cortea is seeking a Senior/Staff AI Engineer to build the evaluation and observability infrastructure for production-grade LLM agents in the audit domain. This role focuses on backend engineering, data pipelines, and AI quality metrics to ensure reliability and performance of multimodal retrieval systems. You will design automated quality gates, analyze agent traces, and optimize LLM usage within a fast-scaling Berlin-based startup.
Responsibilities
- Build online and offline evaluation systems for LLM agents using golden datasets and human review workflows.
- Create automated quality gates to test prompts, context, models, and agent logic before production deployment.
- Analyze large volumes of agent traces in analytical databases (BigQuery, ClickHouse) to identify failure modes and optimization opportunities.
- Manage observability tools for tracing, monitoring, debugging, and experiment management.
- Collaborate with backend engineers to improve the speed and reliability of retrieval and reasoning agents.
Requirements
- Strong Python and/or backend engineering experience.
- Solid understanding of LLM and agent system evaluation methods (deterministic checks, ground truth, LLM-as-judge).
- Experience deploying and operating cloud systems, ideally on GCP.
C
cortea
Berlin · DE · 13+ employees
Cortea (AI Audit Systems GmbH) is a software company developing AI-powered audit agents for professional audit firms. Their platform provides purpose-built AI designed for financial and IT auditors to increase audit quality, consistency, and traceability.
