Search
Go
Freelance Agent Evaluation Engineer
Summary by AIEngineer.careers
Freelance Agent Evaluation Engineer is a LLM Evaluation Engineer role (part-time). with Mindrift. in LISBOA, PT. Imported listing (source: expertini.work). Apply on the employer's site (expertini.work).
Imported listing (source: expertini.work) · Apply on expertini.work
LISBOA · PT
—
Part-time
MID
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from expertini.work
Mindrift is seeking a Freelance Agent Evaluation Engineer to design and build datasets for evaluating AI coding agents. The role involves creating realistic simulated developer environments and crafting challenging tasks to test how well models handle real-world development scenarios.
Responsibilities
- Build realistic developer environments with codebases, infrastructure, and context
- Design tasks from intermediate states of these environments
- Craft prompts and define success criteria for AI agent solvability
- Write tests to verify agent solutions and accept valid approaches
Requirements
- Strong understanding of software development workflows
- Ability to design complex, realistic technical scenarios
- Proficiency in English (native or near-native level)
- Experience with AI systems or coding tools is advantageous
Nice to Have
- Background in AI/ML evaluation or testing
- Experience with LLM prompting techniques
- Familiarity with agent-based architectures
Tech Stack
Prompt EngineeringAI AgentsSoftware DevelopmentTesting & QA
Mindrift
Amsterdam · NL · 2779+ employees
Mindrift is an AI training and expert-sourcing platform created by Toloka. It connects domain experts and freelance contributors with technology companies to provide high-quality training data for generative AI models, including tasks like supervised fine-tuning, RLHF, model evaluation, and red-teaming.
Related Jobs
English-Speaking AI Data Evaluation Specialist | Large Language Models
Workster Jobs
LISBOA · PT
LISBOA · PT
Generative AI Analyst | Estonian
Estonian
REMOTE · WORLDWIDE
REMOTE · WORLDWIDE
Senior/Staff AI Engineer, Quality & Evals (m/f/x)
cortea
BERLIN · DE
BERLIN · DE
Staff Software Engineer, Search Evaluation and Quality Team
MOUNTAIN VIEW · US
$207K–$300K
MOUNTAIN VIEW · US
$207K–$300K
Generative AI & Data Science Engineer
AMS OSRAM
LISBOA · PT
LISBOA · PT
Senior Data Scientist, AI Evaluations Platform
En Us
TORONTO · CA
TORONTO · CA