LLM Evaluation Engineer

ThirdLaw is building the control layer for AI in the enterprise. This role focuses on building the evaluation layer that decides whether LLM prompts and outputs are acceptable, using real-time guardrails and semantic judgment systems. You'll work with foundation models, vector search, and rule engines to enforce enterprise policies.

• Design and build real-time evaluation logic for LLM prompts and outputs.
• Implement evaluation strategies using semantic similarity, foundation model scoring, and rule-based systems.
• Integrate model outputs with downstream enforcement actions (redaction, escalation, blocking).
• Prototype, tune, and productize small language models and prompt templates for classification.
• Collaborate with data infrastructure engineers to connect evaluation logic with storage layers.
• Build tools to observe and debug evaluator performance across real-world data distributions.

• 7+ years of experience in ML systems or AI engineering, with 1-2 years working directly with LLMs.
• Deep understanding of foundation models (OpenAI, Claude, Mistral, Llama) via APIs or open source.
• Hands-on experience with vector search (FAISS, Qdrant, Weaviate) and embeddings pipelines.
• Proven ability to implement real-time evaluation logic using semantic similarity, classifiers, or structured rules.