Principal AI Quality Engineer

Nexaminds seeks a Principal AI Quality Engineer to lead validation and quality strategy for AI-powered systems, including LLMs, RAG applications, and agent-based workflows. The role involves building automated evaluation pipelines, implementing AI guardrails, prompt regression testing, and production monitoring of AI output quality. Candidates need deep experience in quality engineering, LLM evaluation, Node.js, and CI/CD integration.

• Define and lead AI validation strategy for LLMs, RAG systems, and AI agents.
• Build automated validation pipelines integrated into CI/CD.
• Establish evaluation frameworks covering correctness, relevance, groundedness, consistency, and hallucination rate.
• Design and maintain prompt regression testing to catch silent quality degradation.
• Introduce AI guardrails and confidence scoring to flag unsafe or low-confidence outputs.
• Define and track quality metrics/KPIs for output consistency and prompt regression stability.

• 7+ years in Quality Engineering, Software Development, or related discipline, with hands-on AI/ML or LLM experience.
• Experience designing validation/evaluation frameworks for non-deterministic systems such as LLMs or ML applications.
• Strong understanding of RAG, agentic workflows, and LLM evaluation concepts (hallucination detection, groundedness, relevance, consistency).