Principal Data Scientist - Agent Builder

Elastic is building a conversational and agentic platform for searching data. The Principal Data Scientist will define evaluation strategies for RAG and agent systems, influence product direction, and mentor teams. This is an applied leadership role working on cutting-edge AI/ML problems.

• Define evaluation strategy for conversational/agentic search, including offline/online evaluation, golden datasets, LLM-as-judge, A/B testing.
• Lead quality metrics and decision frameworks for RAG, agents, model selection, routing.
• Improve retrieval and re-ranking approaches.
• Partner with engineering to productionize evaluation pipelines.
• Influence roadmap and mentor data scientists.

• 8+ years of applied DS/ML experience with deep expertise in IR, NLP, ranking, semantic search, RAG or LLM products.
• Strong track record in evaluation for production AI/ML systems.
• Experience influencing product strategy through data.
• Hands-on with Python, PyTorch, Transformers, Pandas.
• Understanding of retrieval systems and metrics (nDCG, MRR, Recall@k).
• Experience with Elasticsearch or similar distributed data systems.