Staff / Senior Software Engineer (Agentic Search) - Crawler

Nebius is building an agent-native search platform for AI systems. This role focuses on content acquisition and crawling infrastructure to discover, fetch, and refresh data at internet scale. The engineer will design distributed crawling systems and ingestion workflows to support AI agent retrieval and reasoning.

• Design, implement, and operate web-scale crawling systems for acquiring content from the internet
• Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
• Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
• Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
• Ensure reliable operation of crawling infrastructure under high-throughput conditions
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
• Monitor resource usage, bandwidth consumption, and infrastructure cost
• Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
• Enable safe experimentation with crawling strategies and content acquisition policies

• 5+ years of experience building backend or distributed systems