Search
Go
This job is no longer accepting applications
This listing was closed on Aug 13, 2026.
Browse Open JobsStaff / Senior Software Engineer (Agentic Search) - Crawler
Summary by AIEngineer.careers
Staff / Senior Software Engineer (Agentic Search) - Crawler is a Software Engineer role (full-time). with Nebius. in AMSTERDAM, NL. Imported listing (source: jobs-radar.com). Apply on the employer's site (jobs-radar.com).
Imported listing (source: jobs-radar.com) · Apply on jobs-radar.com
AMSTERDAM · NL
—
Full-time
SENIOR
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from jobs-radar.com
Nebius is building an agent-native search platform for AI systems. This role focuses on content acquisition and crawling infrastructure to discover, fetch, and refresh data at internet scale. The engineer will design distributed crawling systems and ingestion workflows to support AI agent retrieval and reasoning.
Responsibilities
- Design, implement, and operate web-scale crawling systems for acquiring content from the internet
- Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
- Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
- Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
- Ensure reliable operation of crawling infrastructure under high-throughput conditions
- Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
- Monitor resource usage, bandwidth consumption, and infrastructure cost
- Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
- Enable safe experimentation with crawling strategies and content acquisition policies
Requirements
Nebius
Amsterdam · NL · 1590+ employees
Nebius is an AI cloud company providing full-stack infrastructure for AI developers and enterprises. Its platform supports the complete AI journey, including data storage, model training, tuning, and production runtime deployment.
