Senior Machine Learning Engineer, LLM Inference Optimization

Sourced from feeny.ai

Nebius is hiring a Senior Machine Learning Engineer on its Applied AI team to own LLM and VLM inference optimization for the Nebius Token Factory. The role spans model compression, quantization, speculative decoding, KV-cache optimization, inference engines, and distributed serving architecture to improve latency, throughput, GPU utilization, and cost per token. You will benchmark and validate optimizations through reproducible harnesses and safe production rollouts.

• Own optimization projects for model families, customer endpoints, or serving backends
• Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo
• Develop production model-compression workflows including quantization, distillation, and low-bit serving
• Implement speculative decoding, KV-cache optimization, prefix caching, chunked prefill, and disaggregated prefill/decode
• Design LLM request routers and scheduling policies balancing utilization, latency, and KV-cache locality
• Scale dense and mixture-of-experts inference across multi-node GPU clusters
• Build reproducible benchmark harnesses measuring TTFT, TPOT, throughput, latency, and cost per token
• Collaborate with GPU kernel and platform engineers to trace bottlenecks across the stack

• Strong engineering skills in Python and PyTorch
• Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference
• Practical knowledge of a modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton, Ray Serve, or KServe
• Understanding of transformer inference bottlenecks: KV cache, attention, memory bandwidth, batching, parallelism
• Ability to quantify trade-offs among latency, throughput, quality, utilization, and cost
• Experience with distributed inference architecture such as request routing, distributed scheduling, PDD, or expert parallelism
• Strong cross-team communication and collaboration skills

• Quantization-aware training, post-training quantization, FP8, INT8, INT4, NVFP4, MXFP4, AWQ, GPTQ, SmoothQuant
• Distillation, speculative decoding, EAGLE, Medusa, multi-token prediction
• Supporting agentic workloads with tool calling, structured outputs, and streaming APIs
• Familiarity with CUDA or Triton
• Open-source contributions to vLLM, SGLang, TensorRT-LLM, FlashInfer, LMCache, PyTorch, Triton, Ray, or KServe
• Experience with NCCL, NVLink, Infiniband, or RoCE