Skip to main content

LLM Inference Engineer (Mid, Sr, Staff)

Summary by AIEngineer.careers

LLM Inference Engineer (Mid, Sr, Staff) is a AI Inference Engineer role (full-time). with Hippocratic AI. in MENLO PARK, CA (ONSITE), US. Imported listing (source: mikeishiring.com). Apply on the employer's site (mikeishiring.com).

Imported listing (source: mikeishiring.com) · Apply on mikeishiring.com

How our listings work

MENLO PARK, CA (ONSITE) · US
—
Full-time
SENIOR
Apply Now

More AI jobs in United States

The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.

Imported job description

Sourced from mikeishiring.com

Hippocratic AI is hiring an LLM Inference Engineer to own the serving infrastructure behind its healthcare-focused LLM platform. You will optimize distributed inference to deliver sub-100ms responses at scale, improving latency, throughput, and cost per inference. The role spans multi-node serving architectures, quantization, speculative decoding, and disaggregated serving across GPU types. This is an on-site role in Menlo Park, CA.

Responsibilities

  • Design and implement multi-node serving architectures for distributed LLM inference
  • Optimize multi-LoRA serving systems
  • Apply advanced quantization techniques (FP4/FP6) while preserving model quality
  • Implement speculative decoding and other latency optimization strategies
  • Develop disaggregated serving with optimized caching for prefill and decoding phases
  • Benchmark and improve system performance across deployment scenarios and GPU types

Requirements

  • Experience optimizing LLM inference systems at scale
  • Proven expertise with distributed serving architectures for large language models
  • Hands-on experience implementing quantization for transformer models
  • Strong understanding of speculative decoding, including Eagle approaches
  • Proficiency in Python and C++
  • Experience with CUDA programming and GPU optimization

Nice to Have

  • Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
  • Experience with custom CUDA kernels
  • Track record deploying inference systems in production
  • Deep understanding of performance optimization systems

Benefits

Equity / stock optionsHealth insurance

Tech Stack

C++CUDADistributed InferenceMulti-LoRAPythonQuantizationSGLangspeculative decodingTensorRT-LLMvLLM
Hippocratic AI

Hippocratic AI

Palo Alto · US · 150+ employees

Hippocratic AI is a healthcare technology company building a safety-focused large language model (LLM) designed specifically for healthcare applications. Its AI agents are designed to assist health systems, payors, and pharmaceutical organizations with non-diagnostic clinical tasks.

Weekly job alert

Get a weekly email when new AI engineering jobs match your filters. Free — confirm your address to start.

Related Jobs