LLM Inference Engineer

Hippocratic AI is seeking an LLM Inference Engineer to optimize large language model serving infrastructure. The role involves designing multi-node serving architectures, applying quantization techniques, and implementing speculative decoding. Candidates need expertise in Python, C++, CUDA, and GPU optimization.

• Design and implement multi-node serving architectures for distributed LLM inference; Optimize multi-LoRA serving systems; Apply advanced quantization techniques (FP4/FP6); Implement speculative decoding and other latency optimization strategies; Develop disaggregated serving solutions with optimized caching strategies; Continuously benchmark and improve system performance.

• Experience optimizing LLM inference systems at scale; Proven expertise with distributed serving architectures; Hands-on experience implementing quantization techniques; Strong understanding of modern inference optimization methods including speculative decoding; Proficiency in Python and C++; Experience with CUDA programming and GPU optimization.

• Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM; Experience with custom CUDA kernels; Track record of deploying inference systems in production environments; Deep understanding of performance optimization systems.