Search
Go
This job is no longer accepting applications
This listing was closed on Jul 19, 2026.
Browse Open JobsSenior ML Engineer (Inference Serving)
OUOHO US
Summary by AIEngineer.careers
Senior ML Engineer (Inference Serving) is a AI Infrastructure Engineer role (full-time). with OHO US. in SAN FRANCISCO, US. Compensation shown: $200K–$275K. Imported listing (source: oho.us). Apply on the employer's site (oho.us).
Imported listing (source: oho.us) · Apply on oho.us
SAN FRANCISCO · US
$200K–$275K
Full-time
SENIOR
The sections below reproduce the third-party job description for reference. AIEngineer.careers does not write or control this text.
Imported job description
Sourced from oho.us
Join a stealth AI hardware startup building a custom AI SoC and inference serving stack. As a Senior ML Engineer, you will architect high-performance multi-node inference stacks and implement optimizations in frameworks like vLLM and PyTorch. You will work on advanced cluster scheduling and contribute to open-source AI infrastructure.
Responsibilities
- Serve as a core contributor on a small, senior team building state-of-the-art inference serving and cluster scheduling capabilities for a custom AI SoC.
- Architect high-performance multi-node inference stacks, designing and tuning throughput and latency from the ground up.
- Implement advanced optimisation strategies across TP/PP/EP hybrids, continuous batching, and KV cache management.
- Drive performance improvements inside leading inference frameworks including vLLM, SGLang, and PyTorch.
- Develop advanced cluster scheduling algorithms for large-scale open-source models.
- Engage with the open-source community, upstreaming optimisations and influencing AI infrastructure projects.
- Apply best practices in performance benchmarking, testing, and debugging for a production-grade stack on novel silicon.
Requirements
- Strong Python, C++ and PyTorch engineering fundamentals with a track record of shipping high-quality software.
OU
OHO US
Boston · US · 65+ employees
There are two distinct companies known as OHO: OHO US is a specialized recruitment firm for tech and engineering roles, while OHO is a digital marketing and strategy agency with over 25 years of experience.
