Member of Technical Staff, Inference Infrastructure

About Us

We are a small team of researchers and engineers focused on pushing the frontier of what agents are capable of. We believe the next leap in agent performance will come not from bigger general models, but from specialization and continual learning: agents that get better at their specific job the more they do it. We build the systems that make this possible, turning real-world usage into lasting improvement.

Summary

We are looking for a senior engineer to design and own the systems that serve our models in production. Our customers run many specialized models that are updated often as they learn from real-world use. You will build inference infrastructure that is fast, reliable, and cost-efficient, and that makes it easy to deploy new model versions safely. You will also work closely with the RL team, since fast inference is at the heart of RL training. You will have a large influence on our technical direction.

Responsibilities

• Design, build, and operate our production inference platform for serving many specialized models at scale.
• Improve latency, throughput, and GPU utilization using techniques such as batching, caching, scheduling, quantization, and speculative decoding.
• Build systems to serve many fine-tuned model variants efficiently on shared hardware.
• Build safe, fast rollout systems for new model versions, including canary releases, A/B tests, and quick rollbacks.
• Support deployments in customer cloud environments and VPCs, with strong security and isolation.
• Work with the RL infrastructure team to make inference fast and efficient for rollout generation during training.
• Capture production traces and feedback in a way that can feed back into training.
• Build monitoring, alerting, and on-call practices that keep the platform reliable.
• Evaluate and adopt new models, hardware, and inference engines as they appear.
• Set engineering standards and mentor other engineers as the team grows.

Qualifications

• Experience building or operating ML inference systems in production.
• Deep experience with LLM serving, including KV caching, continuous batching, request scheduling, and parallelism strategies.
• Hands-on experience with inference engines such as vLLM, SGLang, or TensorRT-LLM, ideally including working on their internals.
• Strong Python skills. C++, Rust, or Go is a plus.
• Experience with Kubernetes, autoscaling, load balancing, and running GPU fleets across cloud providers.
• Strong understanding of transformer architectures and what drives inference performance.
• A track record of owning critical production systems, including incident response.
• Comfortable with ambiguity and able to own large problems end to end.

Nice to Have

• Experience building inference infrastructure at a frontier lab or AI infrastructure company.
• Contributions to open-source inference engines.
• Experience with GPU kernel programming (CUDA, Triton) and profiling.
• Experience with multi-LoRA serving, disaggregated prefill and decode, or Mixture of Experts models.
• Experience integrating inference with RL training systems.