Design a GPU Inference Scheduler
Published 3 October 2026
A popular Anthropic system design question.
Problem statement¶
The system has only 8 GPUs and must serve two types of model requests.
A small-model request needs exactly one GPU and can execute independently of other small requests.
A large-model request must acquire all eight GPUs simultaneously:
A large request cannot begin with fewer than eight GPUs and cannot share any of those GPUs with another request while it is running.
Requests from both workload classes arrive continuously.
The goal is to design a scheduler that preserves these resource-isolation requirements while balancing small-model throughput against large-model latency.
Follow-Up — Gang Scheduling and Starvation Prevention¶
The interviewer first asked how the scheduler should represent the large model's requirement to obtain all eight GPUs atomically.
A major follow-up was what happens when all GPUs are continuously occupied by small-model requests while a large request is waiting.
The design needed to ensure that large requests eventually receive all eight GPUs rather than being indefinitely delayed by a constant stream of new small jobs.
The discussion included how queue state, request age, SLAs, workload pressure, and expected execution times could influence the scheduling decision.
Follow-Up — Admission Control and GPU Utilization¶
Another major area was deciding when to stop launching new small-model requests once a large request is waiting.
Stopping admission too early could cause GPUs to become idle one at a time while the scheduler waits for the remaining small jobs to finish. Continuing to admit small jobs for too long could significantly increase the large request's queueing latency.
The interviewer therefore asked how to balance:
- Avoiding starvation for large-model requests
- Maintaining high GPU utilization and small-model throughput
Predicted remaining execution time, queue depth, request SLAs, and current system load were all relevant signals to discuss.