-
How fast can LLM inference go?
Comparing InferenceMAX to the hardware limits
-
Control Layer Benchmarking
Benchmarking the Doubleword Control Layer
-
The Doubleword Control Layer
We're releasing our AI Gateway. Why are AI gateways hard to do right, and why do we think we've done it right.
-
LLM guided scheduling
An idea on how to use LLMs to help with scheduling
-
Paged attention
A discussion of paged attention
-
Using caching for fast speculative decoding
Speculative decoding speeds up LLM inference, but using another model works poorly.
-
Scheduling in inference engines
A look into how inference engines choose which requests to process