LLM inference thoughtshttps://fergusfinn.com/https://fergusfinn.com/blog/adaptive-speculation/https://fergusfinn.com/blog/adaptive-speculation/A follow-on to the economics of speculative decoding, we run the inference lab simulator on MTP & DFlash drafters with real acceptance data, and find out whether adaptively choosing the draft length is worth it. Mon, 22 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/adaptive-speculation/mdhttps://fergusfinn.com/blog/control-layer-benchmarking/https://fergusfinn.com/blog/control-layer-benchmarking/Benchmarking the Doubleword Control Layer Tue, 21 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/control-layer-benchmarking/mdhttps://fergusfinn.com/blog/control-layer/https://fergusfinn.com/blog/control-layer/We're releasing our AI Gateway. Why are AI gateways hard to do right, and why do we think we've done it right. Tue, 21 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/control-layer/mdhttps://fergusfinn.com/blog/economics-of-speculative-decoding/https://fergusfinn.com/blog/economics-of-speculative-decoding/Two underexplored axes: what MoE routing does to the decode roofline, and how compressed attention takes away the slack that used to make speculated tokens free. Mon, 08 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/economics-of-speculative-decoding/mdhttps://fergusfinn.com/blog/deepseek-v4-flash-mi300x/https://fergusfinn.com/blog/deepseek-v4-flash-mi300x/A story of sharp edges, segfaults, and standardsMon, 01 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/deepseek-v4-flash-mi300x/mdhttps://fergusfinn.com/blog/fast-sglang-starts/https://fergusfinn.com/blog/fast-sglang-starts/Checkpoint/restore with CRIU and cuda-checkpoint, from 12 minutes to 10 seconds on a B200. Mon, 06 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/fast-sglang-starts/mdhttps://fergusfinn.com/blog/faster-than-speed-of-light/https://fergusfinn.com/blog/faster-than-speed-of-light/The fastest a memory bound kernel can go is set by the time required to transfer the data to the SMs. How can we do better? Tue, 26 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/faster-than-speed-of-light/mdhttps://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/https://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/How expert-parallel dispatch and combine kernels work, built up from scratch: the high-throughput shape and the low-latency one. Wed, 10 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/mdhttps://fergusfinn.com/blog/infiniband-roce-rdma/https://fergusfinn.com/blog/infiniband-roce-rdma/How a technically superior but economically isolated solution slowly lost ground to a good-enough one built on infrastructure everyone already owns. Fri, 19 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/infiniband-roce-rdma/mdhttps://fergusfinn.com/blog/kv-entropy-coder/https://fergusfinn.com/blog/kv-entropy-coder/Lossless compression of a target model's KV cache by up to 4×, using a cheaper predictor model to drive an arithmetic coder. Fri, 08 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/kv-entropy-coder/mdhttps://fergusfinn.com/blog/llm-guided-scheduling/https://fergusfinn.com/blog/llm-guided-scheduling/An idea on how to use LLMs to help with scheduling Tue, 23 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/llm-guided-scheduling/mdhttps://fergusfinn.com/blog/inference-arithmetic/https://fergusfinn.com/blog/inference-arithmetic/Comparing InferenceMAX to the hardware limits Wed, 22 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/inference-arithmetic/mdhttps://fergusfinn.com/blog/paged-attention/https://fergusfinn.com/blog/paged-attention/A discussion of paged attention Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/paged-attention/mdhttps://fergusfinn.com/blog/nvlink-scale-up/https://fergusfinn.com/blog/nvlink-scale-up/Why scale-up links are fast and short, how NVSwitch grew the domain from a board to a rack, and what TPU, UALink, and scale-up Ethernet do differently. Tue, 21 Jul 2026 00:00:00 GMThttps://fergusfinn.com/blog/nvlink-scale-up/mdhttps://fergusfinn.com/blog/llm-powered-content-discovery/https://fergusfinn.com/blog/llm-powered-content-discovery/Building a content discovery system using parallel primitives and BST-based ranking with LLM comparisons Fri, 16 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/llm-powered-content-discovery/mdhttps://fergusfinn.com/blog/scheduling-in-inference-engines/https://fergusfinn.com/blog/scheduling-in-inference-engines/A look into how inference engines choose which requests to process Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/scheduling-in-inference-engines/mdhttps://fergusfinn.com/blog/spacelike-speculative-decoding/https://fergusfinn.com/blog/spacelike-speculative-decoding/Speculative decoding speeds up LLM inference, but using another model works poorly. Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/spacelike-speculative-decoding/mdhttps://fergusfinn.com/blog/speculating-on-the-margin/https://fergusfinn.com/blog/speculating-on-the-margin/Some thinking about how to trade off batching and speculative decoding in a running inference engine. Thu, 02 Jul 2026 00:00:00 GMThttps://fergusfinn.com/blog/speculating-on-the-margin/mdhttps://fergusfinn.com/blog/parallel-primitives-blog/https://fergusfinn.com/blog/parallel-primitives-blog/Exploring coordination patterns from parallel computing for multi-agent LLM systems Wed, 31 Dec 2025 00:00:00 GMThttps://fergusfinn.com/blog/parallel-primitives-blog/mdhttps://fergusfinn.com/blog/bst-expensive-comparisons/https://fergusfinn.com/blog/bst-expensive-comparisons/A lock-free binary search tree optimized for expensive async comparisons, with threaded linked list for O(1) sorted iteration Tue, 13 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/bst-expensive-comparisons/mdhttps://fergusfinn.com/blog/arxiv-llm-search/https://fergusfinn.com/blog/arxiv-llm-search/Applying parallel primitives to search and rank 2.4 million arXiv papers using LLM judgments Thu, 01 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/arxiv-llm-search/mdhttps://fergusfinn.com/blog/uccl-ep-without-owning-the-nic/https://fergusfinn.com/blog/uccl-ep-without-owning-the-nic/How UCCL reimplements the DeepEP kernels for arbitrary hardware by swapping the transport out from under them. Fri, 12 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/uccl-ep-without-owning-the-nic/mdhttps://fergusfinn.com/blog/understanding-rans/https://fergusfinn.com/blog/understanding-rans/An intro to rANS: an entropy coding method for losslessly encoding & decoding streams of bytes quickly. Tue, 21 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/understanding-rans/mdhttps://fergusfinn.com/blog/throughputmaxxing-v4-flash-single-node/https://fergusfinn.com/blog/throughputmaxxing-v4-flash-single-node/Single-node serving throughput for DeepSeek-V4-Flash: baseline, MLA/DP-attention, and MoE kernel choice.Thu, 23 Jul 2026 00:00:00 GMThttps://fergusfinn.com/blog/throughputmaxxing-v4-flash-single-node/mdhttps://fergusfinn.com/blog/understanding-tans/https://fergusfinn.com/blog/understanding-tans/An intro to tANS: a table-based entropy coder that removes rANS's per-symbol division while keeping Shannon-optimal compression. Mon, 27 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/understanding-tans/mdhttps://fergusfinn.com/blog/weight-entropy/https://fergusfinn.com/blog/weight-entropy/An empirical investigation into the byte-level entropy of model weights across numeric formats and model families. Tue, 05 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/weight-entropy/mdhttps://fergusfinn.com/blog/what-happens-when-a-gpu-reads-memory/https://fergusfinn.com/blog/what-happens-when-a-gpu-reads-memory/Following an LDG.E SASS instruction through the hardware units in an RTX 4090. Thu, 13 Aug 2026 00:00:00 GMThttps://fergusfinn.com/blog/what-happens-when-a-gpu-reads-memory/mdhttps://fergusfinn.com/blog/what-happens-when-a-gpu-writes-memory/https://fergusfinn.com/blog/what-happens-when-a-gpu-writes-memory/Following an STG.E SASS instruction through the hardware units in an RTX 4090. Fri, 28 Aug 2026 00:00:00 GMThttps://fergusfinn.com/blog/what-happens-when-a-gpu-writes-memory/mdhttps://fergusfinn.com/blog/what-happens-when-you-checkpoint-a-cuda-process/https://fergusfinn.com/blog/what-happens-when-you-checkpoint-a-cuda-process/Freezing a live CUDA process to host memory and thawing it again, what the driver does — and doesn't — do to make that work, and how understanding that lets us restore CUDA processes up to 4x faster. Thu, 09 Jul 2026 00:00:00 GMThttps://fergusfinn.com/blog/what-happens-when-you-checkpoint-a-cuda-process/mdhttps://fergusfinn.com/blog/what-happens-when-you-run-a-gpu-kernel/https://fergusfinn.com/blog/what-happens-when-you-run-a-gpu-kernel/Tracing one vector-add kernel from nvcc all the way down to the warps that execute it. Mon, 29 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/what-happens-when-you-run-a-gpu-kernel/mdhttps://fergusfinn.com/blog/when-to-disaggregate/https://fergusfinn.com/blog/when-to-disaggregate/Three checkable conditions under which disaggregated prefill is never worse than aggregated serving, and the reasons it's usually better. Plus an inventory-theory model of traffic drift: cold starts, safety stock, and a map of the shocks a deployment can ride out. Tue, 11 Aug 2026 00:00:00 GMThttps://fergusfinn.com/blog/when-to-disaggregate/mdhttps://fergusfinn.com/blog/weighted-fallback-flattening/https://fergusfinn.com/blog/weighted-fallback-flattening/When a weighted-random fallback rejects samples and retries without replacement, high error rates cause low-weight models to be selected far more often than their weights suggest. Tue, 17 Feb 2026 00:00:00 GMThttps://fergusfinn.com/blog/weighted-fallback-flattening/md