LLM inference thoughtshttps://fergusfinn.com/https://fergusfinn.com/blog/control-layer-benchmarking/https://fergusfinn.com/blog/control-layer-benchmarking/Benchmarking the Doubleword Control Layer Tue, 21 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/control-layer-benchmarking/mdhttps://fergusfinn.com/blog/deepseek-v4-flash-mi300x/https://fergusfinn.com/blog/deepseek-v4-flash-mi300x/A story of sharp edges, segfaults, and standardsMon, 01 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/deepseek-v4-flash-mi300x/mdhttps://fergusfinn.com/blog/control-layer/https://fergusfinn.com/blog/control-layer/We're releasing our AI Gateway. Why are AI gateways hard to do right, and why do we think we've done it right. Tue, 21 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/control-layer/mdhttps://fergusfinn.com/blog/economics-of-speculative-decoding/https://fergusfinn.com/blog/economics-of-speculative-decoding/Two underexplored axes: what MoE routing does to the decode roofline, and how compressed attention takes away the slack that used to make speculated tokens free. Mon, 08 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/economics-of-speculative-decoding/mdhttps://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/https://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/A walk through of the anatomy of an expert-parallel dispatch and combine kernel. Fri, 05 Jun 2026 00:00:00 GMThttps://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/mdhttps://fergusfinn.com/blog/faster-than-speed-of-light/https://fergusfinn.com/blog/faster-than-speed-of-light/The fastest a memory bound kernel can go is set by the time required to transfer the data to the SMs. How can we do better? Tue, 26 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/faster-than-speed-of-light/mdhttps://fergusfinn.com/blog/fast-sglang-starts/https://fergusfinn.com/blog/fast-sglang-starts/Checkpoint/restore with CRIU and cuda-checkpoint, from 12 minutes to 10 seconds on a B200. Mon, 06 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/fast-sglang-starts/mdhttps://fergusfinn.com/blog/inference-arithmetic/https://fergusfinn.com/blog/inference-arithmetic/Comparing InferenceMAX to the hardware limits Wed, 22 Oct 2025 00:00:00 GMThttps://fergusfinn.com/blog/inference-arithmetic/mdhttps://fergusfinn.com/blog/kv-entropy-coder/https://fergusfinn.com/blog/kv-entropy-coder/Lossless compression of a target model's KV cache by up to 4×, using a cheaper predictor model to drive an arithmetic coder. Fri, 08 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/kv-entropy-coder/mdhttps://fergusfinn.com/blog/llm-guided-scheduling/https://fergusfinn.com/blog/llm-guided-scheduling/An idea on how to use LLMs to help with scheduling Tue, 23 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/llm-guided-scheduling/mdhttps://fergusfinn.com/blog/paged-attention/https://fergusfinn.com/blog/paged-attention/A discussion of paged attention Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/paged-attention/mdhttps://fergusfinn.com/blog/scheduling-in-inference-engines/https://fergusfinn.com/blog/scheduling-in-inference-engines/A look into how inference engines choose which requests to process Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/scheduling-in-inference-engines/mdhttps://fergusfinn.com/blog/spacelike-speculative-decoding/https://fergusfinn.com/blog/spacelike-speculative-decoding/Speculative decoding speeds up LLM inference, but using another model works poorly. Mon, 22 Sep 2025 00:00:00 GMThttps://fergusfinn.com/blog/spacelike-speculative-decoding/mdhttps://fergusfinn.com/blog/llm-powered-content-discovery/https://fergusfinn.com/blog/llm-powered-content-discovery/Building a content discovery system using parallel primitives and BST-based ranking with LLM comparisons Fri, 16 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/llm-powered-content-discovery/mdhttps://fergusfinn.com/blog/weight-entropy/https://fergusfinn.com/blog/weight-entropy/An empirical investigation into the byte-level entropy of model weights across numeric formats and model families. Tue, 05 May 2026 00:00:00 GMThttps://fergusfinn.com/blog/weight-entropy/mdhttps://fergusfinn.com/blog/understanding-tans/https://fergusfinn.com/blog/understanding-tans/An intro to tANS: a table-based entropy coder that removes rANS's per-symbol division while keeping Shannon-optimal compression. Mon, 27 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/understanding-tans/mdhttps://fergusfinn.com/blog/understanding-rans/https://fergusfinn.com/blog/understanding-rans/An intro to rANS: an entropy coding method for losslessly encoding & decoding streams of bytes quickly. Tue, 21 Apr 2026 00:00:00 GMThttps://fergusfinn.com/blog/understanding-rans/mdhttps://fergusfinn.com/blog/weighted-fallback-flattening/https://fergusfinn.com/blog/weighted-fallback-flattening/When a weighted-random fallback rejects samples and retries without replacement, high error rates cause low-weight models to be selected far more often than their weights suggest. Tue, 17 Feb 2026 00:00:00 GMThttps://fergusfinn.com/blog/weighted-fallback-flattening/mdhttps://fergusfinn.com/blog/parallel-primitives-blog/https://fergusfinn.com/blog/parallel-primitives-blog/Exploring coordination patterns from parallel computing for multi-agent LLM systems Wed, 31 Dec 2025 00:00:00 GMThttps://fergusfinn.com/blog/parallel-primitives-blog/mdhttps://fergusfinn.com/blog/bst-expensive-comparisons/https://fergusfinn.com/blog/bst-expensive-comparisons/A lock-free binary search tree optimized for expensive async comparisons, with threaded linked list for O(1) sorted iteration Tue, 13 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/bst-expensive-comparisons/mdhttps://fergusfinn.com/blog/arxiv-llm-search/https://fergusfinn.com/blog/arxiv-llm-search/Applying parallel primitives to search and rank 2.4 million arXiv papers using LLM judgments Thu, 01 Jan 2026 00:00:00 GMThttps://fergusfinn.com/blog/arxiv-llm-search/md