Optimizing local AI: kernels, prefill and decode

The engineering starts before the benchmark

My local stack has involved serving engines, quantized checkpoints, GPU kernels, tool parsers, caches and evaluation adapters. Much of the underlying kernel work comes from open-source maintainers and community builds. My contribution has been hands-on integration, local patches, controlled bisects, configuration tuning, and finding where apparently compatible pieces stop agreeing.

The DeepSeek bring-up records show why a supported GPU architecture is not a yes-or-no property of a whole model. An attention kernel could work on workstation Blackwell while an output projection or expert kernel still assumed a different architecture. A capability check could pass while a compiled wheel lacked the necessary implementation. Changing a version label was not enough.

The successful path required matching the compiler and runtime, selecting compatible kernels, managing loading-time memory separately from steady-state memory, and testing the resulting system. I would not attribute the later speed gains entirely to my patches: checkpoint, quantization, engine and speculative-decoding changes also mattered.

Prove that the fast path is still correct

An August Qwen3.8-27B campaign separated the model, quantization and serving path rather than judging the model from one bad output. One bisect isolated a failure to an FP8-plus-MTP combination in that particular engine. FP8 without multi-token prediction behaved correctly in both eager and compiled modes; another quantized path with MTP also passed the recorded checks. The interaction mattered more than either feature’s name.

A separate multi-GPU incident involved the Linux kernel, not a CUDA compute kernel. After a system update, the driver reported peer access as available, but real transfers told a different story. Direct cross-GPU copies mismatched in all 20 recorded trials; host-staged copies matched in all 20. Collective communication could hang as well.

The workaround avoided the broken peer-to-peer path and tested actual data movement. That experience made a small numerical correctness probe part of the bring-up process. A fast server that silently changes data is not an optimization, and an advertised capability is not an end-to-end test.

Prefill, decode and waiting are different measurements

Prefill processes the prompt; decode generates new tokens. Time to first token also includes waiting for a server slot and other overhead. An agent with a long history can spend much of its time on prompt processing even when its visible token stream looks fast.

My April concurrent benchmark recorded first-token timing, generation timing and task results separately. In a five-way run, front-of-queue requests reached a first token in roughly half a second, while queued requests waited about 45 and 57 seconds. Dividing their prompt lengths by that wait produced a misleadingly terrible “prefill” rate. It measured queueing as well as computation.

The post-first-token generation rates in that run were much closer—about 61–66 tokens per second. The lesson was not that the kernel suddenly became a hundred times slower for one request. It was that I needed to distinguish queue delay, prompt processing, per-user decode speed and aggregate throughput.

For long-running agents, I also watch the growth of the conversation and the useful cache working set. A fast short completion is not representative of repeated tool calls carrying a large history. Cached and uncached prompt tests need to be labeled, not blended into a single speed claim.

A concrete tradeoff: speculative depth versus useful throughput

An August 27 Qwen3.8-Flash-Next sweep on my two-GPU workstation tested multi-token prediction (MTP), which proposes several tokens ahead for verification. Deeper speculation improved the isolated decode number slightly, but did not improve every workload. It also consumed memory that otherwise supported the key/value cache.

The table is a dated configuration sweep, not a current hardware leaderboard. C=16 means aggregate throughput with sixteen concurrent requests; it is not the speed an individual agent saw. The launch choice also kept memory headroom for the workstation rather than using the most aggressive setting.

The chosen configuration needed a quality check, too

Increasing the batch-token limit recovered much of the prefill throughput lost to speculation, while giving up some isolated decode speed. Depth 12 offered little extra decode speed and substantially worse concurrent throughput than depth 4. The choice was a workload tradeoff, not a hunt for the largest number in one column.

The accompanying five-task coding checks recorded 400/400 on the functional portion for both MTP runs and the no-MTP control. Total scores were 475, 480 and 482 out of 500; the spread came from the separate code-quality heuristic. That small sample did not demonstrate a general quality loss—or prove that MTP is always quality-neutral.

These are not the four-task Prime agent scores discussed in my other evaluation note. They come from a different suite and should not be combined. The campaign also had several generations of answer extraction and sampling adapters. A number without its adapter version can be an invalid comparison.

What actually serves on the workstation

The launch recipes behind this note are public as blackwell-serving-kit, with the DeepSeek work in its own repository. The table is the measured state of each configuration on the two-GPU workstation between April and September 2026: one launcher per row, the single-stream decode rate, and the aggregate rate at the concurrency noted. Rows are not interchangeable: different models, quantizations, engines and dates.

Two of the findings in the repository are worth stating here. The peer-to-peer workaround described above cost nothing measurable on the DeepSeek serve (185.2 tokens per second with P2P disabled against 185.7 with it on), and a two-minute probe now runs after every kernel update before any engine flag is blamed. And the GLM-5.3-Flash NVFP4 checkpoint did not fit for two independent reasons that are written up separately: a quantization that left the attention stack in BF16, and an SM120 sparse-attention kernel that asserts on models without rotary position dimensions.

Better evaluators required software work, not just more GPU

Kernel tuning and evaluator quality are connected operationally, but they are not the same improvement. A faster inference path does not make a score more meaningful. In one coding incident, fixing final-answer extraction changed the recorded score from 17/100 to 97/100 without changing the model’s answer at all.

For video, using a persistent, stronger speech-recognition service avoided repeatedly loading a small local transcriber and helped keep rendering and evaluation from fighting over resources. The current LTX quality-gate code can still fall back to a smaller transcriber. That fallback must travel with the result: changing the evaluator backend can change what a word-error score means.

I also check evaluator availability and parsing explicitly. An empty extracted script or a skipped visual check is missing evidence, not a passed creative requirement. These repairs improve the validity of the evaluation; serving and kernel work make it more practical to run it often.

On the agent side, tool parsers, sampling defaults and reasoning budgets affect whether inference becomes action. A local GLM experiment became much more useful with a bounded thinking budget rather than letting reasoning consume the turn. Timeouts and context-limit exits remain inconclusive capability results, while still counting as unfinished work for the person waiting.

How much of this work did I actually record?

There is a substantial trail: dated experiment notes, launch recipes, benchmark runners, raw run artifacts, a factorized Qwen campaign manifest, and correction notes when an earlier explanation turned out to be wrong. Together they document months of bring-up and optimization work.

They are not a complete timesheet. Some old checkpoints were removed for storage, some settings changed between campaigns, and some investigations ended with a working recipe rather than a final report. I can reconstruct the sequence below; I cannot honestly turn it into a precise lifetime total of hours or an across-the-board speedup.

Why I keep doing this locally

What draws me to local AI and open source is the ability to inspect the whole chain. I can preserve a working version, compare an adaptation against its base, examine a numerical failure, and build an agent around a task rather than a generic chat window.

Open weights, open-source software and a community-maintained kernel are different things; I do not assume every component has the same license or level of visibility. But the inspectable parts let me turn a failure into something reusable: a test, a correction, a safer default, or a recipe the next agent can follow.

My preferred performance metric is becoming time to a correct, accepted artifact under the workload I actually run. Decode speed still matters. So do the minutes before the first token, the evaluator that catches a bad result, and the recovery path when a long-running task goes wrong.