Refusal ablation, measured

A label on an upload is not a measurement

Refusal-direction ablation follows a published result: instruction-tuned models mediate refusal through a single direction in the residual stream, and projecting that direction out of the weights removes the behaviour. Model hubs are full of uploads that claim to have done it. Almost none of them show what it cost, and I had seen enough “abliterated” builds produce nonsense to want the numbers.

The study model is Ornith-1.5-35B-A3B, a 35B hybrid-attention mixture-of-experts model with 3B active parameters, in bf16, served by oMLX on a Mac Studio and by vLLM on two RTX PRO 6000 GPUs. Three things made it a measurement rather than a demonstration: a benign prompt set matched topic for topic to the refusal set, so over-refusal and under-refusal are both visible; a capability gate, the five-task coding suite from my coding-agent-bench, run against the base and every variant on the same engine; and a check of the one community upload of this model against its base.

The direction is there from the early-mid stack

Thirty-two refusal-triggering prompts and thirty-two benign counterparts run through the chat template, and the residual stream at the last prompt token is recorded after every layer. Each layer gets its own mean-difference direction, scored by how far apart the two projected sets sit in pooled-standard-deviation units.

The separation is not confined to the late layers where it peaks. It passes four standard deviations at layer 13 and six by layer 24, peaking at 6.43 on layer 32 of 40. That shape is the reason the late-layers-only variant below fails: a refusal signal present from the middle of the stack cannot be removed by editing the top of it.

The embedding edit is what breaks the model

The winning configuration applies each layer’s own direction to every projection that writes into the residual stream, on all forty layers, and leaves the token embedding alone. Refusals go from ten or eleven of twelve to zero, the matched benign set produces no false refusals, and the coding suite stays at 489/500, identical to the base task for task. Decode and prefill speed are unchanged.

The original recipe also edits the token embedding. Applied that way, the same directions still reach zero refusals but drop the suite to 400/500: four tasks are untouched and the recursive-descent parser collapses from 98 to 13. Reducing the edit scale to 0.85 and 0.70 leaves the parser at 13 and 11. It is structural, not dosage. Restricting the edit to layers 20 and above brings the refusals back to ten of twelve and still breaks the parser, and a single global direction from the best layer, applied everywhere, is too weak: eleven of twelve becomes nine.

The community upload of this model, which claims all-forty-layer directional ablation and shows no benchmarks on its card, refuses ten of twelve prompts. That is the base rate. It was a no-op.

A dense model tells the same story

A same-recipe pair of Qwen3.8-27B builds, an abliterated model and its non-abliterated sibling from the same forge with the same quantization layout, scores 483/500 each, identical task for task, at the same speed. Earlier results that had me calling abliterated Qwen junk, with suite scores between 30 and 67 percent, traced to bad conversions and a broken CUDA path, not to the ablation. Blaming the technique for an infrastructure fault is the kind of mistake the gate exists to catch.

Two open items, kept visible

The same per-layer directions ported to the Hugging Face safetensors checkpoint and served by vLLM scored 379/500 against a 489 base on the same engine, with the parser at 13 and the trie at 72. The candidates are the stacked-expert edit in the HF path, the speculative-decoding draft head being edited with the global direction, or a packing mismatch. It is not root-caused, and the repository says so rather than quietly dropping the row.

Separately, a community-abliterated DeepSeek-V4-Flash derailed three times in twelve agentic tasks, producing runs of begin-of-sentence tokens followed by off-distribution text, where the non-ablated comparison model had none. Speculative decoding and the ablation edit were not isolated in that run, so it stays an observation, not a finding.

What I would insist on before trusting an abliteration

The repository holds the scripts, the prompt set and the measurements. No weights are distributed. What it really contributes is the discipline around the technique:

• Confirm the base model refuses the probe set before measuring the variant, or you will prove an ablation that did nothing.
• Judge the final answer, not the reasoning trace; these models deliberate about refusing and then comply, or the reverse. Keep the token budget generous or truncated thinking scores as a refusal.
• Run a benign set matched by topic, so over-refusal is measured and not assumed away.
• Gate on a capability suite against the base on the same engine, and accept the variant only within a few points.
• A/B every community upload against its base before believing the card.