# DFlash mixed INT8 / INT4 auxiliary artifacts

Source: `/home/lab/models/ornith-modular/dflash-bf16`, from
`z-lab/Qwen3.5-9B-DFlash`, revision
`5fc3b3d474760f18c516db87d84c37edbfd3ede6`.

The unchanged source is a six-layer auxiliary trained for Qwen3.5-9B, not a
standalone Ornith model. Acceptance and acceleration with Ornith must be measured.
It supplies no tokenizer; keep the verified target tokenizer.

## Native runtime constraints inspected before conversion

Installed SGLang 0.5.19:

- `srt/models/dflash.py` passes quantization configuration to attention and MLP
  linear modules and maps stacked QKV/gate-up packed parameters in `load_weights`.
- `fc` is ordinary `nn.Linear` for this model and therefore must stay BF16.
- `kv_proj_only` has a quantized fallback, and the worker disables fused context
  materialization when QKV is quantized. Preserving QKV keeps the dense fast path.

Installed vLLM 0.29.0:

- `model_executor/models/qwen3_dflash.py` propagates independent draft quantization
  configuration into MLP, `o_proj`, QKV and `fc`.
- `o_proj` is invoked through its linear module (`output, _ = self.o_proj(...)`),
  and MLP is the ordinary quantization-aware `Qwen3MLP`.
- `_build_context_kv_buffers` slices `a.qkv_proj.weight[a.q_size:]` directly;
  `_project_context_kv` uses `F.linear` over those dense fused weights. CT WNA16
  registers `weight_packed`, not a dense `weight`. Thus fully quantized QKV is
  not natively compatible with this installed DFlash implementation.
- Preserve QKV and `fc`. Ignore rules explicitly cover separate and runtime-fused
  QKV names, including `qkv_proj`, and prefixed `fc`.

No engine source is patched. No GPU operation is part of artifact creation.

## Exact scheme

Both artifacts quantize only 24 matrices: six layers times gate/up/down MLP and
attention `o_proj`. All 45 other tensors, including QKV, `fc`, and norms, are
verified byte-identical to the source and retain their dtypes.

- `/home/lab/models/ornith-modular/dflash-mlp-int8`: mixed W8A16 + BF16.
- `/home/lab/models/ornith-modular/dflash-mlp-int4`: mixed W4A16 + BF16.

Both conversions completed and verified with exit status 0. The INT8 weight
file is 1,592,917,520 bytes (1.4835 GiB); INT4 is 1,089,600,904 bytes (1.0148 GiB).
Source weight file: 2,583,816,465 bytes. These are artifact file sizes, not total
runtime VRAM: the engine may additionally share/copy embeddings, logits head,
KV state and workspaces. CPU conversion plus verification took approximately
29 seconds per artifact. Both manifests are copied next to this report.

The suffix `mlp` is shorthand; the manifests explicitly enumerate the additional
`o_proj` matrices. Neither artifact should be described as fully INT4 or INT8.
Both use symmetric groups of 128, official compressed-tensors `pack-quantized`,
and isolated CPU `llmcompressor.model_free_ptq` RTN/memoryless-minmax. No training
or calibration examples are used. Configuration retains the original architecture
and all original fields; only `quantization_config` is added.

Each destination must not exist before running `dflash_ptq.py --bits 8` or
`--bits 4`. Successful completion writes `ptq-manifest.json` with exact selected
names, preserved tensor hashes, source/output hashes, package versions and script
hashes. An artifact is not ready until that manifest exists and the command exits
successfully. The manifest deliberately does not assert runtime or quality success.

## Runtime handoff, not yet runtime proof

SGLang: use the new draft model path and explicit
`--speculative-draft-model-quantization compressed-tensors`, leaving target and
other settings unchanged for the comparison.

vLLM: set the speculative draft model to the new path, independent draft
`quantization: compressed-tensors` and `draft_load_config: {"load_format":"auto"}`
so a target-specific loader is not inherited. Keep the already validated target
configuration. Both group128 symmetric INT8 and INT4 have native Marlin kernels
on the RTX3090 SM86; actual DFlash loading/acceptance still needs GPU validation.
