# Native MTP: read-only feasibility review

Initial review used only installed SGLang 0.5.19/vLLM 0.29.0 source and safetensors
headers under `/home/lab/models/ornith-modular/original-text`. The parent later
authorized CPU extraction/conversion to fulfill the requested auxiliary comparison
despite the small memory benefit. `mtp_ptq.py` implements that separate step.
No model is run and no engine is modified by this script.

## Recommendation

### Completed CPU artifacts

- `/home/lab/models/ornith-modular/mtp-bf16-compact`: verified 17 original tensors.
- `/home/lab/models/ornith-modular/mtp-linear-int8`: verified mixed INT8 artifact,
  4,348,621,176 bytes of weight files (before runtime sharing).
- `/home/lab/models/ornith-modular/mtp-linear-int4`: verified mixed INT4 artifact,
  4,243,763,560 bytes of weight files (before runtime sharing).

All commands exited successfully. Every preserved tensor is byte-identical,
configuration is unchanged except the quantization block, and tokenizer/template
hashes match. All three safetensors indices were separately checked against actual
header names, shard locations and total payload sizes. Manifests accompany each
artifact and are copied locally beside this report. No GPU validation is claimed;
the vLLM inheritance blocker described below applies to independent mixed variants.

Keep the original BF16 MTP branch for current trials. Selective native CT W8/W4A16
is technically plausible in SGLang, but maximum tensor savings are only about
0.1923 / 0.2899 GiB respectively. Those are theoretical packed tensor savings,
not measured runtime savings; extra workspaces could reduce the benefit.

The selected source contains **15 MTP tensors**, not 22. Their total raw payload is
486,581,248 bytes (0.4531 GiB). Seven attention/MLP matrices account for
419,430,400 bytes. `mtp.fc.weight` alone is 67,108,864 bytes and remains BF16 for
compatibility with SGLang's ordinary `nn.Linear`.

| Scheme for seven matrices | Estimated packed payload, including scales/shapes | Saved relative to BF16 |
|---|---:|---:|
| W8A16 symmetric group128 | 212,992,112 bytes | 0.1923 GiB |
| W4A16 symmetric group128 | 108,134,512 bytes | 0.2899 GiB |

The estimate assumes BF16 per-group scales and standard CT packed INT32 weight
containers. It excludes safetensors metadata overhead, which is small here.

## Compact native BF16 checkpoint feasibility

A common-runtime subset would retain exactly:

- All 15 `mtp.*` tensors, unchanged.
- `model.language_model.embed_tokens.weight`, 2,034,237,440 bytes.
- `lm_head.weight`, 2,034,237,440 bytes.

Total raw payload: **4,555,056,128 bytes (4.2423 GiB), 17 tensors**. Keep the original
text-model configuration and tokenizer; do not invent a new architecture or
change the original decoder-layer count. Native speculative model selection
constructs the MTP runtime class and its one auxiliary layer. Rebuild the weight
index so that it lists only these tensors and the new immutable shard(s).

The authorized extraction reduces checkpoint I/O/disk size versus scanning the
complete target checkpoint. Artifact readiness is determined by successful exit
and a verified `extraction-manifest.json`, not directory existence alone.
It does **not** eliminate the initial embedding/head allocations: runtime code
constructs them before subsequent sharing. Thus it must not be advertised as
fixing a 4.24-GiB initial-loading peak. No original shard may be modified.

SGLang's MTP loader skips every checkpoint tensor not containing `mtp`. Shared
embedding/head setup supplies those weights afterward. The larger common subset
above is retained for vLLM, whose loader explicitly accepts MTP, embedding and
head tensors and skips the unrelated language-model layers.

## Native CT compatibility and exact prospective scheme

SGLang `srt/models/qwen3_5_mtp.py`:

- `_mtp_quant_config` retains compressed-tensors; its BF16 exceptions concern
  other quantizers and do not generically disable CT.
- `fc` is `nn.Linear` and must remain BF16.
- The auxiliary decoder is constructed via `Qwen3_5ForCausalLM(..., is_nextn=True)`
  with the quantization configuration. It uses full attention.
- `load_weights` selects `mtp.*`, strips `self_attn` as needed, and maps Q/K/V and
  gate/up packed parameters to the fused runtime modules and their weight loaders.
- `set_embed_and_head` and `set_lm_head_from_target` support post-loading sharing.

vLLM `model_executor/models/qwen3_5_mtp.py`:

- `Qwen3_5MultiTokenPredictor` constructs a full-attention `Qwen3_5DecoderLayer`
  with `vllm_config.quant_config`. Follow-up source inspection and the vLLM
  agent's actual loading failure confirm this is inherited from the target,
  not independently resolved from the draft checkpoint in this runtime.
- `fc` is quantization-aware `ColumnParallelLinear`, but must still be excluded
  for a common artifact because SGLang uses ordinary `nn.Linear` there.
- `Qwen3_5MTP.load_weights` maps `mtp.*` to its model subtree, accepts embeddings
  and head, and ignores unrelated main-decoder tensors.
- Attention and MLP use normal quantization-aware decoder paths; there is no
  DFlash-style direct slicing of dense QKV weights in this MTP class.
- Mamba prefix cache mode `all` is explicitly rejected; keep an already-supported
  setting rather than changing it incidentally for quantization trials.

### Important vLLM independent-quantization blocker

`v1/spec_decode/llm_base_proposer.py::_create_draft_vllm_config` begins with the
target VllmConfig and replaces kernel/attention/KV settings, but not `quant_config`.
`_get_model` passes draft `model_config` separately. Then
`model_executor/model_loader/utils.py::initialize_model` constructs the new-style
MTP class with the inherited VllmConfig. Unlike DFlash, Qwen3.5 MTP does not call
`get_draft_quant_config`.

Consequently, independently specifying draft quantization is not sufficient here:
the observed BF16 draft with CT target expects `lm_head.weight_packed` and fails.
The compact INT8/INT4 artifacts described below retain BF16 head and are therefore
SGLang candidates only until separately demonstrated otherwise. They must not
be advertised as native independent vLLM candidates merely because their linear
kernels exist. Using the same FULL-INT8 target checkpoint as vLLM's MTP source is
the coherent native configuration currently being tested by the vLLM agent, with
its original BF16 MTP branch and matching CT head. No runtime patch or target
configuration rewrite was performed in this task.

The authorized CPU model-free PTQ on the new compact source subset selects only:

```
mtp.layers.0.mlp.gate_proj.weight
mtp.layers.0.mlp.up_proj.weight
mtp.layers.0.mlp.down_proj.weight
mtp.layers.0.self_attn.q_proj.weight
mtp.layers.0.self_attn.k_proj.weight
mtp.layers.0.self_attn.v_proj.weight
mtp.layers.0.self_attn.o_proj.weight
```

Use the existing isolated CPU environment and official compressed-tensors
`pack-quantized`, W8A16 or W4A16, symmetric group128, no activation quantization.
Ignore `re:.*fc$`, `re:.*norm.*`, `re:.*embed_tokens.*`, `re:.*lm_head.*`.
All seven matrix input dimensions are divisible by128 and output dimensions by64,
compatible with the previously verified native Marlin geometry on SM86/TP1.
Check exact selection before conversion, preserve all other tensor bytes,
leave source/config architecture unchanged, and produce full manifests.

Names should explicitly describe partial quantization (for example
`mtp-linear-int8`/`mtp-linear-int4`), not imply all weights are INT8/INT4. Independent
draft quantization/load configuration is required. GPU loader validation and
acceptance/performance trials would still be necessary; source inspection is not
runtime proof.
