# Ornith-1.5-9B / SGLang / ia-lab GPU1

This deployment is isolated to physical GPU1, UUID `GPU-f4a988cb-c310-b618-5a9c-4462d4d33add`, using `CUDA_VISIBLE_DEVICES`. Inside the process CUDA calls it device0; this is **not** physical GPU0. Llama on physicalGPU0 is untouched.

- Runtime: existing `/home/lab/venvs/sglang`, SGLang0.5.19, PyTorch2.13.0+cu130, Transformers5.12.1.
- Original model: `/home/lab/models/Ornith-1.5-9B-original`, pinned original revision `489cb97981b8654bcfcf30ce1f94ed1b62e07b53`, readonly.
- Quantization: native SGLang online FP8 weight-only compression; Ampere uses Marlin W8A16. Original disk checkpoint remains BF16. This is not a4bit export or QLoRA model. Embeddings, norms and non-quantized layers retain their original/compute precision.
- Activations/KV cache: BF16. No FP8 tensor-core claim is made for RTX3090.
- API: `http://127.0.0.1:8130/v1`, served model `ornith-sglang`.
- Initial context8192, maximum2concurrent requests,80%static GPU memory, eager execution, Triton attention and Mamba kernels. This is a functional initial configuration, not a tuned benchmark.
- Network binds localhost only. Access remotely using an SSH tunnel, not a public unauthenticated port.

## Controls

```sh
bash /home/lab/bench/ornith/sglang/manage.sh status
bash /home/lab/bench/ornith/sglang/manage.sh stop
bash /home/lab/bench/ornith/sglang/manage.sh start
```

`start` launches visibly in `sglang:Codex` only if its shell is idle. `stop` verifies the exact server command, model alias, port and GPU UUID before sending SIGINT; it does not use broad process-kill commands. `status` checks actual HTTP health and models, not merely PID existence.

The server survives SSH/Herdr disconnects because it runs in tmux. No boot-time service has been enabled. After a reboot, use `manage.sh start` after creating/attaching the `sglang` tmux session if needed.

Test: `python3 /home/lab/bench/ornith/sglang/smoke.py`. Evidence: `server.log`, `smoke.log`, `smoke-result.json`. Verified2026-09-18: HTTP health200, `/v1/models`200, chat200 with final content `SGLANG_OK`, finish_reason `stop`, 5 completion tokens and no reasoning tokens; repeat warm request1.435seconds. This is a smoke latency, not a benchmark. First startup spent about7minutes compiling local CUDA kernels. Log tab follows output; Monitor shows onlyGPU1.

## Compatibility notes

SGLang's installed Qwen3.5 implementation lacks the mapping required by its BitsAndBytes loader. No architecture patch was applied; supported FP8+Marlin is used instead.

`--language-model-only` currently only supports MuseGlimmer in this installed runtime, so the normal Qwen3.5 architecture is loaded. Text is tested; image/audio/video are not. A missing FFmpeg/torchcodec optional import warning does not prevent text serving. No global codecs, drivers or packages were installed.

Sources: https://docs.sglang.io/docs/advanced_features/quantization and https://huggingface.co/ornith-ai/Ornith-1.5-9B .
