# Ornith modular control

The installer deploys code only, using Python's standard library. It never
changes an agent runtime, global package, GPU process, or live service.
Root must coordinate the first migration with the engine owners.

## Files and intended installation

- `ornithctl.py` -> `/home/lab/bench/ornith/modular/control/ornithctl.py`
- `ornith_gateway.py` -> same `control/` directory
- Run `install.sh` as lab to test fixtures, back up changed code, install it,
  and link `/home/lab/.local/bin/ornithctl` without replacing unknown commands.
- `test_control.py` is a local fixture suite; it does not access GPUs/systemd.

The registry is `/home/lab/bench/ornith/modular/profiles/<engine>.json`:

```json
{
  "engine": "vllm",
  "endpoint": "http://127.0.0.1:8140",
  "gpu_uuid": "GPU-REPLACE-WITH-VERIFIED-GPU2-UUID",
  "baseline_id": "baseline",
  "profiles": [{
    "id": "baseline",
    "status": "validated",
    "reason": "Existing configuration with bounded end-to-end proof",
    "command": ["/bin/bash", "/absolute/immutable/baseline-launch.sh"],
    "env": {},
    "model_id": "actual-served-model-id",
    "context": 32768,
    "concurrency": 1,
    "evidence": ["/absolute/path/to/validation.json"]
  }]
}
```

Optional arguments after the absolute launcher are supported (for example the
profile ID used by the real engine launchers). An optional `health_timeout`
number sets startup health wait in seconds, default600, allowed1–3600.
Launchers must honor `ORNITH_PORT`, remain in the foreground (`exec` the worker),
and must not override the forced `CUDA_VISIBLE_DEVICES` UUID. Do not call an old
`manage.sh` that backgrounds an untracked process or assumes the public port.
Archive baseline scripts/configs read-only with hashes before any migration.
Registry and launcher writes are administrative/trusted actions. The controller
requires the validated label plus nonempty evidence; it does not independently
re-grade historical evidence. Read-only `list`/`status` never dump environment.

## CLI

```sh
ornithctl sglang list
ornithctl sglang status
ornithctl sglang use baseline
ornithctl sglang rollback
ornithctl vllm list
ornithctl vllm use profile-id
```

Only `validated` profiles with evidence and a two-element absolute Bash launcher
command (plus optional arguments) can activate. `experimental` and `unavailable` profiles remain visible,
with reasons. There is deliberately no force flag.

## First migration (root coordinates; NOT performed here)

1. Preserve immutable baseline and validated evidence. Check original PID,
   process command, GPU UUID, ports, and session ownership.
2. Drain the existing engine using its current safe mechanism; because no gateway
   yet exists, prevent new callers at the client layer and wait for outstanding
   requests. Controller cannot safely adopt the untracked process.
3. Stop ONLY that positively identified original worker. Never kill the tmux
   session, llama/GPU0, or processes discovered solely by a broad command match.
4. Confirm both public and backend ports are vacant. Ensure environment has
   `XDG_RUNTIME_DIR=/run/user/1000` and working `systemctl --user`.
5. Activate baseline using `ornithctl <engine> use baseline`. On first use it
   starts the managed backend, health/generation-probes it, then starts gateway.
6. Verify `/health`, `/v1/models`, a completion and a streamed chat completion on
   public port. Check PID/GPU UUID, model ID, context and the unchanged GPU0 PID.
7. Keep the migration revert command for the original baseline available if
   initial activation fails. No previous managed profile exists on first use.

Port assignments (HTTP loopback only, deliberately no external exposure):

| Engine | Public gateway | Managed backend |
| --- | --- | --- |
| SGLang | 8130 | 8131 |
| vLLM | 8140 | 8141 |

User transient units are `ornithctl-<engine>.service` and
`ornithctl-<engine>-gateway.service`. The controller verifies `Transient=yes`
and its exact description marker before stopping/reusing unit names. Workers
are confined to their unit cgroup; no PID searching, `pkill`, or foreign-unit
adoption exists. An already listening backend port is rejected before launch.
These transient units do not auto-start after reboot; boot persistence would
require a separately reviewed persistent-unit installation.

## Switching and rollback

Private control sockets reside in `/run/user/1000/ornithctl/`, directory0700,
socket0600. Drain atomically refuses new HTTP requests with503/Retry-After while
existing HTTP/SSE requests continue. At zero active requests the controller stops
the old owned worker, starts the candidate on the fixed backend port, waits up to
600s by default for health, and performs one non-streaming completion with at most16 tokens
and90s request timeout. Successful smoke is plumbing proof, not model quality.

A drain exceeding600s restores admission and never stops the old worker.
A candidate failure stops only its owned unit and automatically restores the
previous profile snapshot, re-probes it, and resumes admission. If recovery also
fails, admission stays closed and state becomes `recovery-required`.

State is atomically replaced/fsynced under `modular/state/`; each engine has an
exclusive nonblocking lock. Profile snapshots preserve rollback independently of
subsequent registry edits. A controller interruption leaves `phase=changing`;
further changes are refused pending operator investigation. Do not blindly edit
that phase toready. Inspect current unit, backend, generation and saved profiles;
only after restoring a known worker and evidence should an operator repair state
and resume the private gateway. This is intentionally fail-closed, not an
automatic recovery guess after a control-process crash.

Logs: `journalctl --user -u ornithctl-vllm.service` (or sglang). Gateway does not
log query strings, request bodies, authentication headers, or streamed output.
SSE is forwarded incrementally without aggregation or replay. Chunked request
bodies are supported up to32MiB. WebSockets/upgrades are not supported. Idle
upstream reads time out after600s; client request-body/socket writes after60s.
Workers and gateway execute through `env -i`: only basic explicit environment
and profile environment reach them, never arbitrary private shell/user-manager
tokens. Required custom cache/runtime settings belong in the profile explicitly.

## Validation

```sh
cd work/modular/control
python3 -m unittest -v test_control.py
```

Tests use temporary fixtures and ephemeral localhost ports only. They cover
successful switching/explicit rollback, automatic rollback, recovery failure,
timeout preserving the old worker, profile/evidence rejection, concurrent locks,
read-only status, foreign-unit/backend safety, initial activation, HTTP body/status
forwarding, chunked requests, first SSE event before completion, drain refusal,
waiting for active requests, and admission restoration.
