# Reproducible evaluation

Input: **JSONL, one recording object per line — not a JSON array.**

Requires Python 3.10+ and the packaged requirements. Use mono PCM16/16 kHz WAV
input. For other formats, convert locally before creating the manifest:

```bash
ffmpeg -i input.m4a -ac 1 -ar 16000 -c:a pcm_s16le input.wav
python -m pip install -r scripts/requirements.txt
```

Create a JSONL manifest. Audio paths are relative to the manifest. Reference
text and vocabulary are optional, but references are needed for WER and terms
for paired vocabulary tests:

```json
{"id":"clip-1","audio":"input.wav","reference":"The patient takes metformin.","vocabulary":["metformin"]}
```

Start with an offline plan, then a small authorized live run:

```bash
python scripts/evaluate.py manifest.jsonl --transport realtime \
  --model omi-medical-1 --output results --client-location us-east --dry-run
python scripts/evaluate.py manifest.jsonl --transport realtime \
  --model omi-medical-1 --output results --client-location us-east
```

The default realtime end policy is `stop` at audio end, with no artificial
silence tail. `--end-policy finalize` instead sends an explicit end-of-turn
signal, waits for its matching ack, then stops. Report the policy. Keep the
same audio bounds and end policy for every provider comparison. An audio
file may contain leading/trailing silence: the runner does not trim it or
pretend its end is the true speech boundary.

For batch vocabulary on/off use **Medical**, not streaming:

```bash
python scripts/evaluate.py manifest.jsonl --transport batch \
  --model omi-medical-1 --compare-vocabulary --output vocab-results \
  --client-location us-east
```

Both arms disable stored dictionaries. The on arm sends the manifest's exact
term list. Output retains the vocabulary audit; the runner fails an on arm
that lacks one instead of claiming terms applied. It does not change account
privacy settings; incompatible zero-retention settings fail explicitly.

## Outputs and clocks

- `attempts.jsonl`: append-only per-file results, including failures, input and
  configuration hashes, requested/served model, ready/options receipt, confirmed
  text, closing record, audits and timings. Treat this as sensitive customer
  data. New files are owner-readable only; no API keys are written.
- `summary.json`: attempted/success/failed counts, coverage, normalized
  corpus WER, and median/p95 clocks separately for each arm. Percentiles are
  over successful measured samples and state their sample counts. Failures
  stay visible outside those distributions. WER normalization lowercases,
  replaces punctuation with spaces and preserves numbers; it is not a
  clinical correctness score or Coval's scorer.
- `comparison.csv`: per-file statuses, word edits, text/record clocks and
  configuration arm, suitable for paired analysis. Transcripts stay in JSONL.

Realtime clocks:

1. First visible text: first nonempty draft/context/turn text after first PCM
   send. Leading silence and time spent speaking are included.
2. First confirmed turn: first `turn_final` after first PCM send; includes
   time spent speaking. This is **not** post-speech response latency.
3. Last confirmed turn relative to **audio file end**: may be negative if the
   file has trailing silence. It is not automatic speech-end latency unless
   the file actually ends at speech end.
4. Closing record after Stop: a separate session-end clock.
5. Session wall time: includes connection setup, pacing and closure.

Score concatenated `turn_final` text and the closing record separately. A
closing record cannot retrospectively improve a reported confirmed-turn WER.
No partial fallback. An incomplete/review-required record is retained as a
failed result, rather than silently counted as a complete success.

Batch wall time is upload through usable result, including async polling.
It is not realtime latency or engine-only speed.

`--resume` skips completed input/configuration hashes (including previous
failures). `--resume --retry-failed` explicitly retries failures and keeps
the previous attempt. Resume after a changed manifest/model/options creates
new attempts. `--concurrency` defaults to 1; respect available project limits.
No automatic whole-file retries or silent option/model fallbacks occur.