# Video Intelligence (real CV + optional evidence extras)

Existing endpoint — **no new API**.

`POST /api/v1/video-intelligence` · header `X-Internal-Token`

## Pipeline

1. Decode `video_base64` or fetch `source_url` (http(s); optional local/`file://` when `VIDEO_ALLOW_LOCAL_SOURCE=true`)
2. Sample frames at `sample_fps`, capped by `max_frames` / `VIDEO_MAX_FRAMES` — **distributed across the full duration** (not a prefix-only window)
3. Validate DeepFace/ArcFace/anti-spoof deps
4. Face detection + anti-spoof liveness on sampled frames
5. Optional ArcFace match if `registered_photo_base64` / `id_image_base64` provided
6. Build **visual_events** from face counts / quality / anti-spoof (observations only)
7. Optional `include_transcript`: **FFmpeg demux → mono 16 kHz PCM WAV → `TranscriptionService().transcribe(...)`**
8. Align notable visual events ↔ transcript segments by timestamp proximity (highlights)
9. Optional `include_report`: structured recruiter report via text `generate_json` (+ local fallback)

Identity and liveness remain separate. Engagement signals are derived from detection stats (not LLM guesses).

### Not in this pipeline (by design)

- YOLO / ultralytics object detection
- Multimodal Gemini vision (LLM client is text-only)
- Async job queue or separate report GET
- Hiring suitability / personality / honesty from face, gaze, or posture

## Required

`consent_given: true`

## Request fields (supported)

| Field | Default | Notes |
|-------|---------|--------|
| `interview_id` | required | |
| `video_base64` **or** `source_url` | one required | Mutually exclusive |
| `sample_fps` | `1.0` | 0.1–5.0 |
| `max_frames` | `VIDEO_MAX_FRAMES` (16) | 1–60 |
| `registered_photo_base64` / `id_image_base64` | omit | Optional identity reference |
| `consent_given` | `false` | Must be `true` |
| `include_transcript` | `false` | Opt-in audio demux + STT |
| `include_report` | `false` | Opt-in structured report |
| `interview_topics` | `[]` | Optional shared context for highlights (single topic only is attached) |
| `options.transcript_provider` | `openai` | `openai` \| `whisper` (alias) \| `stub` |
| `options.language` | `en` | Passed to STT |
| `options.stub_transcript` / `stub_segments` | — | Stub STT only (tests/dev) |

## Transcript path (`include_transcript: true`)

1. `extract_audio_wav_from_video` (FFmpeg) → mono, 16 kHz, PCM WAV  
2. WAV → `TranscriptionService` with `options.transcript_provider`  
3. Response field `transcript`: `{ text, segments[{start_ms,end_ms,text}], provider, model, stub, error_code, message }`

### Transcript↔video alignment (highlights)

**Timed segments are required** for co-occurring highlights (`relationship: co_occurring`).  
Visual-only highlights mean either transcript was not requested, STT failed, or segments had no usable timestamps.

| Source of timestamps | Behavior |
|----------------------|----------|
| `whisper-1` + `verbose_json` | Real STT segment `start`/`end` (preferred) |
| Model returns text only (e.g. `gpt-4o-mini-transcribe`) | Approximate segments synthesized from text + WAV duration; limitation `transcript_segments_approximate` |
| No text / no duration | `transcript_segments_missing` → highlights stay visual-only; completeness `partial` |

When `include_transcript: true` and provider is not stub, video intel defaults to **`whisper-1`** (`require_segments=true`) so alignment works. Override with `options.transcription_model` if needed.

**Providers actually supported**

| `transcript_provider` | Backend |
|----------------------|---------|
| `openai` | OpenAI STT (`OPENAI_API_KEY`); video path defaults model to `whisper-1` for segments |
| `whisper` | Same OpenAI client (legacy alias) + `verbose_json` segments |
| `stub` / `local` / `mock` | In-process stub — **not production speech**. Set `stub_transcript` / `stub_segments` for deterministic text |

Request `transcript_provider` is honored (not overridden by env when explicitly set).

**Failure codes** (biometric analysis can still `status=succeeded`; completeness becomes `partial`):

| `transcript.error_code` | Meaning |
|-------------------------|---------|
| `no_audio_track` | Video has no audio stream |
| `ffmpeg_missing` / `ffmpeg_timeout` / `audio_demux_failed` | Demux problems |
| `audio_too_large` | Extracted WAV over `VIDEO_TRANSCRIPT_MAX_AUDIO_BYTES` |
| `invalid_audio` | Empty/undecodable audio payload for STT |
| `stt_not_configured` | Missing API key for OpenAI STT |
| `stt_provider_failed` | Provider HTTP/API error |
| `empty_transcript` / `transcription_empty` | STT returned no text (incl. near-silent audio) |
| `transcription_failed` | Unexpected STT exception |

Near-silent tracks demux successfully but add a warning; STT may then return empty text — that is **not** claimed as a successful transcript.

## Response semantics

| Field | Meaning |
|-------|---------|
| `status` | Processing / analysis gate: succeeded when liveness passed and identity was not a mismatch |
| `liveness_passed` | Anti-spoof + presence thresholds |
| `identity_matched` | `true`/`false` when a reference photo was supplied; **`null` if not requested** |
| `identity.error_code` | `identity_not_requested` when no reference photo |
| `verified` | **`true` only if identity matched AND liveness passed**. Without a reference photo, `verified` stays **`false`** even when `status=succeeded` and liveness passed |
| `analysis_completeness` | `biometric_only` (no extras) · `full` (requested extras OK) · `partial` (extras failed or segments missing) |
| `transcript` / `report` | `null` when the corresponding flag is `false` |

Report / highlights **do not invent** spoken text. Excerpts and `transcript_excerpt` come only from STT segments/text. Local report follow-ups mention transcript accuracy only when segments exist.

## Example (core — backward compatible)

```json
{
  "interview_id": "int-vid-1",
  "video_base64": "<mp4/webm/avi>",
  "sample_fps": 1.0,
  "registered_photo_base64": "<optional enrolled photo>",
  "consent_given": true,
  "consent_id": "consent-vid-1"
}
```

## Example (evidence extras)

```json
{
  "interview_id": "int-vid-2",
  "source_url": "https://example.com/interview.mp4",
  "sample_fps": 1.0,
  "max_frames": 46,
  "consent_given": true,
  "include_transcript": true,
  "include_report": true,
  "interview_topics": ["Tell me about a conflict"],
  "options": {
    "transcript_provider": "stub",
    "stub_transcript": "I led the migration last quarter.",
    "stub_segments": [
      { "start_ms": 0, "end_ms": 2000, "text": "I led the migration last quarter." }
    ]
  }
}
```

## Env

| Variable | Purpose |
|----------|---------|
| `VIDEO_MAX_UPLOAD_BYTES` | Max upload / download size |
| `VIDEO_MAX_FRAMES` | Default frame cap (default **16**) |
| `VIDEO_MAX_DURATION_SECONDS` | Max duration |
| `VIDEO_ALIGN_WINDOW_MS` | Transcript↔visual proximity window |
| `VIDEO_FFMPEG_TIMEOUT_SECONDS` | Demux timeout |
| `VIDEO_TRANSCRIPT_MAX_AUDIO_BYTES` | Max demuxed WAV size |
| `VIDEO_ALLOW_LOCAL_SOURCE` | Dev-only local/`file://` sources |
| `FFMPEG_PATH` | System ffmpeg binary |
| `FACE_LIVENESS_SCORE_THRESHOLD` | Default **0.50** (mean over face-detected frames) |
| `FACE_LIVENESS_FRAME_PASS_RATIO` | Default **0.67** (among scored frames) |
| `FACE_MIN_PRESENCE_RATIO` | Default **0.67** (fraction of samples with a face) |
| `OPENAI_API_KEY` / transcription model settings | Required for live OpenAI/Whisper STT |

Live interactive docs: `http://localhost:8000/docs` (FastAPI reflects current Pydantic schemas). Regenerate committed swagger with the snippet in `docs/swagger/README.md` after schema changes.
