# TJ AI / AI Interviewer Pro — AI/ML Final Evaluation Report

**Date:** 2026-10-22  
**Auditor role:** Senior AI/ML Engineer / Backend Architect / QA (read-only)  
**Scope:** AI/ML gateway services under `app/` (Python / FastAPI)  
**Primary sources:** source code, `docs/SOW_AI_COMPLIANCE.md`, `docs/API_NAMES_FROM_PDF.md`, OpenAPI (`openapi.yml` / live FastAPI), tests  
**SRS/SOW v2.0 PDF:** not present in-repo; compliance mapped via `docs/SOW_AI_COMPLIANCE.md` + API name map  

**Method:** Trace endpoint → service → provider/persistence → tests. No production writes. No paid live provider soak tests. Secrets not reproduced.

**Counting rule:** One row per discrete service capability in Section 4 (plus discovered extras). Status is mutually exclusive. LLM paths that call `generate_json` with a working key are treated as **implemented with graceful local fallback**, not “stub-only,” unless the code **never** calls the live LLM.

---

## Section 1: Executive Summary

| Category | Count | How counted |
|----------|------:|-------------|
| Services audited | **42** | Section 4 checklist + manager-clone/voice/ID/coding/proctoring/etc. |
| Fully implemented (usable, real integrations or deterministic design) | **22** | End-to-end code path + tests or clear offline verification |
| Partially implemented | **14** | Works but missing SOW-grade AI, persistence, or advertised CV features |
| Failed / incorrect vs stated intent | **2** | Behavior contradicts “AI” screening claims or broken persistence contract |
| Missing / not wired | **3** | YOLO, multimodal Gemini video, eval-monitoring route |
| Blocked for live-provider verification | **1** | Full `test_stubs` suite hangs when live API keys present (isolation gap) |

**Overall readiness:** **READY WITH LIMITATIONS**

This gateway is a **broad, SOW-aligned API surface** with a consistent pattern: live LLM/embeddings/STT/TTS/DeepFace when configured, otherwise deterministic local fallbacks (`stub=True`). It is **not** a multi-tenant HR system of record (Platform/NestJS owns catalogs). Several “AI” endpoints are **heuristic-only** or **bypass LLM**. Manager clone and market/talent features overstate autonomy. Video/face are real CV but **not** YOLO/multimodal vision.

### Five most important findings

1. `POST /resume-match/score` **always returns** `stub=True` **local scoring** — `generate_json` is imported but unused (`app/services/screening/resume_match.py`). SOW “AI screening” is not LLM-backed.
2. **Manager clone** `create` **does not call** `save_manager_clone` — returns a new UUID without persisting the clone row; not attached to interview sessions (`enterprise.py` vs `manager_store.py`).
3. **Case-study / scenario *generate* always bypass LLM** — forced `local()` (`case_studies.py`, `scenarios.py`); grade paths can use LLM.
4. **Video intelligence honestly omits YOLO and multimodal Gemini** — code/docs state face/OpenCV/DeepFace only (`video/events.py`, `VIDEO_INTELLIGENCE.md`).
5. **Test isolation risk:** broad `tests/test_stubs.py` can hang with live OpenAI keys; offline-safe suites pass (91 tests in focused runs with keys cleared).

---

## Section 2: Service-by-Service Status Matrix

**Legend:** FULL = implemented; PARTIAL = gaps; FAIL = incorrect/broken contract; MISSING = not in code; BLOCKED = could not fully verify live path.

| Service | Status | Endpoint(s) | Implementation Evidence | Model / Library | Tests Performed | Gaps / Risks |
|---|---|---|---|---|---|---|
| A1 Resume / doc parse | FULL | `POST /documents/parse`, `GET /documents/resume-profile` | `DocumentsService` + **`ResumeService`** (versioned `candidate_resume_profiles`) | Local extract + LLM/local structure | `test_resume_service.py` **PASS** | Prefer `candidate_id` for reuse across flows |
| A2 JD parse | FULL | same (`parse_target=job_description`) | `structure_job_description` + LLM enrich | Same | Same | Relies on request payload, not job catalog |
| A3 Embeddings / semantic match | DISABLED | `POST /semantic-skill-match` *(commented out)* | Service remains internal | — | route 404 (stub) | HTTP removed |
| B1 Question generator | FULL | `/questions/generate`, workflows `questions/generate` | `QuestionsService` (from-role internal) | LLM + local | `test_questions_from_role_resume.py` **PASS** | Public `/from-role` removed |
| B2 AI interview engine | REMOVED (HTTP) | `/interviews/sessions/*` removed; Platform owns live session | Legacy `InterviewSessionService` + `interview_sessions` table kept but unused by APIs | — | session flow tests deleted | Prefer `final-evaluation` (stateless) |
| B3 Answer eval / scorecards | FULL | `/interviews/final-evaluation` (internal rubric), `/scorecards/generate` *(commented out)*, `/answers/evaluate` | `InterviewsService.evaluate_answer` (internal), scorecards helpers | LLM rubric + calibration / local dims | `test_evaluate_answer_rubric.py` **PASS** | Per-answer HTTP route removed; scorecards route kept but unregistered |
| B4 Manager clone / persona | PARTIAL / FAIL-persist | `/manager-clone`, personas | `ManagerCloneService.create` skips `save_manager_clone`; versions SQLAlchemy | LLM style_summary + local | stubs + `test_manager_clone_version.py` | Create not persisted; not wired into sessions; RAG placeholder |
| C1 STT | FULL | `/transcription`, session `audio_base64` | `OpenAISTTProvider` / `StubSTTProvider` | OpenAI gpt-4o-mini-transcribe / whisper-1 | voice flow + segment tests **PASS** (keys cleared → stub) | Needs key for production quality |
| C2 Audio analyzer | FULL | `/communication-analysis` (optional audio; Librosa internal) | Librosa features + local scorers | Librosa + FFmpeg | `test_audio_analysis.py` **PASS** | Standalone audio HTTP removed |
| C3 TTS | FULL | `/tts/synthesize` | `synthesize_speech` OpenAI | gpt-4o-mini-tts | `test_tts_voice_clone.py` **PASS** | No local TTS fallback |
| C4 Voice cloning | FULL | manager-clones create/PATCH/`voice.generate_voice` | OpenAI custom voice from **uploaded** sample + consent | OpenAI custom voice | TTS/clone tests **PASS** (mocked) | Builtin names (coral) not used for clone setup |
| D1 Video intelligence | PARTIAL | `/video-intelligence` | OpenCV frames + DeepFace + optional STT align + text report | OpenCV, DeepFace, FFmpeg, text LLM | `test_video_phase2_security.py` **PASS**; VI suite not fully re-run this session | **No YOLO; no multimodal Gemini** |
| D2 Face enroll + verify | FULL | `/face-verification/enroll`, `/verify` | Live enroll (encrypted) + ArcFace match | DeepFace/TF | face unit tests **PASS** | `/manual-review` disabled |
| D3 Liveness / anti-spoof | FULL | same engine | MiniFASNet-class via DeepFace + static-replay gates | DeepFace anti-spoof | `test_face_engine_liveness.py` **PASS** | Model limits; not legal ID proof |
| D4 Behavioral analysis | DISABLED | `/behavioral-analysis` *(commented out)* | Service may remain internal | — | route 404 (stub) | HTTP removed |
| E1 Job matching | FULL (rank only) | `/job-matching/recommend` | Platform DB → `job_catalog` (required); ResumeService + LLM/local overlap; no placeholders | LLM/local | stub + empty-catalog 422 | Platform must query jobs and send catalog |
| E2 Assessments suite | PARTIAL | assessments/aptitude/puzzles/coding/case/scenario | Mix of LLM, auto-grade, **forced local generate** for case/scenario | LLM/local/exec | scoring tests exist; grade gaps for case/scenario | Case/scenario generate always stub |
| E3 Skill gap | FULL | `/skill-gap/analyze` | LLM + local gap math | LLM/local | stubs | Request-supplied skills, not DB |
| E4 Hiring fit / candidate performance | FULL | `/hiring-fit/predict`, `/candidates/{id}/performance*` | LLM/local; hiring fit forces human review | LLM/local | `test_final_evaluation` hiring-fit assert **PASS** | Not auto-hire; combined `/candidate-scoring` removed |
| E5 Market research / talent scout | PARTIAL | `/market-research/questions`, `/talent-scouting/rank` | Text LLM/local | LLM/local | limited tests | **No live talent DB / web crawl** |
| F1 Bias / DEI | PARTIAL | `/bias-detection/analyze`, `/dei-analytics/analyze` | Phrase lists / sample heuristics + LLM | LLM/local | stubs | Not statistical fairness audits |
| F2 Reports / analytics | FULL | `/reports/*`, interview-analytics, highlights | PDF ReportLab; analytics deterministic | LLM + ReportLab / local | PDF + analytics tests | Reports dir filesystem |
| F3 LLM orchestration | FULL | shared | `app/llm.py` `chat_json`; `generate_json` fallback | OpenAI/Anthropic/Gemini | Indirect | Keys required for live |
| ID verification | DISABLED | `/id-verification/verify` *(commented out)* | Was OCR + authenticity | — | route 404 | Replaced by face enroll/verify |
| Proctoring | REMOVED | — | Platform-owned | — | — | Gateway route removed |
| Coding judge | FULL | `/coding/judge` | Optional local Python exec + LLM | Local exec + LLM | assessment scoring tests | Sandbox limits |
| Reference checks | FULL | `/reference-checks?action=…` | Questionnaire + submit_response store + report | LLM/local | `test_reference_checks.py` | No outbound email |
| Interview workflows/tracks | FULL | `/interview-workflows…`, `/interview-tracks/generate` | LLM + local + persisted workflow | LLM/local | workflow tests | `/interview-preview/*` removed |
| Translate / cultural | FULL | `/translate`, `/cultural-sensitivity/adapt` | LLM + local | LLM/local | stubs | Quality depends on provider |
| Confidence scoring | MERGED | `/communication-analysis` (`confidence_score` null without audio) | Audio delivery indicators only | Librosa | `test_voice_analysis.py` | Never from text alone |
| Communication analysis | FULL | `/communication-analysis` | Text dims + optional audio/confidence; one GPT pass | Local/LLM + Librosa | dedicated tests **PASS** | Replaces audio + confidence HTTP |
| Progress analytics | PARTIAL | `/progress-analytics/analyze` | Aggregation over supplied data | LLM/local | stubs | Needs Platform history |
| Badges / profiles | PARTIAL | `/badges/recommend`, `/candidate-profiles/summarize` | LLM/local summarize | LLM/local | stubs | No profile store |
| Prep plans | FULL | `/prep-plans/generate` | LLM + local | LLM/local | stubs | — |
| YOLO object detect | MISSING | (VI events) | Explicitly not wired | — | N/A | Documented absence |
| Gemini multimodal video | MISSING | (VI report) | Text LLM only | Gemini text JSON only | N/A | Documented absence |
| Eval monitoring route | MISSING | commented in `router.py` | Stubs commented out | — | N/A | Intentionally unregistered |

---

## Section 3: End-to-End Workflow Findings

### Workflow 1: Resume → Job Matching

**Implemented sequence:** `POST /documents/parse` with `candidate_id` → versioned `candidate_resume_profiles` → `resume-match` / `prep-plans` / `job-matching` / workflow questions via **ResumeService** (same content_hash reused; no re-parse).  
**Also:** `GET /documents/resume-profile?candidate_id=` for latest structured resume.  
**Completes?** Screening composition with durable resume profile; Platform still owns CRM.  
**Reproduce:** parse twice with same `candidate_id` (second `resume_reused=true`); match with only `candidate_id` + JD.

### Workflow 2: AI Interview

**Sequence:** Interview Design (`mode` text|voice|video + questions/preview) → Platform live session → `final-evaluation` (stateless; internal per-answer scoring).  
**Persistence:** Design workflows in DB; live session owned by Platform.  
**Tested:** `test_interview_modes_eval.py`, `test_final_evaluation.py`, workflows/rubric/face **PASS**.  
**Gaps:** media metrics require Platform-supplied evidence; no fabricated A/V scores.

### Workflow 3: Voice Interview

**Sequence:** session answer with `audio_base64` → STT → evaluate → continue; optional `/tts/synthesize`.  
**Tested:** `test_interview_voice_flow.py`, TTS/clone tests **PASS**.  
**Gaps:** prompts remain text by default; streaming not implemented.

### Workflow 4: Video Interview

**Sequence:** consent → video bytes/URL → OpenCV sample → DeepFace → optional demux/STT/align → text report.  
**Missing:** YOLO, multimodal Gemini, async job status API.  
**Security:** URL SSRF guards (`url_safety`) tested in `test_video_phase2_security.py` **PASS**.

### Workflow 5: Assessment → Skill Gap

**Sequence:** generate → grade (key auto or LLM) → client feeds results into `/skill-gap/analyze`.  
**Gaps:** case/scenario generate always local; no automatic wiring from grade → skill-gap store.

### Workflow 6: Candidate Scoring & Fairness

**Sequence:** client aggregates evidence → final-evaluation / `/hiring-fit/predict` → persist → `/candidates/{id}/performance`*; optional `/bias-detection`.  
**Gaps:** bias/DEI are not rigorous fairness audits; hiring fit correctly forces human review.

### Workflow 7: Manager Persona & Voice

**Sequence:** personas catalog + manager-clone style summary + separate voice-cloning/TTS.  
**Broken link:** clone create not persisted; not injected into `InterviewSessionService`.  
**Voice:** separate OpenAI custom/builtin path with consent for custom.

---

## Section 4: Endpoint and Integration Coverage

- **Registered operations (local FastAPI):** **45** across **40** paths (`app.routes.router` / OpenAPI).  
- **Staging** `openapi.yml` **snapshot:** **74** (older deploy).  
- **Auth:** `X-Internal-Token` on essentially all non-health routes (`app/deps.py`).  
- **Persistence islands:** interview JSON files; voice profile JSON; face/ID audits; manager-clone Postgres (versions); report PDFs on disk. **No unified candidate/job ORM.**  
- **Shared AI entry:** `generate_json` → `chat_json` (OpenAI → Anthropic → Gemini).  
- **Routes without deep AI:** resume-match (heuristic), case/scenario generate (forced local), proctoring (event list), analytics (deterministic).  
- **Undocumented vs aspirational:** YOLO/multimodal still appear in product discussions but are **explicitly absent** in code.  
- `eval_monitoring`**:** intentionally not registered.

---

## Section 5: AI/ML Quality and Correctness

| Area | Observation |
|------|-------------|
| Suitability | Rubric eval + session orchestration are solid for an AI gateway; talent/market/bias names oversell. |
| Validation | Pydantic schemas generally strong; face/ID consent enforced. |
| Scores | Interview dimensions calibrated; resume-match is keyword-ish overlap. Embeddings may fall back to hash vectors. |
| Reproducibility | Local factories deterministic; live LLM not. |
| Fallback | Widespread `stub=True` — good reliability, risk of silent quality drop if keys missing. |
| Explainability | Many responses include summaries/matched skills; behavioral/confidence labels need careful product wording. |
| Fairness | Bias/DEI endpoints ≠ production fairness program. |
| Biometrics | Face/liveness real DeepFace; still not legal identity assurance; ID authenticity is heuristic. |

---

## Section 6: Security, Privacy, Reliability

| Topic | Finding |
|-------|---------|
| Auth | Internal token required; no public bypass after demo-mode revert |
| Secrets | Settings from `.env`; do not log tokens (verify logging hygiene in ops) |
| Consent | Face/ID/video require `consent_given`; custom voice needs consent artifacts |
| SSRF | Video HTTP sources validated (`video/url_safety.py`) |
| Resource limits | Video/audio byte/duration/frame caps in config |
| Temp cleanup | Video temp retention helpers exist |
| Code exec | Coding judge local Python — needs sandbox hardening for untrusted code |
| PII | Encrypted ID image store option; face audits JSON — retention policy Platform-owned |
| Reliability | LLM timeouts in session path; CV lazy-loaded to protect startup |

---

## Section 7: Prioritized Defect List

| ID | Sev | Service | Observed | Expected | Evidence | Fix | Regression |
|----|-----|---------|----------|----------|----------|-----|------------|
| F-01 | High | Resume match | Always `stub=True`, no `generate_json` call | SOW AI screening or honest “heuristic” naming | `resume_match.py` L165–184 | Wire LLM or document as deterministic | API test asserts provider path |
| F-02 | High | Manager clone | `create` never `save_manager_clone` | Persist clone for version APIs | `enterprise.py` L37–67 vs `manager_store.save_manager_clone` | Persist on create; wire to sessions or narrow docs | Version tests with create |
| F-03 | Medium | Case/scenario generate | Forced local bypass | LLM generate like assessments **or** document | `case_studies.py` / `scenarios.py` | Enable `generate_json` or mark stub-only in SOW | Generate tests with provider mock |
| F-04 | Medium | VI marketing | YOLO/Gemini multimodal assumed | Match `VIDEO_INTELLIGENCE.md` | `events.py` L1–5 | Align product claims | Doc review checklist |
| F-05 | Medium | Tests | `test_stubs` can hang with live keys | Offline by default | Observed hang this audit | Autouse mock `generate_json`/`chat_json` | CI without network |
| F-06 | Medium | Talent/market | Text generation only | Live sources or rename | enterprise talent/market services | Integrate Platform data or label “synthetic” | Contract tests |
| F-07 | Low | Bias/DEI | Phrase heuristics | Statistical fairness or disclaimer | bias/dei services | Scope docs + sample-size gates | Fairness fixture tests |
| F-08 | Low | Proctoring | Client events only | Capture agent or rename | `reports/proctoring.py` | Clarify API contract | — |
| F-09 | Low | RAG | In-memory placeholder | Vector store or remove claim | `enterprise/rag.py` | Implement or document experimental | — |
| F-10 | Info | Staging OpenAPI | Keep staging in sync with local **45** ops | Deploy parity | `openapi.yml` vs local FastAPI | Redeploy staging | OpenAPI diff CI |

---

## Section 8: Final Readiness Decision

| Dimension | Verdict |
|-----------|---------|
| Code implementation | **Strong** breadth; several honesty gaps |
| Functional correctness | **Good** for interview/voice/face/ID/audio with deps |
| Integration readiness | **Partial** — Platform must supply catalogs/jobs/candidates |
| Test coverage | **Good** focused suites; **weak** live-key isolation for stubs |
| Security/privacy | **Adequate** for internal gateway; biometrics need ops policy |
| Production readiness | **READY WITH LIMITATIONS** |

**Overall: READY WITH LIMITATIONS**

Ship as an **AI Intelligence Gateway** behind Platform with documented limitations. Do **not** claim full SOW “Done” for resume LLM screening, manager digital twin, YOLO video, or live talent markets without fixes above.

---

## Section 9: Remediation Roadmap

### Phase 1 — Critical honesty & data integrity

- F-01 resume-match LLM or rename  
- F-02 persist manager clone / fix API contract  
- F-05 test isolation for CI

### Phase 2 — Core interview & evaluation

- Keep session/eval/final-eval; add optional auto final-eval on complete  
- Ensure STT/TTS keys monitored in staging

### Phase 3 — Audio / video / face

- Keep DeepFace path; calibrate liveness on real spoof set  
- Do not promise YOLO/multimodal until built  
- Face/ID consent + retention runbooks

### Phase 4 — Assessments / matching / reporting

- Enable or formally stub case/scenario generate  
- Job-match: Platform job DB → `job_catalog` → Gateway rank (`docs/JOB_MATCHING.md`)  
- PDF/report retention

### Phase 5 — Fairness, security, observability

- Reframe bias/DEI endpoints  
- Coding judge sandbox  
- Metrics (`/metrics`) + LLM latency logging without prompt leakage  
- Staging/OpenAPI deploy parity

---

## Section 10: Reproduction Commands

```bash
cd /home/esfera/Desktop/Arshdeep/Projects/tj
source venv/bin/activate

# Offline-safe focused suites (verified this audit)
OPENAI_API_KEY= ANTHROPIC_API_KEY= GOOGLE_API_KEY= \
  python -m pytest \
  tests/test_face_engine_liveness.py \
  tests/test_face_portrait.py \
  tests/test_face_verification.py \
  tests/test_video_phase2_security.py \
  tests/test_lazy_cv_startup.py \
  tests/test_final_evaluation.py \
  tests/test_manager_clones_rest.py \
  tests/test_evaluate_answer_rubric.py \
  tests/test_audio_analysis.py \
  tests/test_tts_voice_clone.py \
  tests/test_questions_from_role_resume.py \
  -q

# App + OpenAPI
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 1
# Docs: http://localhost:8000/docs
# Standalone staging spec: openapi.yml → https://editor.swagger.io/

# Auth header (do not print token):
# curl -sS http://localhost:8000/api/v1/health
# curl -sS -H "X-Internal-Token: $INTERNAL_API_TOKEN" "http://localhost:8000/api/v1/personas"
# optional filter: ?company_id=co-acme-001
```

**Test evidence this session:**  

- 42 passed (face/ID/semantic/video-security/lazy-cv)  
- 49 passed (interview/final-eval/rubric/audio/TTS/questions with keys cleared)  
- Full `test_stubs.py` **BLOCKED**/hung when live keys likely present — treat as environment limitation, not suite green.

---

## Concise Verdict

| Genuinely complete | Incomplete / overstated | Blocked |
|--------------------|-------------------------|---------|
| Interview session loop, rubric eval, final-eval scaffolding | Resume-match “AI”, manager clone persistence/session bind | Live paid LLM soak / full stubs suite with keys |
| STT/TTS/voice-clone paths (with keys), Librosa audio | Case/scenario generate (forced stub) | DeepFace on hosts without TF (deps_unavailable) |
| Face/ID/video (OpenCV+DeepFace as documented) | YOLO, multimodal Gemini, vector DB, live talent market | — |
| Documents parse, questions, most LLM+fallback services | Bias/DEI/proctoring naming | — |

### Next three highest-priority actions

1. Fix or re-label **resume-match** (F-01).
2. Persist **manager clone** on create and document session non-integration (F-02).
3. Force **offline mocks** in CI for `test_stubs` (F-05), then re-run full suite.

---

*End of audit. No code fixes applied in this pass.*
