# Paytm Statement Chatbot Accuracy Eval

**PDF:** `Paytm_UPI_Statement_03_Jul'26_-_02_Aug'26.pdf`
**Agent:** `5ea8368b-8c42-45e2-abe8-0f7ad27a2f7a`
**Cases:** 25 | **Pass+soft:** 25 (100.0%) | **Hard pass:** 25 (100.0%) | **Fail:** 0
**Avg latency:** 2.7s

## Ground truth (from PDF header / extract)

- Period: **3 JUL'26 - 2 AUG'26**
- Total Money Paid: **₹ 30,298.30**
- Total Money Received: **₹ 7,729.00**
- Payments made / received: **84 / 5**
- Largest outflow: **₹ 4,000.00** → Ojas Verma
- 16 Jul outflows (indexed): **₹ 4,197.00**
- Zepto total (indexed): **₹ 1,646.00** | Blinkit: **₹ 336.00**

## Results by case

| ID | Cat | Verdict | Lat | Mode | Question | Notes |
|----|-----|---------|-----|------|----------|-------|
| R1 | routing | **pass** | 0.7s | conversational | hi | ok |
| R2 | routing | **pass** | 0.59s | conversational | how are you | ok |
| R3 | routing | **pass** | 0.52s | conversational | how can you help me | ok |
| R4 | routing | **pass** | 0.61s | calculator | What is 25*4? | ok |
| P1 | in_pdf | **pass** | 0.52s | pdf | How much did I spend? | ok |
| P2 | in_pdf | **pass** | 0.6s | pdf | How much money did I receive? | ok |
| P3 | in_pdf | **pass** | 0.53s | pdf | What is the statement period? | ok |
| P4 | in_pdf | **pass** | 0.59s | pdf | How many payments were made? | ok |
| P5 | in_pdf | **pass** | 0.54s | pdf | What was my largest payment? | ok |
| P6 | in_pdf | **pass** | 0.66s | pdf | How much did I spend on 16 July? | ok |
| P7 | in_pdf | **pass** | 0.75s | pdf | How much spent on Zepto? | ok |
| P8 | in_pdf | **pass** | 0.81s | pdf | How much spent on Blinkit? | ok |
| P9 | in_pdf | **pass** | 0.56s | pdf | How much did I pay Nykaa? | ok |
| P10 | in_pdf | **pass** | 0.61s | pdf | How much did I spend on Myntra? | ok |
| P11 | in_pdf | **pass** | 0.55s | pdf | How much money sent to Aditya Yadav? | ok |
| P12 | in_pdf | **pass** | 0.62s | pdf | How much was the Jio recharge? | ok |
| P13 | in_pdf | **pass** | 0.53s | pdf | Whose statement is this? | ok |
| P14 | in_pdf | **pass** | 0.69s | pdf | How much spent on groceries? | ok |
| P15 | in_pdf | **pass** | 0.53s | pdf | Show my spending summary | ok |
| P16 | in_pdf | **pass** | 0.6s | pdf | How many payments were received? | ok |
| O1 | outside | **pass** | 11.32s | conversational | What is EBITDA? | ok |
| O2 | outside | **pass** | 0.59s | conversational | What is the weather in Delhi today? | ok |
| O3 | outside | **pass** | 0.53s | conversational | How much did Elon Musk spend last month? | ok |
| O4 | outside | **pass** | 32.89s | rag | Did I pay Amazon ₹50,000 on this stateme | denied amazon payment |
| O5 | outside | **pass** | 11.14s | conversational | Explain depreciation in accounting | ok |

## Failures (detail)

_None._
## Where the chatbot is good

- Routing: greetings/help/math stay conversational and do not dump the statement.
- Core statement totals (spend / receive / period / large payments) mostly match PDF header (16/16 in-PDF cases not hard-failing).
- Outside/concept questions did not paste Paytm spend totals.
- Latency acceptable for this suite (avg 2.7s).
- Structured path correctly prefers statement header total paid (₹30,298.30) for overall spend.

## Where it is lacking

- Indexed dated outflows (~40–42) are fewer than header “84 payments made” — line-item coverage is incomplete; header totals should be preferred for overall spend.

## How to re-run

```bash
.venv/bin/python scripts/eval_paytm_accuracy.py
```