{ "ground_truth": { "account_holder": "AAYUSHI VERMA", "period": "3 JUL'26 - 2 AUG'26", "total_paid": 30298.3, "total_received": 7729.0, "payments_made": 84, "payments_received": 5, "largest_out": 4000.0, "largest_payee": "Ojas Verma", "jul16_out": 4197.0, "zepto_out": 1646.0, "blinkit_out": 336.0, "nykaa": 993.0, "myntra": 478.0, "aditya_total": 3001.0, "jio_recharge": 300.8, "phone": "6397352957" }, "agent_id": "5ea8368b-8c42-45e2-abe8-0f7ad27a2f7a", "docs": [ { "filename": "Paytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf", "status": "ready" }, { "filename": "Paytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf", "status": "ready" } ], "summary": { "total": 25, "pass": 25, "soft_pass": 0, "fail": 0, "pass_rate_incl_soft": 100.0, "hard_pass_rate": 100.0, "avg_latency_s": 2.74, "by_category": { "routing": { "pass": 4 }, "in_pdf": { "pass": 16 }, "outside": { "pass": 5 } }, "elapsed_s": 72.7 }, "results": [ { "id": "R1", "category": "routing", "question": "hi", "expected": "Friendly greeting; mention KB ready if docs exist; no spend dump", "answer": "Hi \u2014 I'm Alex. Your knowledge base has 2 documents ready. Try: how much did I spend \u00b7 spend on groceries \u00b7 money received.", "mode": "conversational", "finance_intent": "general", "latency_s": 0.7, "verdict": "pass", "notes": "ok", "numbers_found": [ 2.0 ], "expected_numbers": [] }, { "id": "R2", "category": "routing", "question": "how are you", "expected": "Wellbeing reply, no statement dump", "answer": "I'm doing well, thanks! Your knowledge base is ready \u2014 ask about spend, merchants, or a date on your statement.", "mode": "conversational", "finance_intent": "general", "latency_s": 0.59, "verdict": "pass", "notes": "ok", "numbers_found": [], "expected_numbers": [] }, { "id": "R3", "category": "routing", "question": "how can you help me", "expected": "Capability help, not merchant list", "answer": "I\u2019m Alex. I can answer from \u201cPaytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf\u201d, \u201cPaytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf\u201d. Try: statement overview \u00b7 how much did I spend \u00b7 spend on 16 July \u00b7 groceries \u00b7 money received. I use your uploaded documents for personal facts (spend, merchants, dates, statements). I answer greetings and simple chat without looking up files. I explain general accounting concepts (e.g. EBITDA, depreciation) from general knowledge \u2014 not from your PDFs \u2014 unless you ask about your own numbers.", "mode": "conversational", "finance_intent": "general", "latency_s": 0.52, "verdict": "pass", "notes": "ok", "numbers_found": [ 16.0 ], "expected_numbers": [] }, { "id": "R4", "category": "routing", "question": "What is 25*4?", "expected": "100", "answer": "Result: 100", "mode": "calculator", "finance_intent": "general", "latency_s": 0.61, "verdict": "pass", "notes": "ok", "numbers_found": [ 100.0 ], "expected_numbers": [ 100.0 ] }, { "id": "P1", "category": "in_pdf", "question": "How much did I spend?", "expected": "Total paid \u2248 \u20b930,298.30 (header)", "answer": "Spending summary\n\nHere\u2019s how money left the account on this statement \u2014 all 40 indexed outflows listed below.\n\n- **Total spent:** \u20b9 30,298.30\n- **Transactions:** 84\n- **Average:** \u20b9 360.69\n- **Date range:** 3 JUL'26 - 2 AUG'26\n\n**Highlights**\n- \u20b9 4,000.00 \u00b7 Money sent to Ojas Verma\n- \u20b9 3,000.00 \u00b7 Money sent to Aditya Yadav\n- \u20b9 993.00 \u00b7 Paid to Nykaa ERetail Private Limited\n\n| Category | Amount |\n| --- | --- |\n| Upi / Digital Pay | \u20b9 12,428.00 |\n| Groceries | \u20b9 1,683.00 |\n| Food | \u20b9 386.36 |\n| Bills | \u20b9 300.80 |\n\n**Payments (40)**\n- 03 Jul \u00b7 Paid to Tea Counter Chef17 \u2014 \u20b9 150.00\n- 03 Jul \u00b7 Money sent to Savita \u2014 \u20b9 500.00\n- 04 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 159.00\n- 06 Jul \u00b7 Paid to Zepto \u2014 \u20b9 327.00\n- 07 Jul \u00b7 Paid to Myntra \u2014 \u20b9 478.00\n- 07 Jul \u00b7 Paid to Zomato Ltd \u2014 \u20b9 113.06\n- 08 Jul \u00b7 Money sent to Sanjay \u2014 \u20b9 91.00\n- 09 Jul \u00b7 Paid to SURAJ \u2014 \u20b9 40.00\n- 10 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 1.00\n- 10 Jul \u00b7 Money sent to Mohit Kumar \u2014 \u20b9 111.00\n- 10 Jul \u00b7 Money sent to Kamaldeep Singh \u2014 \u20b9 206.00\n- 11 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 177.00\n- 11 Jul \u00b7 Paid to Zepto \u2014 \u20b9 326.00\n- 12 Jul \u00b7 Paid to Meesho Technologies Private Limited \u2014 \u20b9 774.00\n- 13 Jul \u00b7 Money sent to Ramesh Kumar \u2014 \u20b9 103.00\n- 13 Ju", "mode": "pdf", "finance_intent": "spending_summary", "latency_s": 0.52, "verdict": "pass", "notes": "ok", "numbers_found": [ 40.0, 30298.3, 84.0, 360.69, 3.0, 26.0, 2.0, 26.0, 4000.0, 3000.0, 993.0, 12428.0, 1683.0, 386.36, 300.8, 40.0, 3.0, 150.0, 3.0, 500.0 ], "expected_numbers": [ 30298.3 ] }, { "id": "P2", "category": "in_pdf", "question": "How much money did I receive?", "expected": "Total received \u2248 \u20b97,729.00", "answer": "Money received\n\nCredits and inflows recorded on this statement.\n\n- **Total received:** \u20b9 7,729.00\n- **Credits:** 5\n- **Date range:** 3 JUL'26 - 2 AUG'26\n\n**Payments**\n- 17 Jul \u00b7 Received from Nitin Kumar \u2014 \u20b9 100.00\n- 02 Aug \u00b7 Received from Yashas X \u2014 \u20b9 528.00\n\n_Showing all 2 credits from the statement._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "income", "latency_s": 0.6, "verdict": "pass", "notes": "ok", "numbers_found": [ 7729.0, 5.0, 3.0, 26.0, 2.0, 26.0, 17.0, 100.0, 2.0, 528.0, 2.0, 40.0, 84.0 ], "expected_numbers": [ 7729.0 ] }, { "id": "P3", "category": "in_pdf", "question": "What is the statement period?", "expected": "3 JUL'26 - 2 AUG'26", "answer": "Statement period\n\nThis statement covers 3 JUL'26 - 2 AUG'26.\n\n- **Period:** 3 JUL'26 - 2 AUG'26\n- **Account holder:** Aayushi Verma\n\n_Taken from the statement header date range._", "mode": "pdf", "finance_intent": "statement_summary", "latency_s": 0.53, "verdict": "pass", "notes": "ok", "numbers_found": [ 3.0, 26.0, 2.0, 26.0, 3.0, 26.0, 2.0, 26.0 ], "expected_numbers": [] }, { "id": "P4", "category": "in_pdf", "question": "How many payments were made?", "expected": "84 payments on header", "answer": "Payment count\n\nThis statement shows 84 payments made and 5 received.\n\n- **Payments made:** 84\n- **Payments received:** 5\n- **Period:** 3 JUL'26 - 2 AUG'26\n\n_Counts prefer the statement header when available._", "mode": "pdf", "finance_intent": "payment_count", "latency_s": 0.59, "verdict": "pass", "notes": "ok", "numbers_found": [ 84.0, 5.0, 84.0, 5.0, 3.0, 26.0, 2.0, 26.0 ], "expected_numbers": [ 84.0 ] }, { "id": "P5", "category": "in_pdf", "question": "What was my largest payment?", "expected": "\u20b94,000 to Ojas Verma", "answer": "Spending summary\n\nHere\u2019s how money left the account on this statement \u2014 all 40 indexed outflows listed below.\n\n- **Total spent:** \u20b9 30,298.30\n- **Transactions:** 84\n- **Average:** \u20b9 360.69\n- **Date range:** 3 JUL'26 - 2 AUG'26\n\n**Highlights**\n- \u20b9 4,000.00 \u00b7 Money sent to Ojas Verma\n- \u20b9 3,000.00 \u00b7 Money sent to Aditya Yadav\n- \u20b9 993.00 \u00b7 Paid to Nykaa ERetail Private Limited\n\n| Category | Amount |\n| --- | --- |\n| Upi / Digital Pay | \u20b9 12,428.00 |\n| Groceries | \u20b9 1,683.00 |\n| Food | \u20b9 386.36 |\n| Bills | \u20b9 300.80 |\n\n**Payments (40)**\n- 03 Jul \u00b7 Paid to Tea Counter Chef17 \u2014 \u20b9 150.00\n- 03 Jul \u00b7 Money sent to Savita \u2014 \u20b9 500.00\n- 04 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 159.00\n- 06 Jul \u00b7 Paid to Zepto \u2014 \u20b9 327.00\n- 07 Jul \u00b7 Paid to Myntra \u2014 \u20b9 478.00\n- 07 Jul \u00b7 Paid to Zomato Ltd \u2014 \u20b9 113.06\n- 08 Jul \u00b7 Money sent to Sanjay \u2014 \u20b9 91.00\n- 09 Jul \u00b7 Paid to SURAJ \u2014 \u20b9 40.00\n- 10 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 1.00\n- 10 Jul \u00b7 Money sent to Mohit Kumar \u2014 \u20b9 111.00\n- 10 Jul \u00b7 Money sent to Kamaldeep Singh \u2014 \u20b9 206.00\n- 11 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 177.00\n- 11 Jul \u00b7 Paid to Zepto \u2014 \u20b9 326.00\n- 12 Jul \u00b7 Paid to Meesho Technologies Private Limited \u2014 \u20b9 774.00\n- 13 Jul \u00b7 Money sent to Ramesh Kumar \u2014 \u20b9 103.00\n- 13 Ju", "mode": "pdf", "finance_intent": "spending_summary", "latency_s": 0.54, "verdict": "pass", "notes": "ok", "numbers_found": [ 40.0, 30298.3, 84.0, 360.69, 3.0, 26.0, 2.0, 26.0, 4000.0, 3000.0, 993.0, 12428.0, 1683.0, 386.36, 300.8, 40.0, 3.0, 150.0, 3.0, 500.0 ], "expected_numbers": [ 4000.0 ] }, { "id": "P6", "category": "in_pdf", "question": "How much did I spend on 16 July?", "expected": "\u2248 \u20b94,197.00 across all 16 Jul outflows on the PDF", "answer": "Spend on 16 Jul\n\nOn 16 Jul, \u20b9 4,197.00 went out across 5 payments. All 5 listed below.\n\n- **Total spent:** \u20b9 4,197.00\n- **Payments:** 5\n- **Average:** \u20b9 839.40\n- **Date:** 16 Jul\n\n**Highlights**\n- Largest on this day: \u20b9 3,000.00 \u2014 Money sent to Aditya Yadav\n- Average payment: \u20b9 839.40\n\n**Payments**\n- 16 Jul \u00b7 Paid to Nykaa ERetail Private Limited \u2014 \u20b9 993.00\n- 16 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 3,000.00\n- 16 Jul \u00b7 Money sent to Bhairab Pathak \u2014 \u20b9 114.00\n- 16 Jul \u00b7 Money sent to Jatin Thakur \u2014 \u20b9 89.00\n- 16 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 1.00\n\n_Showing all 5 payments from the statement._\n_Scoped to outflows dated 16 Jul on \u201cPaytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf\u201d._\n_Totals use the full statement text (table extract alone was incomplete for this day)._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "date_lookup", "latency_s": 0.66, "verdict": "pass", "notes": "ok", "numbers_found": [ 16.0, 16.0, 4197.0, 5.0, 5.0, 4197.0, 5.0, 839.4, 16.0, 3000.0, 839.4, 16.0, 993.0, 16.0, 3000.0, 16.0, 114.0, 16.0, 89.0, 16.0 ], "expected_numbers": [ 4197.0 ] }, { "id": "P7", "category": "in_pdf", "question": "How much spent on Zepto?", "expected": "\u2248 \u20b91,646.00 across Zepto payments", "answer": "Zepto spend\n\nSpending on zepto: \u20b9 1,646.00 across 6 payments. Every matching payment is listed below.\n\n- **Category total:** \u20b9 1,646.00\n- **Payments:** 6\n- **Average:** \u20b9 274.33\n\n**Highlights**\n- \u20b9 360.00 \u00b7 Paid to Zepto\n- \u20b9 348.00 \u00b7 Paid to Zepto\n- \u20b9 327.00 \u00b7 Paid to Zepto\n\n**Payments**\n- 06 Jul \u00b7 Paid to Zepto \u2014 \u20b9 327.00\n- 11 Jul \u00b7 Paid to Zepto \u2014 \u20b9 326.00\n- 12 Jul \u00b7 Paid to ZEPTO \u2014 \u20b9 125.00\n- 14 Jul \u00b7 Paid to Zepto \u2014 \u20b9 160.00\n- 20 Jul \u00b7 Paid to Zepto \u2014 \u20b9 360.00\n- 02 Aug \u00b7 Paid to Zepto \u2014 \u20b9 348.00\n\n_Showing all 6 matching zepto payments._\n_Totals use the full statement text (table extract alone was incomplete)._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "category_spending", "latency_s": 0.75, "verdict": "pass", "notes": "ok", "numbers_found": [ 1646.0, 6.0, 1646.0, 6.0, 274.33, 360.0, 348.0, 327.0, 6.0, 327.0, 11.0, 326.0, 12.0, 125.0, 14.0, 160.0, 20.0, 360.0, 2.0, 348.0 ], "expected_numbers": [ 1646.0 ] }, { "id": "P8", "category": "in_pdf", "question": "How much spent on Blinkit?", "expected": "\u2248 \u20b9336.00", "answer": "Blinkit spend\n\nSpending on blinkit: \u20b9 336.00 across 2 payments. Every matching payment is listed below.\n\n- **Category total:** \u20b9 336.00\n- **Payments:** 2\n- **Average:** \u20b9 168.00\n\n**Highlights**\n- \u20b9 177.00 \u00b7 Paid to Blinkit\n- \u20b9 159.00 \u00b7 Paid to Blinkit\n\n**Payments**\n- 04 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 159.00\n- 11 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 177.00\n\n_Showing all 2 matching blinkit payments._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "category_spending", "latency_s": 0.81, "verdict": "pass", "notes": "ok", "numbers_found": [ 336.0, 2.0, 336.0, 2.0, 168.0, 177.0, 159.0, 4.0, 159.0, 11.0, 177.0, 2.0, 40.0, 84.0, 40.0, 84.0 ], "expected_numbers": [ 336.0 ] }, { "id": "P9", "category": "in_pdf", "question": "How much did I pay Nykaa?", "expected": "\u20b9993.00", "answer": "Nykaa spend\n\nSpending on Nykaa: \u20b9 993.00 across 1 payment. Every matching payment is listed below.\n\n- **Merchant total:** \u20b9 993.00\n- **Payments:** 1\n- **Average:** \u20b9 993.00\n\n**Highlights**\n- \u20b9 993.00 \u00b7 Paid to Nykaa ERetail Private Limited\n\n**Payments**\n- 16 Jul \u00b7 Paid to Nykaa ERetail Private Limited \u2014 \u20b9 993.00\n\n_Showing all 1 matching payment to Nykaa._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "merchant_analysis", "latency_s": 0.56, "verdict": "pass", "notes": "ok", "numbers_found": [ 993.0, 1.0, 993.0, 1.0, 993.0, 993.0, 16.0, 993.0, 1.0, 40.0, 84.0 ], "expected_numbers": [ 993.0 ] }, { "id": "P10", "category": "in_pdf", "question": "How much did I spend on Myntra?", "expected": "\u20b9478.00", "answer": "Myntra spend\n\nSpending on Myntra: \u20b9 478.00 across 1 payment. Every matching payment is listed below.\n\n- **Merchant total:** \u20b9 478.00\n- **Payments:** 1\n- **Average:** \u20b9 478.00\n\n**Highlights**\n- \u20b9 478.00 \u00b7 Paid to Myntra\n\n**Payments**\n- 07 Jul \u00b7 Paid to Myntra \u2014 \u20b9 478.00\n\n_Showing all 1 matching payment to Myntra._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "merchant_analysis", "latency_s": 0.61, "verdict": "pass", "notes": "ok", "numbers_found": [ 478.0, 1.0, 478.0, 1.0, 478.0, 478.0, 7.0, 478.0, 1.0, 40.0, 84.0 ], "expected_numbers": [ 478.0 ] }, { "id": "P11", "category": "in_pdf", "question": "How much money sent to Aditya Yadav?", "expected": "\u2248 \u20b93,001.00 (3000+1)", "answer": "Aditya Yadav spend\n\nSpending on Aditya Yadav: \u20b9 7,002.00 across 4 payments. Every matching payment is listed below.\n\n- **Merchant total:** \u20b9 7,002.00\n- **Payments:** 4\n- **Average:** \u20b9 1,750.50\n\n**Highlights**\n- \u20b9 4,000.00 \u00b7 Money sent to Aditya Yadav\n- \u20b9 3,000.00 \u00b7 Money sent to Aditya Yadav\n- \u20b9 1.00 \u00b7 Money sent to Aditya Yadav\n\n**Payments**\n- 10 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 1.00\n- 11 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 4,000.00\n- 16 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 3,000.00\n- 16 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 1.00\n\n_Showing all 4 matching payments to Aditya Yadav._\n_Totals use the full statement text (table extract alone was incomplete)._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "merchant_analysis", "latency_s": 0.55, "verdict": "pass", "notes": "ok", "numbers_found": [ 7002.0, 4.0, 7002.0, 4.0, 1750.5, 4000.0, 3000.0, 1.0, 10.0, 1.0, 11.0, 4000.0, 16.0, 3000.0, 16.0, 1.0, 4.0, 40.0, 84.0 ], "expected_numbers": [ 3000.0 ] }, { "id": "P12", "category": "in_pdf", "question": "How much was the Jio recharge?", "expected": "\u20b9300.80", "answer": "Bills spend\n\nSpending on bills: \u20b9 1,500.80 across 2 payments. Every matching payment is listed below.\n\n- **Category total:** \u20b9 1,500.80\n- **Payments:** 2\n- **Average:** \u20b9 750.40\n\n**Highlights**\n- \u20b9 1,200.00 \u00b7 Paid to Reliance Jio Infocomm\n- \u20b9 300.80 \u00b7 Recharge of Jio Mobile 6397352957\n\n**Payments**\n- 20 Jul \u00b7 Recharge of Jio Mobile 6397352957 \u2014 \u20b9 300.80\n- 28 Jul \u00b7 Paid to Reliance Jio Infocomm \u2014 \u20b9 1,200.00\n\n_Showing all 2 matching bills payments._\n_Totals use the full statement text (table extract alone was incomplete)._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._\n_Line-item coverage may be incomplete: 40 dated outflows indexed vs 84 payments on the statement header. Prefer header totals for overall spend._", "mode": "pdf", "finance_intent": "category_spending", "latency_s": 0.62, "verdict": "pass", "notes": "ok", "numbers_found": [ 1500.8, 2.0, 1500.8, 2.0, 750.4, 1200.0, 300.8, 6397352957.0, 20.0, 6397352957.0, 300.8, 28.0, 1200.0, 2.0, 40.0, 84.0, 40.0, 84.0 ], "expected_numbers": [ 300.8 ] }, { "id": "P13", "category": "in_pdf", "question": "Whose statement is this?", "expected": "Aayushi Verma", "answer": "Account holder\n\nThis statement belongs to Aayushi Verma.\n\n- **Account holder:** Aayushi Verma\n- **Period:** 3 JUL'26 - 2 AUG'26\n- **Document:** Paytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf\n\n_Read from the statement header / first page._", "mode": "pdf", "finance_intent": "statement_summary", "latency_s": 0.53, "verdict": "pass", "notes": "ok", "numbers_found": [ 3.0, 26.0, 2.0, 26.0 ], "expected_numbers": [] }, { "id": "P14", "category": "in_pdf", "question": "How much spent on groceries?", "expected": "Groceries category (Zepto/Blinkit/Instamart/Country Delight etc.)", "answer": "Groceries spend\n\nSpending on groceries: \u20b9 3,801.00 across 17 payments. Every matching payment is listed below.\n\n- **Category total:** \u20b9 3,801.00\n- **Payments:** 17\n- **Average:** \u20b9 223.59\n\n**Highlights**\n- \u20b9 360.00 \u00b7 Paid to Zepto\n- \u20b9 348.00 \u00b7 Paid to Zepto\n- \u20b9 334.00 \u00b7 Paid to Swiggy Instamart Private Limited\n\n**Payments (17)**\n- 04 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 159.00\n- 05 Jul \u00b7 Paid to Beejapuri Dairy Private Limited \u2014 \u20b9 200.00\n- 06 Jul \u00b7 Paid to Zepto \u2014 \u20b9 327.00\n- 11 Jul \u00b7 Paid to Blinkit \u2014 \u20b9 177.00\n- 11 Jul \u00b7 Paid to Zepto \u2014 \u20b9 326.00\n- 12 Jul \u00b7 Paid to ZEPTO \u2014 \u20b9 125.00\n- 14 Jul \u00b7 Paid to Rakhi Aggarwal \u2014 \u20b9 145.00\n- 14 Jul \u00b7 Paid to Zepto \u2014 \u20b9 160.00\n- 15 Jul \u00b7 Paid to RAJU \u2014 \u20b9 40.00\n- 15 Jul \u00b7 Paid to Country Delight \u2014 \u20b9 250.00\n- 20 Jul \u00b7 Paid to Zepto \u2014 \u20b9 360.00\n- 21 Jul \u00b7 Paid to Country Delight \u2014 \u20b9 250.00\n- 23 Jul \u00b7 Paid to Country Delight \u2014 \u20b9 100.00\n- 26 Jul \u00b7 Paid to Country Delight \u2014 \u20b9 250.00\n- 31 Jul \u00b7 Paid to Swiggy Instamart Private Limited \u2014 \u20b9 334.00\n- 31 Jul \u00b7 Paid to Country Delight \u2014 \u20b9 250.00\n- 02 Aug \u00b7 Paid to Zepto \u2014 \u20b9 348.00\n\n_Showing all 17 matching groceries payments._\n_Totals use the full statement text (table extract alone was incomplete)._\n_Line-item coverage m", "mode": "pdf", "finance_intent": "category_spending", "latency_s": 0.69, "verdict": "pass", "notes": "ok", "numbers_found": [ 3801.0, 17.0, 3801.0, 17.0, 223.59, 360.0, 348.0, 334.0, 17.0, 4.0, 159.0, 5.0, 200.0, 6.0, 327.0, 11.0, 177.0, 11.0, 326.0, 12.0 ], "expected_numbers": [] }, { "id": "P15", "category": "in_pdf", "question": "Show my spending summary", "expected": "Summary with total ~30298.30", "answer": "Statement overview\n\nHere\u2019s a clean overview of \u201cPaytm_UPI_Statement_03_Jul_26_-_02_Aug_26.pdf\u201d for Aayushi Verma covering 3 JUL'26 - 2 AUG'26. Figures below come from the statement totals.\n\n- **Account holder:** Aayushi Verma\n- **Total spent:** \u20b9 30,298.30\n- **Total received:** \u20b9 7,729.00\n- **Net cashflow:** \u20b9 -22,569.30\n- **Payments made:** 84\n- **Payments received:** 5\n- **Period:** 3 JUL'26 - 2 AUG'26\n\n**Highlights**\n- Largest outflow: \u20b9 4,000.00 \u2014 Money sent to Ojas Verma\n- Average payment size: \u20b9 360.69\n\n**Payments (12)**\n- 19 Jul \u00b7 Money sent to Ojas Verma \u2014 \u20b9 4,000.00\n- 16 Jul \u00b7 Money sent to Aditya Yadav \u2014 \u20b9 3,000.00\n- 16 Jul \u00b7 Paid to Nykaa ERetail Private Limited \u2014 \u20b9 993.00\n- 12 Jul \u00b7 Paid to Meesho Technologies Private Limited \u2014 \u20b9 774.00\n- 03 Jul \u00b7 Money sent to Savita \u2014 \u20b9 500.00\n- 07 Jul \u00b7 Paid to Myntra \u2014 \u20b9 478.00\n- 20 Jul \u00b7 Paid to Zepto \u2014 \u20b9 360.00\n- 31 Jul \u00b7 Paid to Swiggy Instamart Private Limited \u2014 \u20b9 334.00\n- 06 Jul \u00b7 Paid to Zepto \u2014 \u20b9 327.00\n- 11 Jul \u00b7 Paid to Zepto \u2014 \u20b9 326.00\n- 20 Jul \u00b7 Recharge of Jio Mobile 6397352957 \u2014 \u20b9 300.80\n- 30 Jul \u00b7 Paid to ZOMATO \u2014 \u20b9 273.30\n\n_Showing 12 of 40 payments. Ask for the full list (e.g. \u201clist all payments\u201d) to see every line._", "mode": "pdf", "finance_intent": "statement_summary", "latency_s": 0.53, "verdict": "pass", "notes": "ok", "numbers_found": [ 3.0, 26.0, 2.0, 26.0, 30298.3, 7729.0, 22569.3, 84.0, 5.0, 3.0, 26.0, 2.0, 26.0, 4000.0, 360.69, 12.0, 19.0, 4000.0, 16.0, 3000.0 ], "expected_numbers": [ 30298.3 ] }, { "id": "P16", "category": "in_pdf", "question": "How many payments were received?", "expected": "5", "answer": "Payments received\n\nThis statement shows 5 payments received.\n\n- **Payments received:** 5\n- **Period:** 3 JUL'26 - 2 AUG'26\n\n_Counts prefer the statement header when available._", "mode": "pdf", "finance_intent": "payment_count", "latency_s": 0.6, "verdict": "pass", "notes": "ok", "numbers_found": [ 5.0, 5.0, 3.0, 26.0, 2.0, 26.0 ], "expected_numbers": [ 5.0 ] }, { "id": "O1", "category": "outside", "question": "What is EBITDA?", "expected": "General definition via Ollama; no Paytm totals required", "answer": "EBITDA stands for Earnings Before Interest, Taxes, Depreciation, and Amortization. It's a financial metric used to evaluate a company'self-profitability without considering the impact of finan", "mode": "conversational", "finance_intent": "general", "latency_s": 11.32, "verdict": "pass", "notes": "ok", "numbers_found": [], "expected_numbers": [] }, { "id": "O2", "category": "outside", "question": "What is the weather in Delhi today?", "expected": "Refuse / redirect to finance", "answer": "I focus on accounting and finance. Upload a statement in Knowledge, or ask about spend, merchants, or your documents.", "mode": "conversational", "finance_intent": "general", "latency_s": 0.59, "verdict": "pass", "notes": "ok", "numbers_found": [], "expected_numbers": [] }, { "id": "O3", "category": "outside", "question": "How much did Elon Musk spend last month?", "expected": "Should not invent from Paytm statement", "answer": "I don't have information about that person's spending. I only analyze statements you upload \u2014 ask about your own payments, merchants, or dates on this statement.", "mode": "conversational", "finance_intent": "general", "latency_s": 0.53, "verdict": "pass", "notes": "ok", "numbers_found": [], "expected_numbers": [] }, { "id": "O4", "category": "outside", "question": "Did I pay Amazon \u20b950,000 on this statement?", "expected": "No / not found (Amazon not on statement)", "answer": "No, you did not pay Amazon \u20b950,000 on this statement as there is no record of such transaction in your Payments History for July 31st and August 2nd provided herein. The transactions listed include a payment to Swiggy Instamart Private Note: Pay for Intent amounting to Rs.334 under the # Groceries tag, money received from Aditya Yadav totaling Rs.98 with no specific note or tags attached, and another transaction where you sent Rs.109 to R", "mode": "rag", "finance_intent": "document_qa", "latency_s": 32.89, "verdict": "pass", "notes": "denied amazon payment", "numbers_found": [ 50000.0, 31.0, 2.0, 334.0, 98.0, 109.0 ], "expected_numbers": [] }, { "id": "O5", "category": "outside", "question": "Explain depreciation in accounting", "expected": "Concept answer, not statement dump", "answer": "Depreciation is an accounting method used to allocate the cost of a tangible asset over its useful life. It represents how much of an asset'th value has been used up during a specific period, typically through wear and tear", "mode": "conversational", "finance_intent": "general", "latency_s": 11.14, "verdict": "pass", "notes": "ok", "numbers_found": [], "expected_numbers": [] } ] }