What the reviews say, what this would change, and how I know the engine is right.
Negative reviews mentioning each theme
Highlighted bars = claim-lifecycle themes TrackBack addresses. Keyword-coded with Hinglish variants; spot-check precision 37/40 on a random sample. Code in research/themes.py.
Steady, not a one-off spike
December is a partial month (data starts 17 Dec). Total review volume grew ~5× from Jun to Aug 2026 (mostly 5★), while negative reviews stayed at 50–90 a month. The lifecycle problem is structural.
1,000 simulated claims through the engine
Outcome of each claim
95% settled in seconds without a person. Even a “no” is a win: it arrives instantly with the exact rule, instead of after 8–10 days of back-and-forth. The claim mix is a stated assumption (ASSUMED_MIX).
How I’d measure it in production
- North star
- Share of missing-cashback cases resolved with no ticket raised (auto-file, auto re-check, or clear “no”).
- User outcome
- Median time from “didn’t track” to first definitive answer. Today: 8–10 days (reviews). Target: under 1 minute for 90%.
- Guardrails
- Network rejection rate of auto-filed claims (must not rise above the manual baseline); reopen/dispute rate on “not eligible”; cashback cost per order.
- Lagging
- 1–2★ reviews mentioning tracking/pending/cancelled per 1,000 reviews; repeat purchase through the app after a claim.
Support hours per month
95% decided without a person comes from the simulated claim mix (1,000 claims). Every input here is an assumption to replace with CashKaro’s real ticket data in week one. The bigger win isn’t hours: it’s the user getting an answer in seconds instead of 8–10 days, inside the return window.
Proving the engine does what the spec says
Money decisions need tests, not vibes. The suite plants one broken rule per scenario and adds noise that must not change the answer. Honest caveat: it proves the rules match the spec, not that the spec matches CashKaro’s real networks. That’s a backtest on historical claims, and it’s the first thing I’d run (PRD §9).
Confusion matrix (rows = planted, cols = engine)
| Auto-filed | Wait + auto re-check | Needs one detail | Not eligible (reason shown) | Human review | Already tracked | |
|---|---|---|---|---|---|---|
| Auto-filed | 25 | |||||
| Wait + auto re-check | 25 | |||||
| Needs one detail | 50 | |||||
| Not eligible (reason shown) | 225 | |||||
| Human review | 50 | |||||
| Already tracked | 25 |
Boundary cases
- ✓E01 Order 23.9 h after the visit is inside Amazon's 24 h window → Auto-filed
- ✓E02 Order 24.1 h after the visit is outside it → Not eligible (reason shown)
- ✓E03 User opened the app 10 min after buying (common) — not attributable → Not eligible (reason shown)
- ✓E04 Flipkart credits one item; an excluded gift card first in the cart must not eat that slot → Auto-filed
- ✓E05 Claim on the last day of the window is accepted → Auto-filed
- ✓E06 Exchange at Myntra doesn't void cashback → Auto-filed
- ✓E07 Exchange at Amazon does → Not eligible (reason shown)
- ✓E08 Myntra's own coupon is fine → Auto-filed
- ✓E09 Outside coupon at a store that allows any coupon → Auto-filed
- ✓E10 ₹4,999 order with no invoice goes through → Auto-filed
- ✓E11 ₹5,000 order with no invoice asks for it → Needs one detail
- ✓E12 High value but no visit: say no now, don't waste a reviewer's time → Not eligible (reason shown)
- ✓E13 Too early but an outside coupon already voids it: don't make the user wait 3 days for a no → Not eligible (reason shown)
- ✓E14 Cashback capped at ₹1,500 on Amazon still needs a human (over ₹1,000) → Human review
- ✓E15 Flagged account never auto-files, even for ₹30 → Human review
- ✓E16 Nykaa visit 72 h before the order, window is 24 h: outside → Not eligible (reason shown)
Order extraction: where AI earns its place
13 documents: 9 clean confirmation emails and 4 messy ones (a forwarded email, an SMS, a Hinglish chat message, a cancellation notice).
| Extractor | All fields right | Field accuracy | Misses |
|---|---|---|---|
| Rule-based parser (fallback) | 84.6% | 93.6% | messy_sms: items · messy_hinglish: amount, items, status, coupon |
| LLM (openai/gpt-oss-120b on Groq) | 92.3% | 98.7% | messy_sms: items |
The parser is the safety net when the API is down; it breaks on exactly the inputs real users send (Hinglish, SMS). Either way, extracted fields go through sanity checks (order-ID format per store, items vs total, date range) and the user confirms them before the engine runs.