A merchant writes a request in plain words. The model must pick the right action on the account, fill in the right record, and stop when the account does not allow it. Tested on 150 requests it had never seen.
One request the rules fixed, one that fine-tuning fixed, and one that needed both. Each card shows the account records involved, what the merchant wrote and the correct action. Press Play to replay what each setup did.
Each bar is a separate setup tested on all 150 requests; the last combines the two before it. Share handled exactly right; the line under each bar counts wrong money-moving actions: refunds, cancellations and invoices that should not have happened.
Each rule was switched off on its own and the full test re-run, so its number is what the score loses without it. Rules we tried and dropped were added on their own the same way.
How far the model's answers were from the correct ones during training, every 5 steps. Lower is better. One pass through the examples.
Share correct by type, on the unseen test.
Click a request to replay it, or run all 150 at once.