The model reads one page of a bank's, insurer's or payment company's annual report and calculates what the analyst asks: a growth rate, a share of a total, a change between years. Questions come from FinQA, a public benchmark written by finance experts; none were seen in training.
Each card shows the table from the report, the question and the benchmark's answer. Press Play to replay what each setup answered.
Each bar is a separate setup on the same 144 unseen questions, all from the shorter half of the benchmark's 297, so every score is a little higher than on the full set. Share answered correctly, to within 1%.
Each was tried on 100 practice questions from the benchmark's separate development set. The number is its change against the model alone (64%). The mistakes are mostly the wrong row, the wrong year or a misread question, which a calculator cannot repair. A few points either way is within chance at this size.
How far the model's answers were from the correct ones during training, every 5 steps. Lower is better. One pass through the examples.
Share correct by type, on the unseen test.
Click a question to replay it, or run all 144 at once.