5 Surge AI benchmarks in the latest @FireworksAI_HQ Kimi K3.1 Pro research 👀
From expert-level legal, accounting, and tax work to staff-level software engineering.
Great to see our benchmarks helping evaluate the next generation of models.
K3.1 Pro from @Kimi_Moonshot outscores K3 on 14/17 practitioner-built benchmarks on the SII.
Against Opus 5.5, it can cut costs on some workloads or improve quality at a premium on others.
Should you switch your agents? See where K3.1 Pro fits in our new SII model study: bit.ly/kimisiix
1
12
538
⚖️ DAYJOB: LAW, our new benchmark measuring whether agents can handle the work of experienced lawyers ⚖️
An investor is about to put $20 million into a startup. The company’s ownership spreadsheet looks clean. But its official stock ledger shows 350,000 shares held by a former executive that the spreadsheet leaves out. The board approved a buyback, but there’s no proof it ever happened.
Would an AI agent notice? Better question: would it do anything about it?
That’s one of the scenarios in DAYJOB: Law, our new benchmark for evaluating AI agents on real-world legal work, with 2,638 messy case files and nearly 1 GB of data to sort through.
The agent has to recognize that the missing shares could change the investor’s ownership stake and the price they should pay, even though the request never mentioned them.
Congrats 🏅GPT-6.2 Astra, 🥈GPT-6.2 Sol , and 🥉Sonnet 5.5 for the top three spots.
The best model so far, GPT-6 Astra, passes just 36.6% of attempts at a 95% rubric threshold. Nine of the 16 models we’ve tested pass fewer than 14%.
Methodology and leaderboard:
surgehq.ai/blog/dayjob-law
1
3
259
Thank you to the @harvey team for previewing DAYJOB: Law ahead of release and reviewing the realism of our scenarios. ♥️
4
155
GPT 6.2 Astra and GPT 6.2 Sol launched today. We've been running them through the Surge benchmark suite:
11 benchmarks built by professional doctors, accountants, lawyers, and staff software engineers to test whether models can get through an ordinary Tuesday at work.
TL;DR
1. GPT 6.2 Astra is a step forward and the OpenAI model to use for professional work. It takes #1 on Chartography, has OpenAI's best score yet on sudo L7 (although still substantially behind the latest Fable, Opus, and Sonnet models), and beats GPT 6 Astra on every benchmark so far.
2. GPT 6.2 Sol is a mixed upgrade over 6.1 Sol. It got better at coding, charts and agentic instruction following, and worse at healthcare, PDFs, and finance. Re-test before you upgrade.
3. The leaders haven't changed. Anthropic's 5.5 generation still leads on coding, long-context instruction following and every DAYJOB track. Google leads on spreadsheets and PDFs.
–
Chartography: Astra takes #1
Chartography tests whether models can read the charts professionals rely on every day: Kaplan-Meier survival curves, candlestick charts, Sankey diagrams, and Bode plots. Each task tests not only perception, but also expert domain knowledge and multi-step reasoning.
➤ #1 GPT 6.2 Astra: 90.7%
➤ #2 GPT 6.2 Sol: 87.7%
➤ #3 Gemini 4 Argon: 71.6%
➤ #4 GPT 6 Astra: 71.0%
➤ #5 Claude Opus 5.5: 66.3%
6.2 Astra gains +19.7 points over GPT 6 Astra, and Sol gains +34.1 points over GPT 6 Sol (53.6%). These are the largest jumps in today's results.
–
sudo L7: better, but still well behind Claude
sudo L7 asks whether coding agents could get promoted to staff engineers. Agents have to find their way around unfamiliar systems, spot hidden risks, and recognize when working code is still the wrong thing to ship.
➤ #1 Claude Fable 5.2: 52.2%
➤ #2 Claude Sonnet 5.5: 50.6%
➤ #3 Claude Opus 5.5: 46.1%
➤ #4 GPT 6.2 Astra: 38.3% (new)
➤ #5 GPT 6.2 Sol: 36.1% (new)
➤ #6 GPT 6.1 Sol: 33.9%
➤ #8 GPT 6 Astra: 31.1%
–
HANDBOOK.md: a big jump in long-context instruction following
Agents work from a 100-page enterprise handbook across five knowledge work domains, using internal tools and external MCP servers.
➤ #1 Claude Opus 5.5: 52.3%
➤ #4 GPT 6.2 Astra: 45.8%
➤ #7 GPT 6.2 Sol: 34.5%
–
DAYJOB: long-horizon agents doing real professional work
In DAYJOB, agents take on real 9-to-5 workflows inside messy environments.
DAYJOB: Healthcare
➤ #1 Claude Opus 5.5: 48.3%
➤ #2 GPT 6.2 Astra: 34.4%
➤ #12 GPT 6.2 Sol: 17.2%
⚑ Sol drops 8 points from GPT 6.1 Sol's 25.2%, the largest regression we measured today.
DAYJOB: Tax
➤ #1 Claude Sonnet 5.5: 10.8%
➤ #3 GPT 6.2 Astra: 9.2% (tied)
➤ #7 GPT 6.2 Sol: 4.8% (tied)
DAYJOB: Finance
➤ #1 Claude Opus 5.5: 42.5%
➤ #7 GPT 6.2 Sol: 32.8%
➤ GPT 6.2 Astra: still running
–
GDP.xlsx
➤ #1 Gemini 4 Argon: 38.3%
➤ #3 GPT 6.2 Astra: 29.7%
➤ #8 GPT 6.2 Sol: 22.3% (tied)
GDP.pdf
➤ #1 Gemini 4 Carbon: 56.4%
➤ #10 GPT 6.2 Sol: 48.8% (tied)
jeff dean does a lot more than write *correct code*.
sure, his code will pass unit tests. if you give him an elaborately well-specified spec doc, he’ll nail it in his sleep.
but that’s not why you want a jeff dean. you want a jeff to:
- tell you when you’re being stupid
- help you think through architectural tradeoffs, and when your request does (or doesn’t!) warrant a system redesign
- think 10 steps beyond
- build not what you asked, but what you really need
in that light, even the smallest Claude is a better Jeff Dean than the biggest GPT
Who's a better *STAFF* software engineer? Claude or GPT?
We benchmarked the top Claude and GPT models on sudo L7, our coding benchmark for staff+ judgment and behavior.
Every Claude, including Haiku, beat the GPTs.
GPT was an excellent junior coder. But Claude pulled ahead on the skills you get from years of experience, intuition, and expertise.
surgehq.ai/blog/claude-vs-gp…
1
394
Be careful what you grade for.
early result from our Alignment team:
we trained a model on flawed coding RL environments to see if buggy envs would teach it to become a better cheater.
unfortunately, they did.
in one example (a Terminal Bench 4 task), it added 11 redundant fields to a website, then made them invisible by making them 1 pixel wide and positioning them 10,000 pixels off screen - all in order to game the grader.
and it succeeded! it passed 28/28 of the TB4 tests with flying colors.
if you've ever looked at recent models' COTs and wondered why they're so *obsessed with the graders*, it's no wonder
thread 🧵
1
10
1,087
Less than one month until NeurIPS! 🇦🇺
The Surge AI team is heading to Sydney, Dec 6–12.
We’re looking forward to connecting with researchers and teams working on frontier model evaluations, post-training, and the next generation of AI capabilities.
Going to NeurIPS? Let’s find time to meet.
📩 surgehq.ai/neurips
1
10
1,378
Inkling 2 Small redefines sudo L7's cost-performance frontier, our benchmark for staff-level software engineering.
It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the cost.
➡️ Gemini 4 Argon: 2 points higher at 7.8x the cost ($2.64)
➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39)
💰 Inkling 2 Small: $0.34
And it outscores models that cost 3–7× more:
➡️ Muse Spark 1.3: 5.4x the cost ($1.85)
➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47)
➡️ GLM 5.3: 3.4x the cost ($1.15)
Congrats to the @thinkymachines team!
Inkling’s model card cites 5 other Surge benchmarks as well:
→ DAYJOB: Finance
→ Chartography
→ ComplexConstraints
→ EnterpriseBench: CoreCraft
→ GDP.pdf
Check out the full model!
thinkingmachines.ai/news/ink…
1
2
481
people are underestimating just how much headroom there is in open models
we took Surge RL data, trained @NVIDIAAI Nemotron 3.5 Super (using @thinkymachines Tinker!), and beat much larger closed models across 12 benchmarks, including GPT-5.6 Terra on coding
➡️ GDP.pdf: +18.6pp
➡️ HANDBOOK.md: +12.8pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
➡️ Terminal-Bench 2.1: +6.4pp
➡️ DeepSWE: +4.7pp
high-quality RL data makes frontier performance accessible to everyone
How far can great RL data push an open model? Far enough to beat frontier closed models.
We trained @NVIDIAAI's Nemotron 3.5 Super VL (just 12B active parameters) on Surge RL data using @thinkymachines' Tinker.
The result? It beat GPT-5.6 Terra's published score on SWE-Bench Pro V2, and also improved across 11 other benchmarks:
➡️ GDP.pdf: +18.6pp
➡️ HANDBOOK.md: +12.8pp
➡️ AutomationBench: +12.7pp
➡️ Chartography: +11.2pp
➡️ AdvancedIF: +7.7pp
There's enormous untapped potential in open models. We believe that frontier performance doesn't have to come from frontier size. It can come from better data and better post-training, and we're only scratching the surface.
Read more about how our Research team trained Nemotron using Tinker on Surge on data:
surgehq.ai/blog/nvidia-nemot…
Remember the era of fancy prompt engineering? Do elaborate coding agent harnesses actually make agents better?
We benchmarked popular coding setups on 100 real-world software engineering tasks using sudo L7.
The results:
➡️ Best harness additions: +2–3 points
➡️ More thinking: +11.3 points!
➡️ One coding agent setup: 8× the cost, no improvement
In one case, a harness spawned 17 agents to review a simple email change, and all 17 still missed a critical issue.
Sometimes smarter thinking beats more agents.
Full analysis: surgehq.ai/blog/benchmarking…
1
351













