Surge AI (@HelloSurgeAI) header
Surge AI (@HelloSurgeAI) profile picture

Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.

Joined June 2020
Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)Photo by Surge AI (@HelloSurgeAI)
5 Surge AI benchmarks in the latest @FireworksAI_HQ Kimi K3.1 Pro research 👀 From expert-level legal, accounting, and tax work to staff-level software engineering. Great to see our benchmarks helping evaluate the next generation of models.
K3.1 Pro from @Kimi_Moonshot outscores K3 on 14/17 practitioner-built benchmarks on the SII. Against Opus 5.5, it can cut costs on some workloads or improve quality at a premium on others. Should you switch your agents? See where K3.1 Pro fits in our new SII model study: bit.ly/kimisiix
1
12
538
⚖️ DAYJOB: LAW, our new benchmark measuring whether agents can handle the work of experienced lawyers ⚖️ An investor is about to put $20 million into a startup. The company’s ownership spreadsheet looks clean. But its official stock ledger shows 350,000 shares held by a former executive that the spreadsheet leaves out. The board approved a buyback, but there’s no proof it ever happened. Would an AI agent notice? Better question: would it do anything about it? That’s one of the scenarios in DAYJOB: Law, our new benchmark for evaluating AI agents on real-world legal work, with 2,638 messy case files and nearly 1 GB of data to sort through. The agent has to recognize that the missing shares could change the investor’s ownership stake and the price they should pay, even though the request never mentioned them. Congrats 🏅GPT-6.2 Astra, 🥈GPT-6.2 Sol , and 🥉Sonnet 5.5 for the top three spots.
6
3
2
29
2,498
The best model so far, GPT-6 Astra, passes just 36.6% of attempts at a 95% rubric threshold. Nine of the 16 models we’ve tested pass fewer than 14%. Methodology and leaderboard: surgehq.ai/blog/dayjob-law
1
3
259
Thank you to the @harvey team for previewing DAYJOB: Law ahead of release and reviewing the realism of our scenarios. ♥️
4
155
GPT 6.2 Astra and GPT 6.2 Sol launched today. We've been running them through the Surge benchmark suite: 11 benchmarks built by professional doctors, accountants, lawyers, and staff software engineers to test whether models can get through an ordinary Tuesday at work. TL;DR 1. GPT 6.2 Astra is a step forward and the OpenAI model to use for professional work. It takes #1 on Chartography, has OpenAI's best score yet on sudo L7 (although still substantially behind the latest Fable, Opus, and Sonnet models), and beats GPT 6 Astra on every benchmark so far. 2. GPT 6.2 Sol is a mixed upgrade over 6.1 Sol. It got better at coding, charts and agentic instruction following, and worse at healthcare, PDFs, and finance. Re-test before you upgrade. 3. The leaders haven't changed. Anthropic's 5.5 generation still leads on coding, long-context instruction following and every DAYJOB track. Google leads on spreadsheets and PDFs. – Chartography: Astra takes #1 Chartography tests whether models can read the charts professionals rely on every day: Kaplan-Meier survival curves, candlestick charts, Sankey diagrams, and Bode plots. Each task tests not only perception, but also expert domain knowledge and multi-step reasoning. ➤ #1 GPT 6.2 Astra: 90.7% ➤ #2 GPT 6.2 Sol: 87.7% ➤ #3 Gemini 4 Argon: 71.6% ➤ #4 GPT 6 Astra: 71.0% ➤ #5 Claude Opus 5.5: 66.3% 6.2 Astra gains +19.7 points over GPT 6 Astra, and Sol gains +34.1 points over GPT 6 Sol (53.6%). These are the largest jumps in today's results. – sudo L7: better, but still well behind Claude sudo L7 asks whether coding agents could get promoted to staff engineers. Agents have to find their way around unfamiliar systems, spot hidden risks, and recognize when working code is still the wrong thing to ship. ➤ #1 Claude Fable 5.2: 52.2% ➤ #2 Claude Sonnet 5.5: 50.6% ➤ #3 Claude Opus 5.5: 46.1% ➤ #4 GPT 6.2 Astra: 38.3% (new) ➤ #5 GPT 6.2 Sol: 36.1% (new) ➤ #6 GPT 6.1 Sol: 33.9% ➤ #8 GPT 6 Astra: 31.1% – HANDBOOK.md: a big jump in long-context instruction following Agents work from a 100-page enterprise handbook across five knowledge work domains, using internal tools and external MCP servers. ➤ #1 Claude Opus 5.5: 52.3% ➤ #4 GPT 6.2 Astra: 45.8% ➤ #7 GPT 6.2 Sol: 34.5% – DAYJOB: long-horizon agents doing real professional work In DAYJOB, agents take on real 9-to-5 workflows inside messy environments. DAYJOB: Healthcare ➤ #1 Claude Opus 5.5: 48.3% ➤ #2 GPT 6.2 Astra: 34.4% ➤ #12 GPT 6.2 Sol: 17.2% ⚑ Sol drops 8 points from GPT 6.1 Sol's 25.2%, the largest regression we measured today. DAYJOB: Tax ➤ #1 Claude Sonnet 5.5: 10.8% ➤ #3 GPT 6.2 Astra: 9.2% (tied) ➤ #7 GPT 6.2 Sol: 4.8% (tied) DAYJOB: Finance ➤ #1 Claude Opus 5.5: 42.5% ➤ #7 GPT 6.2 Sol: 32.8% ➤ GPT 6.2 Astra: still running – GDP.xlsx ➤ #1 Gemini 4 Argon: 38.3% ➤ #3 GPT 6.2 Astra: 29.7% ➤ #8 GPT 6.2 Sol: 22.3% (tied) GDP.pdf ➤ #1 Gemini 4 Carbon: 56.4% ➤ #10 GPT 6.2 Sol: 48.8% (tied)
18
5
1
52
8,856
Surge AI retweeted
6.2 astra and sol take the 4th and 5th spots on sudo L7, our benchmark for staff-level coding ahead of the earlier gpts, but still behind the latest claudes
2
1
12
1,266
Surge AI retweeted
interesting hacker news comments on DAYJOB: Tax i can absolutely see a world where agents save people thousands of dollars (or more!) every year on their taxes
2
2
19
1,190
Surge AI retweeted
jeff dean does a lot more than write *correct code*. sure, his code will pass unit tests. if you give him an elaborately well-specified spec doc, he’ll nail it in his sleep. but that’s not why you want a jeff dean. you want a jeff to: - tell you when you’re being stupid - help you think through architectural tradeoffs, and when your request does (or doesn’t!) warrant a system redesign - think 10 steps beyond - build not what you asked, but what you really need in that light, even the smallest Claude is a better Jeff Dean than the biggest GPT
4
3
1
25
4,144
Who's a better *STAFF* software engineer? Claude or GPT? We benchmarked the top Claude and GPT models on sudo L7, our coding benchmark for staff+ judgment and behavior. Every Claude, including Haiku, beat the GPTs.
4
3
1
16
1,569
Surge AI retweeted
new coding post-training run on GLM-5.3 +21.3pp on Terminal-Bench4 (34.8% → 56.1%) +15.0pp on SWE-marathon (30% → 45%, pass@1) +14.4pp on sudo L7 (14.5% → 28.9%) +6.5pp on DeepSWE (66.90% → 73.45%)
4
4
27
2,751
Be careful what you grade for.
early result from our Alignment team: we trained a model on flawed coding RL environments to see if buggy envs would teach it to become a better cheater. unfortunately, they did. in one example (a Terminal Bench 4 task), it added 11 redundant fields to a website, then made them invisible by making them 1 pixel wide and positioning them 10,000 pixels off screen - all in order to game the grader. and it succeeded! it passed 28/28 of the TB4 tests with flying colors. if you've ever looked at recent models' COTs and wondered why they're so *obsessed with the graders*, it's no wonder thread 🧵
1
10
1,087
Less than one month until NeurIPS! 🇦🇺 The Surge AI team is heading to Sydney, Dec 6–12. We’re looking forward to connecting with researchers and teams working on frontier model evaluations, post-training, and the next generation of AI capabilities. Going to NeurIPS? Let’s find time to meet. 📩 surgehq.ai/neurips
1
10
1,378
Inkling 2 Small redefines sudo L7's cost-performance frontier, our benchmark for staff-level software engineering. It's the cheapest model on sudo L7 ($0.34 per task!), while punching way above - coming within two points of Gemini 4 Argon and Muse Wonder 1.0 at a fraction of the cost. ➡️ Gemini 4 Argon: 2 points higher at 7.8x the cost ($2.64) ➡️ Muse Wonder 1.0: 2 points higher at 13x the cost ($4.39) 💰 Inkling 2 Small: $0.34 And it outscores models that cost 3–7× more: ➡️ Muse Spark 1.3: 5.4x the cost ($1.85) ➡️ Gemini 3.8 Flash: 7.3x the cost ($2.47) ➡️ GLM 5.3: 3.4x the cost ($1.15) Congrats to the @thinkymachines team!
3
2
2
39
4,068
Inkling’s model card cites 5 other Surge benchmarks as well: → DAYJOB: Finance → Chartography → ComplexConstraints → EnterpriseBench: CoreCraft → GDP.pdf Check out the full model! thinkingmachines.ai/news/ink…
1
2
481
Surge AI retweeted
people are underestimating just how much headroom there is in open models we took Surge RL data, trained @NVIDIAAI Nemotron 3.5 Super (using @thinkymachines Tinker!), and beat much larger closed models across 12 benchmarks, including GPT-5.6 Terra on coding ➡️ GDP.pdf: +18.6pp ➡️ HANDBOOK.md: +12.8pp ➡️ AutomationBench: +12.7pp ➡️ Chartography: +11.2pp ➡️ AdvancedIF: +7.7pp ➡️ Terminal-Bench 2.1: +6.4pp ➡️ DeepSWE: +4.7pp high-quality RL data makes frontier performance accessible to everyone
5
4
1
45
2,695
How far can great RL data push an open model? Far enough to beat frontier closed models. We trained @NVIDIAAI's Nemotron 3.5 Super VL (just 12B active parameters) on Surge RL data using @thinkymachines' Tinker. The result? It beat GPT-5.6 Terra's published score on SWE-Bench Pro V2, and also improved across 11 other benchmarks: ➡️ GDP.pdf: +18.6pp ➡️ HANDBOOK.md: +12.8pp ➡️ AutomationBench: +12.7pp ➡️ Chartography: +11.2pp ➡️ AdvancedIF: +7.7pp There's enormous untapped potential in open models. We believe that frontier performance doesn't have to come from frontier size. It can come from better data and better post-training, and we're only scratching the surface.
4
8
1
62
5,798
Remember the era of fancy prompt engineering? Do elaborate coding agent harnesses actually make agents better? We benchmarked popular coding setups on 100 real-world software engineering tasks using sudo L7. The results: ➡️ Best harness additions: +2–3 points ➡️ More thinking: +11.3 points! ➡️ One coding agent setup: 8× the cost, no improvement In one case, a harness spawned 17 agents to review a simple email change, and all 17 still missed a critical issue. Sometimes smarter thinking beats more agents.
16
1
3
26
2,657