drop a comment on ATIF v2 RFC: github.com/harbor-framework/…
ATIF (v1.0-v1.8) has become the de facto standard for sharing trajectories (including data exchanges) throughout the industry. And now we are redefining ATIF with Google.
ATIF v2 proposes a richer format for agent trajectories: live event streaming, tool calls paired with results, and explicit lineage for retries, forks, and subagents across harnesses. Would love feedback on the RFC!
mbleigh.github.io/harbor/
github.com/harbor-framework/…
1
3
238
terminal bench featured in Microsoft Ignite keynote by @satyanadella
vercel did a blog post that heavily features harbor
vercel.com/blog/agent-evals
exactly, and that’s what harbor is for.
harbor❤️vercel: vercel.com/blog/agent-evals
Building for agents means building evals.
Your next users are agents. Evals are how you build for them.
▪︎ See how agents use your product
▪︎ Test changes before you ship
▪︎ Compare models as they launch
vercel.com/blog/agent-evals
a TerminalBench task > a nature paper
7/9 top rated benchmarks run natively on harbor. harbor is the best.
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier:
25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill)
20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo)
13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti)
10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr)
8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml)
7 votes: Vending-Bench (@axelbacklund, @lukaspet)
5 votes each:
- AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie)
- DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge)
- PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr)
The most common comment was that people don't really trust benchmarks at all.
56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote.
Counts are net: a few benchmarks received negative votes.
Continued:
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚟𝚊𝚕𝚎𝚗𝚕𝚊𝚋𝚜/𝚏𝚛𝚘𝚗𝚝𝚒𝚎𝚛𝚋𝚒𝚘𝚖𝚕
Harbor Framework retweeted
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎!
Rollout servers are APIs / MCPs that you (or your agent) to
Launch jobs
Monitor rollouts
Query results
Pause, cancel, retry, or resume jobs and rollouts
They're the missing control plane to Harbor workloads.
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 which turns harbor into a server lets you submit jobs, query/pause/resume/cancel jobs and rollouts, and retry individual rollouts.
It's a huge unlock for RL and one of the most elegant and powerful features Alex and I built in harbor.
four ways to connect:
1. python sdk: Harbor() / AsyncHarbor()
2. cli: harbor run --server http://localhost:8390
3. mcp: /mcp
4. http: POST /jobs/submit
so much more control and observability unlocked:
docs.harborframework.com/har…
docs.harborframework.com/api…
OWE-Bench is a great addition to coding benchmarks. Run it with Harbor.
Agents can write code. Can they finish real engineering work?
Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.
Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.
OWE-bench top 3 leaderboard:
1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.
Explore the benchmark website: openworldengineer.mercor.com…
Nice continuous benchmarking by Biopharma Bench! Latest version update is on the Harbor Hub:
hub.harborframework.com/data…
Biopharma Bench V0.2 is a huge upgrade from the previous version, mitigating eval awareness with the @harborframework team and working with experts at Lonza, Catalent, Takeda and Sanofi to audit the rubrics and expert judgment. Check it out!
SWE-rebench harbor training tasks!
hub.harborframework.com/data…
🚀 SWE-rebench V2.1 RL is out: 13,337 free RL environments for coding agents (20 languages, Harbor format)!
🔧 Fixed tasks, envs, and graders → fast, reliable rollouts.
Doing RL post-training or research? DM or reply. We can share data and RL envs early for free, so we can make them better for open source!
🙏 Please RT or share with anyone who might need free RL envs!























