Harbor Framework (@harborframework) header
Harbor Framework (@harborframework) profile picture
San Francisco, CA
Joined January 2026
Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)Photo by Harbor Framework (@harborframework)
drop a comment on ATIF v2 RFC: github.com/harbor-framework/…
ATIF (v1.0-v1.8) has become the de facto standard for sharing trajectories (including data exchanges) throughout the industry. And now we are redefining ATIF with Google. ATIF v2 proposes a richer format for agent trajectories: live event streaming, tool calls paired with results, and explicit lineage for retries, forks, and subagents across harnesses. Would love feedback on the RFC! mbleigh.github.io/harbor/ github.com/harbor-framework/…
1
3
238
Harbor Framework retweeted
terminal bench featured in Microsoft Ignite keynote by @satyanadella
At GitHub Universe in October, I talked about multi-model solutions like Copilot Fusion will start dominating benchmarks going forward. But the pace of progress has been faster - ~10% better quality at half cost just few weeks later.
1
2
21
992
v0.26
update to v0.26 and try 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 along with other amazing features. → Harbor Serve → Error policies → Rollout viewer → Agent & sandbox discovery → New integrations → Pre-installed agents → Sandbox reliability → Trajectories & usage tracking
1
9
656
Harbor Framework retweeted
exactly, and that’s what harbor is for. harbor❤️vercel: vercel.com/blog/agent-evals
The old way: users are human, you run user interviews, get inspired, mock up user journeys. The new way: users are agents, you run evals, evals inform the software engineering loop. Good read on how we do evals at Vercel.
1
1
20
794
Building for agents means building evals.
1
2
53
3,047
Harbor Framework retweeted
a TerminalBench task > a nature paper
Given that there are only 66 problems in TerminalBench v4.1 and literally every frontier lab tracks TerminalBench performance, at some point it will be more prestigious and impactful to have authored a TerminalBench problem than to get a paper accepted in Nature or Science
1
3
29
2,122
Harbor Framework retweeted
7/9 top rated benchmarks run natively on harbor. harbor is the best.
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:
2
2
32
1,575
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚟𝚊𝚕𝚎𝚗𝚕𝚊𝚋𝚜/𝚏𝚛𝚘𝚗𝚝𝚒𝚎𝚛𝚋𝚒𝚘𝚖𝚕
Introducing FrontierBioML: 25 long-horizon machine learning tasks in computational biology with 11 models, up to 6 hours per run. Agents submit a prediction script; we run it on a held-out test set. Claude Mythos 5.2 leads (40.3), then Gemini 4 Carbon (39.0) and Gemini 4 Argon (36.8)
2
15
1,395
Harbor Framework retweeted
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎! Rollout servers are APIs / MCPs that you (or your agent) to Launch jobs Monitor rollouts Query results Pause, cancel, retry, or resume jobs and rollouts They're the missing control plane to Harbor workloads.
18
12
5
199
14,980
Harbor Framework retweeted
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 which turns harbor into a server lets you submit jobs, query/pause/resume/cancel jobs and rollouts, and retry individual rollouts. It's a huge unlock for RL and one of the most elegant and powerful features Alex and I built in harbor. four ways to connect: 1. python sdk: Harbor() / AsyncHarbor() 2. cli: harbor run --server http://localhost:8390 3. mcp: /mcp 4. http: POST /jobs/submit so much more control and observability unlocked: docs.harborframework.com/har… docs.harborframework.com/api…
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎! Rollout servers are APIs / MCPs that you (or your agent) to Launch jobs Monitor rollouts Query results Pause, cancel, retry, or resume jobs and rollouts They're the missing control plane to Harbor workloads.
16
9
3
89
6,752
Harbor Framework retweeted
Harbor is the new programming language
Software Engineering became Machine Learning. presented by our very own Alex Shaw.
7
6
2
133
6,199
Harbor Framework retweeted
Software Engineering became Machine Learning. presented by our very own Alex Shaw.
2
3
2
52
6,610
OWE-Bench is a great addition to coding benchmarks. Run it with Harbor.
Agents can write code. Can they finish real engineering work? Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design. Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end. OWE-bench top 3 leaderboard: 1. Fable 5.2 (54.6%) 2. GPT-6 Astra (44.2%) 3. Claude Opus 5.5 at 41.6%. Explore the benchmark website: openworldengineer.mercor.com…
1
2
20
925
Nice continuous benchmarking by Biopharma Bench! Latest version update is on the Harbor Hub: hub.harborframework.com/data…
Biopharma Bench V0.2 is a huge upgrade from the previous version, mitigating eval awareness with the @harborframework team and working with experts at Lonza, Catalent, Takeda and Sanofi to audit the rubrics and expert judgment. Check it out!
2
12
861
SWE-rebench harbor training tasks! hub.harborframework.com/data…
🚀 SWE-rebench V2.1 RL is out: 13,337 free RL environments for coding agents (20 languages, Harbor format)! 🔧 Fixed tasks, envs, and graders → fast, reliable rollouts. Doing RL post-training or research? DM or reply. We can share data and RL envs early for free, so we can make them better for open source! 🙏 Please RT or share with anyone who might need free RL envs!
1
1
7
591
harbor x runta
awesome work, built & run with harbor harbor x runta
269