What’s New in MLflow: November 2026 Roundup www.teapawt.org/i/broadcasts/2vyAXoeqY…
Most teams hand-tune agent prompts until it breaks at scale. This cookbook shows a measured loop: domain judges, expert alignment, and GEPA optimization on MLflow traces.
What's inside:
• Tool-calling LangGraph agent you can adapt
• Domain judges with pass/fail rubrics
• Expert alignment via mlflow.log_feedback() + MemAlign
• GEPA optimization: 40% → 85% pass rate in the example
Run it once, compare baseline vs optimized in the UI.
👉 Check out the cookbook: mlflow.org/cookbook/agent-al…
Reminder: Find out what's new in MLflow LIVE this Wednesday 📅
Join us November 18 at 4:00 PM PT to explore the latest in AI observability, AIOps, AI Governance, and developer experience.
We'll cover MLflow 3.18 features including Jev, Issue Detection, and the new Eval UX, followed by a live Q&A session.
Register 👉 luma.com/mlflow-t8h9
Losing track of which prompt or model worked best? In this MLflow A–Z episode, MLflow Ambassador @shrinathsuresh covers E for experiment tracking.
If you're changing models, prompts, and configurations, MLflow helps you see which version worked best by logging prompts, parameters, evaluation scores, latency, and token usage so every run is comparable.
🔗 Experiment tracking docs:
mlflow.org/docs/latest/ml/tr…
🔗 Watch the full MLflow A–Z series:
youtube.com/playlist?list=PL…
MLflow 3.18.0 is now available! 🎉
🔍 Automatic Issue Detection finds recurring agent failures across your traces, from wrong answers to unusual spending.
📚 The experimental Agent Skill Registry shares reusable skills with coding agents like Claude Code, Codex, and Cursor.
📊 Evaluation comparisons now pin a baseline and surface regressions.
✅ Experimental online evaluation runs your Python scorers on new traces on a self-hosted server.
Full release: mlflow.org/releases/3.18.0/
Five coding agents usually means five API keys and one invoice nobody can explain.
@2twitme shows how to govern that with MLflow AI Gateway. Agents point at one gateway, and it runs guardrails, enforces a budget, and traces every call.
👉 Learn more: mlflow.org/blog/govern-codin…
#MLflow #GenAI #LLMOps #OpenSource
MLflow Ambassador @shrinathsuresh covers D for Dataset Tracking in this MLflow A-Z clip.
Log training and evaluation datasets, version them, and associate those versions with your experiments. Skipping that is a huge mistake for GenAI, because the models are probabilistic.
🎥 youtube.com/watch?v=BoTbLWWV…
MLflow AI Gateway has budget policies. You set a daily, weekly, or monthly spend limit. 👇
When it hits, you can send a webhook to Slack, PagerDuty, or any HTTP alerting and keep serving traffic, or block requests until the window resets. The limit applies to every app that routes through the gateway.
Get started: pip install 'mlflow[genai]'
🔗 mlflow.org/blog/gateway-budg…
6
504
This week at ODSC AI West, Debu Sinha (@databricks) is running a workshop on Evaluating LLM Applications with MLflow.
Teams already run evals. The workshop is the part after: did the model actually get better, which samples regressed, and is the delta statistically significant or just noise. You build that pipeline in open source MLflow, from mlflow.genai.evaluate() through run comparison and a CI gate that fails the build when a regression is real.
🗓️ Wed, Oct 28
🕓 11:00 AM–12:00 PM PT
📍 Hyatt Regency SFO
🔗 odsc.ai/speakers-portfolio/e…
This week at ODSC AI West, Samraj Moorjani (@databricks) is giving a 30-min demo on Hidden Gems in Agent Traces.
Traces show what the agent did. The talk is about what you do next: find repeating failure modes, auto-detect them across a large volume of traces, and check the fix with evals you can rerun. All in open source MLflow.
🗓️ Tues, Oct 27
🕓 11:05–11:35 AM PT
📍 Hyatt Regency SFO
🔗 Schedule: odsc.ai/west/expo/
Teams keep adding agent harnesses. Claude Code, Codex, Cursor, and the rest each keep their own conventions, so observing a run across them is messy.
@2twitme walks through that harness layer, then how @omnigent_ai puts coding agents and custom agents behind one common API. From there, @opentelemetry traces land in MLflow so you can see how the agent actually behaved.
Watch the full walkthrough on how to trace and debug AI coding agents with Omnigent and MLflow.
🎥 youtube.com/watch?v=vvJTyd-e…
Part 3 of the Self-Improvement & MLflow series by MLflow Ambassador Jonathan Bown walks through automating the verification layer of a self-improving AI workflow with MLflow and CI/CD.
A coding agent reads traces at scale and can propose changes to itself, its skills, tools, memory, or the verifier. CI then checks each change before it is approved automatically or by a person.
Read Part 3 👇
opendatascience.com/self-imp…
What happens when every team has its own agents in production? You need more than LLMs. You need a control plane.
For the third video in the MLflow A–Z series, MLflow Ambassador @shrinathsuresh covers C for Control Plane. He explains how MLflow AI Gateway brings access control, budgets, guardrails, traffic routing, and usage tracking into one place.
🔗 Learn more: mlflow.org/blog/gateway-guar…
#MLflow #GenAI #LLMOps
The Claude SDK makes it fast to build an agent. With native @opentelemetry export and MLflow, it is just as fast to know whether the agent works.
🔹 Trace every model call by pointing a standard OTel exporter at MLflow
🔹 See each run as one trace, with model calls and tool calls under one agent span
🔹 Find expensive runs from token usage on the trace
🔹 Score every run with an LLM judge that inspects the trace
🔹 Compare system prompts with the same test prompts and judge
🔗 mlflow.org/blog/claude-sdk-o…
#MLflow #Claude #OpenTelemetry
Part 2 of MLflow Ambassador Jonathan Bown’s series on self-improving AI workflows with MLflow is live.
He walks a support agent through the loop: traces, feedback, expectations, and judges in; a versioned prompt and an evaluation dataset out; then a production alias move so the agent updates without code changes. @mlflow.test turns that dataset into a quality gate.
When the manual pass stops scaling, a coding agent with MLflow skills proposes the themes, scorers, and tests. A human still accepts the changes. CI can run them as MLflow Evaluation runs.
🔗 opendatascience.com/self-imp…
#MLflow #GenAI #ODSC
Claude Platform now supports native @opentelemetry tracing across its agent ecosystem. Bring traces from the Claude family into one backend using OTLP, the OpenTelemetry Protocol:
🔹 Messages API and Claude SDKs: model requests and tool-runner activity
🔹 Claude Code and Agent SDK: agent steps, tool calls, and subagent activity
🔹 Managed Agents: agent turns, model runs, and platform-run tools
We published a guide covers setup OpenTelemetry export across the Claude family, including Claude Platform UI and SDK configuration.
👉 Setup guide: mlflow.org/blog/claude-otel-…
#MLflow #opensource #otel





