All sessions and workshops curated by leading AI/ML practitioners

We caught our LLM quality problem the wrong way: through customer complaints. Three weeks of degraded responses had already shipped. Every monitoring signal we had was green the entire time because we were measuring the wrong things. Latency, error rate, embedding similarity: none of them captured what was actually happening, which was that response quality had quietly gotten worse in ways users noticed but our dashboards could not.

After that incident we rebuilt how we think about LLM observability. The core insight was that output quality is not the same as system health, and you cannot infer one from the other. We needed a separate signal layer built from production behavior: how users responded to answers, where downstream tasks broke down, where retrieval and response stopped agreeing with each other.

Getting this right took a few tries. A naive implementation fires constantly on normal LLM output variance. The real work was designing alerts that distinguish actual degradation from noise, and calibrating thresholds against real incident history rather than theoretical bounds.

You will leave with a taxonomy of LLM drift types and how each one shows up differently in production, the behavioral signals that actually correlate with quality degradation, and an alert pattern that catches real drift early without burying your team in false positives.

Building customer facing agentic systems is notoriously hard. Users need to glue together databases, external services, workflow engines, stream processing systems, backtesting or eval systems and agent observability once the model is in prod.

While doing this they need to ensure a few core guarantees – Freshness, Durability, Idempotency, Backtestability, and Measurability

Most systems only provide one or two of these. We introduce Zipline flow an agentic application framework built to automagically provide these guarantees to agentic workflows.

Instacart’s ad-serving feature store served features to real-time inference from a multi-hundred-node managed Valkey deployment split across multiple clusters. The team had built a proxy layer, a querying SDK, and a compact storage format on top of it, and still hit three limits: instability during any cluster mutation, tail latency from read fan-out, and cost that got worse when they split clusters to manage the first two.

The structural change was to stop scaling out and scale up. A multi-threaded engine let the team run much larger instances, so hundreds of nodes became roughly 100, each inference request touched far fewer shards, and average and P99 latency fell 50%. Because the proxy and SDK already hid the datastore from ML engineers, the swap required no application changes and tens of terabytes moved without anyone above the storage layer noticing.

We were the engine vendor, and the first pass did not reach the latency the team expected. Getting there took client-side tuning on their end and engine changes from our engineering team, including to the compactor. Validation ran side by side: identical backfill, traffic shifted from 0% to 100%, gates at each percentile.

Attendees will leave able to test a datastore against the fan-out shape of their own feature store rather than an ops/sec number, trace how engine concurrency sets instance size, which sets fan-out, which sets P99, and run a datastore migration under live traffic with per-percentile gates and a rollback path. They will also hear where the trade-off comes back as the cluster grows.

There’s a gap between researcher-crafted evaluation frameworks that capture model performance and benchmarks versus how end users actually use AI products. This gap is being exploited in ways that render traditional reward models useless.

What happens after an agent generates an analysis? In this session, I’ll use an investigation of Thinking Machines Lab’s Inkling audio model to show how an interactive Python notebook supports continued experimentation. Working in Kilo Desktop with nteract, we’ll examine audio representations, listen to samples, change parameters, and revise the code behind the results. The workflow combines agent assistance with direct manipulation and explicit execution, leaving an editable computational artifact that a practitioner can inspect, share, and return to. Attendees will learn how to structure agent-assisted investigations so that generated code becomes a useful tool for learning.

“Teams building a sovereign AI factory ask two things of the runtime underneath it. A run that dies on infrastructure at hour eleven has to pick up where it stopped instead of starting over, and the whole system has to run inside their own cloud account and pass a security review that assumes the vendor gets breached. The first sounds like a reliability problem and the second like a compliance problem. This is the story of how we built Union.ai’s runtime on Flyte, and how they turned out to be one design decision about where state lives.

We tackled durability first. An OOM kill, a reclaimed spot node, or an A100 request that never schedules falls between an orchestrator that can’t change the pod and a Kubernetes that has no idea a run exists. We closed that gap by moving run state out of the compute layer, so `except flyte.errors.OOMError` with `task.override(resources=…)` escalates memory inside the same run and keeps the 47 steps that already finished, and `@flyte.trace` turns crash recovery into memoization. On identical pods, a 1,000-leaf fan-out went from 124s to 29s.

Then comes the security review. We started by assuming our own control plane was compromised and kept designing until the only thing an attacker could reach was run metadata: references to data but never the actual payloads, clusters that dial out so nothing dials in, and authorization and keys inside the customer’s perimeter. What we didn’t expect was what that bought us. A control plane holding only references and leases is small enough for one scheduler to keep every queue, quota, and connected cluster in memory, and because every cluster connects outbound, one on AWS, one on a neocloud, and a rack on-prem all look the same to it. That’s what lets priority queues, gang scheduling, and cross-cloud routing work without a networking layer per cluster.

At the end of this talk, you’ll leave with a deeper understanding on the architectural decisions that enabled us to support customers running 12 million+ containers concurrently across multiple Kubernetes clusters and cloud providers.”

Deploying AI applications often involves containers, GPU configuration, dependency management, and infrastructure scaling. Runpod Flash offers a simpler, code-first approach. It lets you define remote functions and hardware requirements in Python and run them on serverless GPUs.

In this talk, we’ll explore how Flash moves Python workloads from local development to cloud deployment, examine its underlying programming model, and build a GPU-backed endpoint. Along the way, we’ll discuss where Flash fits in the AI development stack, the problems it solves, and the trade-offs developers should consider. Attendees will leave with a practical understanding of how to turn local AI code into a scalable service without needing to become infrastructure experts.

Single-shot evaluation, the shape every current AI eval framework ships, is structurally blind to the failures that take agents down in production: patience that runs out at turn six, users who abandon silently, tool calls that lose conversation state, partial successes that masquerade as wins. This talk argues for a different shape, drawn from the digital twin literature and how the approach is already applied in aerospace, autonomous vehicles, and civil engineering: simulate the population of users your agent will meet, run them against the agent, watch what breaks before any human sees it.

Three take aways:

1. Why current AI agent evaluation cannot detect the failures that hit production. The structural reason prompt-and-grade testing (the shape every current framework ships) is blind to patience, abandonment, conversation state, and partial success.
2. What a realistic synthetic user population looks like, drawn from the digital twin literature in other engineering industries. Four properties: continuous trait clusters instead of fixed persona archetypes; probabilistic dropout (real users walk away silently rather than say goodbye); three-way goal scoring (each conversation is scored “achieved,” “partially achieved,” or “not achieved,” instead of just pass/fail); and variable conversation length (each conversation ends when the persona’s patience runs out, not at a fixed turn count).
3. How to wire population-scale evaluation into a release gate. Why run-over-run delta on a held-constant population is the regression-detection primitive AI agents actually need, and how to treat it as a governance lever rather than a research experiment.”