From One-Shot to Agentic: Optimizing Shop Intelligence with DSPy

Shopify merchants can customize almost every aspect of their storefronts. That flexibility makes even basic questions about a shop’s brands, shipping policies, or return terms difficult to answer consistently at scale. Our first approach sent selected pages to a frontier model in a single prompt. It worked initially, but missing evidence, competing objectives, and inference cost limited its quality and coverage.

This talk traces the redesign of Shop Lens from one-shot extraction to a DSPy ReAct agent, then to three specialized agents for shop profiling, fraud signals, and merchant category codes. I’ll explain how we scoped each agent’s context and tools, reused fetched pages through shared infrastructure, and optimized each agent independently with GEPA.
Because live storefronts change, we built ShopSnap to capture and replay stores at annotation time, enabling reproducible evaluation and separate optimization and held-out datasets. We also separated batch orchestration, CPU-based agent execution, and self-hosted OSS model inference across Flink, Kubernetes, and GPU infrastructure.

In the results shared in my earlier talk, the redesigned system achieved approximately 75× lower inference cost and close to 2× improvement on selected quality metrics compared with the original one-shot approach. Attendees will leave with concrete patterns for task decomposition, evidence access, replay-based evaluation, and model selection—and an understanding of how those architectural choices compound.