See the 2026 Sponsorship Guide

Get Tickets

Chains of thought, reasoning, traces and working memory: what they mean for your pipeline

With companies aggressively pushing models and agents into production, there’s a sense we’ve reached a “rubber hits the road” moment. Driven by stakeholders, AI teams are under pressure to ensure these early AI investments start paying off, and fast.

For practitioners, while things are moving a million miles an hour, you can take steps to account for your models’ behaviour: how your models and agents operate, what they’re doing, and why.

That starts with reasoning.

Chain of Thought (CoT) is how most models reason right now, and if you want to understand how context works and how it affects your tooling pipeline, you have to understand reasoning first.

After all, as Safety, Alignment, Compliance, and Sovereignty become central to product development and delivery, a firm grounding in model internals will distinguish average AI shops from best-in-class.

BLUF – Bottom Line Up Front

  • Reasoning models provide a significant change in model capability
  • The terminology around their use can be confusing
  • The chains of thought, traces and context associated with these models represent security and privacy risks we need to understand and manage
  • They also represent an opportunity to monitor and improve model behaviour
  • If you want to build interpretable, transparent agentic pipelines, Chains of Thought are the first, logical method.
  • Chains of Thought are important for your evaluation architecture as well, potentially providing valuable input to LLM-as-judges

Overview and Introduction

This week, we’ll cover Chain of Thought (CoT). CoTs, traces and reasoning are also in the news, both indirectly and directly, as we grapple with an AI alignment crisis. For example, OpenAI’s GPT-6 Astra model may have reduced the amount of traces being exposed to users (Fortune 2026; Transformer News 2026).

Chain of Thought originated as a prompting technique and was later used as a reasoning method taught to models during post-training. When we use it as a prompting technique, we’ll call it CoT prompting. When we’re referring to a reasoning model, we’ll just say CoT. When we’re referring to what users can see of the reasoning trace, we’ll say “trace”.

CoTs are produced by “asking” the model to comply with a request (“describe what is being done”). The model can put whatever it likes into these mental scratchpads.

Specifically, we’ll look at:

  • how it is used
  • what it is
  • whether it works
  • what implications it has
  • why / when it may be important for agentic deployments and projects

How do practitioners use traces?

We’ll ground this in conversations we’ve had with practitioners through our TMLS Steering Committee, community outreach, and personal contacts.

In our discussions with practitioners, we found several direct uses for traces, and also that the idea of topologies was powerful and has inspired a number of agentic workflows.

Here’s a non-exhaustive list of use cases we’re seeing:

  1. Enterprise use cases – when running a reasoning model, traces can be used to help with:
    • triage – understanding why a model did something
    • routing – helping to decide which model to invoke on a subsequent step
    • analyst support – aiding explainability and interpretability of results
    • audit – where the trace is kept for regulatory or compliance purposes
  2. Tool and harness development – broadly speaking, traces are used for: (A) Eliciting and controlling reasoning and (B) User interface and experience.For example, traces can help estimate reasoning costs, display traces to users, maintain context between turns (and between harnesses), manage context, monitor, perform forensic analysis, and so on.Finally, CoTs are frequently used as a signal to test and harden models.
  3. Safety, security, assurance – traces are read as part of a set of control signals and monitored for misbehaviour, jailbreak and prompt-injection analysis, red-teaming, and so on.

Traces produced by CoT are a valuable signal on model behaviour and, like other patterns we’ve seen, provide tools to guide and tune your products and applications.

What is a Chain of Thought (CoT)?

To properly define a chain of thought, we’re going to:

  1. Discuss what a prompt is and how it’s processed
  2. Disambiguate the term CoT
  3. Define how CoT corresponds to context
  4. (Finally) define CoT

Grounding: What happens to your prompt?

  • What is a prompt?
  • How is it processed?
  • What is a thought?
  • How can thoughts be connected into meaningful units?

Note: for the diagrams below, we’ve borrowed and repurposed a few of the excellent diagrams found in Besta et. al. (Besta et al. (2024, v6 2026))

A general prompt pipeline (high level) looks like this:

Drilling down we see how the context is managed in the provider:

Going one level deeper, we can see a reasonable approximation of the steps needed to render an answer to a user’s prompt.

The term Chain of Thought is ambiguous

The term CoT is often conflated in a few ways:

  • Prompting: Where a model is prompted to work things out in a step-by-step, few-shot manner (see: Wei et al. 2022) or zero-shot (see: Kojima et al’s “let’s think step by step”). This technique and similar methods work with any LLM or agentic platform.
  • Trained CoT: A trained behaviour where reasoning models (e.g. Opus 4.x+) are trained by RL. A reasoning model emits an extended reasoning trace before giving its answer, often in a separate reasoning block. (See DeepSeek-AI 2025, for an early published example of CoT in a “production” model)For each step the model takes, it generates tokens and then conditions on them, creating a trace. The trace is not necessarily the CoT, but it can contain useful context. The trace acts as working memory so later tokens can take into account the earlier “reasoning”.This is intuitively why CoT is likely to improve reasoning, as it allows the agent to retain the results of earlier computation on some level. It may also have other properties we don’t completely understand.

NOTE: On reasoning models, traces are unsupervised during RL except for controls around format. The final answer or outcome is what’s rewarded, and this can sometimes lead to novel or unexpected behaviour paths. (See DeepSeek-R1)

You may see the terms “CoT”, “trace”, “reasoning”, “working memory” and “context” used interchangeably. However, they do not mean the same thing:

  • Reasoning is the computation a model performs. This may happen in activations as well as in tokens, and may be wholly or partially absent from the trace.
  • CoT is a method of asking a model to produce intermediate reasoning steps. The CoT is not the steps themselves (that’s the trace), nor are CoTs the process behind the steps (that’s the reasoning)
  • The context is the set of tokens visible to a model at one inference step and may contain instructions, tools, history, retrieved data, prior traces and so on. Context is a limited resource and is managed. A trace is only retained when either the API Provider or Harness retain it.
  • A trace is contained within the context of a model interaction.
  • The trace can serve as a working memory for a model while it is within the model context (remember it may get flushed when it is full)

Dissenting Opinion The ambiguity and uncertainty in this taxonomy of terms reflects an underlying debate around whether or not models actually reason, and the extent of the impact of these items on reasoning outcomes. This is discussed in Apple’s much debated paper “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity” (Shojaee et al. 2025; NeurIPS 2025 poster).

Chain of Thought in context

The trace of the CoT is a part of the context, updated as the user interactions increase. These interactions can be on a per-user, per-conversation, per-organization or on a system-wide basis.

There are no a priori constraints on how these contexts can be used / merged / split / managed other than those imposed by the system (e.g. memory), the provider (e.g. OpenAI), or by policy.

There are a few important gotchas:

  1. The trace you see in your harness / API response / interface may not be the actualreasoning or CoT of the model.In most cases, what you see is a summary of the model’s internal trace, built by another model prior to sending you your answer.It may also not faithfully reflect the model’s underlying reasoning or computation.
  2. Interestingly, Frontier systems can include the complete, unredacted trace from the working model, but it is encrypted (OpenAI 2024; OpenAI Cookbook; Green 2026)
  3. This could be done for a number of reasons – some we know, some we can speculate on.

So, What is a Chain of Thought?

With these elements in hand, we can define a chain of thought!

In layman’s terms, a chain of thought happens when a model writes out any intermediate reasoning steps prior to giving a “final” answer, instead of just giving the final answer.

The definition of a “thought” is not universally agreed upon. In CoT, a thought generally refers to a statement in a paragraph containing a portion of the reasoning process (Besta et al. 2024 v6 2026).

Below is a diagram showing a canonical chain of thought in a simple interaction. Again, note this is a diagram we’ve modified from Besta et al. 2024 v6 2026.

How are Chains of thought (CoT) used?

The intent of CoTs is to increase accuracy on complex, multi-step tasks, especially with few-shot, one-shot or zero-shot tasks (See Wei et al. 2022 and Kojima et al. 2022).

CoT has been used in a few ways:

  • Training: For example, train-time (reinforcement learning during training) and test-time (providing longer periods of reasoning when working on a task) can, in some cases, substantially increase performance in these complex tasks. (See OpenAI 2024 and DeepSeek-R1 for some early work on this)
  • Operationally, chains of thought are also used as a scratchpad or working memory for agents and models. They can be used to help with budgeting effort and overall cost.In Open Weight models, it’s possible to see the full model’s trace; however in Frontier Models (OpenAI, Anthropic) the trace is frequently not communicated directly to the user. What we actually see is a summary of the reasoning provided by another model or agent.However, an encrypted trace may be sent with the response. Why is it encrypted? For a variety of reasons, some we know and others we speculate on:
    1. It may preserve some privacy if intercepted (a single message represents some data exposure, but traces can contain significant details of the entire conversation)It is known that raw CoTs can contain compromising data (see Stolen Thoughts (Green 2026) for a documented vulnerability in CoT infrastructure – now patched – and a description of problematic data contained in the raw CoT)Encryption minimizes exposure for both the provider and the customer in the event of unintentional disclosure.
    2. Monitoring via CoT only works if the CoT is unaltered and hidden from the model (e.g. to prevent reward hacking) (e.g. see the section “Hiding the Chains of Thought“)
    3. This could let Frontier Labs avoid or reduce full disclosure of their reasoning traces and discourage distillation by competing firms.
    4. Directly integrating reward hacking into the system does not work (OpenAI 2024; Baker et al. 2025; OpenAI 2025)

Dissenting Opinion: How effective are CoTs?

This is unclear. There are a lot of open questions about CoT:

  • Faithfulness to performance – it’s not clear if “how faithful a trace is to actual reasoning” affects things like distillation or teaching of student models.
  • We’re not certain what RLVR does, and whether it actually expands a model’s reasoning beyond what the base model is capable of, or if it simply reweights the base model’s behaviour. This is currently disputed. (Wen et al. 2025; Yue et al. 2025)
  • How good a signal it is for monitoring model behaviour. Current consensus disputes its usefulness in all cases, but the practitioners we spoke with flagged it as an important signal and tool for managing their workflows.
  • Whether the monitoring scales – if we have more and more models interacting, it’s not clear whether the value of the trace degrades and by how much.
  • How do we evaluate reasoning as shown in traces? Most labs focus on the final step / answer correctness. It isn’t clear whether validating the intermediate steps (e.g. to see whether they support the final answer) is valuable/helpful.

What’s the State of the Art (SOTA)? How have they progressed?

Below are the three main lines of thinking: Chain of Thought, Tree of Thought, and Graph of Thought.

We can sum these up in the following way:

  • Chain (CoT): is a linear sequence of intermediate reasoning steps. This is the cheapest.
  • CoT with self-consistency (CoT-SC): is several chains from one root node, with the “best” answer selected. “Best” could be based on quantitative, qualitative or algorithmic choice.
  • Tree of Thoughts (ToT): reasoning can branch and backtrack, generating multiple next steps and allowing the model to select the best. This allows for exploration of reasoning spaces.
  • Graph (GoT): allows arbitrary dependencies between thoughts, including aggregation (i.e., a thought with multiple parents), composition (i.e., a thought with multiple children), and so on.

We’ve seen strong indications of how these different topologies can enhance reasoning, however, we haven’t really determined how effective it is for the general case, despite some encouraging work (e.g. See Wen et al. 2025)

The core problem is evaluation and measurement: aggregate performance is easier than specific performance, and interpretability and explainability remain challenging.

There have also been several cases of developing trees, graphs and linear reasoning topologies in agentic workflows, as well, with varying degrees of success.

Of note – evaluation of the efficacy of these more complex pipelines is unclear in the general case, and in our conversations with practitioners, one frequently cited concern was performing proper or complete evaluations in compressed time, or the challenges involved in properly conveying the quality of evaluations (good or bad) to non-technical leadership and decision makers.

Dissenting Opinion: Neuralese and other approaches

OpenAI’s GPT-6 Astra produces far less detailed traces in its output. Why is this? We don’t know for certain (OpenAI hasn’t been forthcoming), but we can make reasonable guesses. Asking models to express thinking in plain English is intuitively inefficient – for example, an optimized sub-language, binary stream or other learned process might be more space-efficient, faster, easier to cache and so on. However, these other languages are hard to understand, may not be mutually intelligible between models and humans, and can lead to more opaque, harder-to-monitor reasoning. This set of languages/communication is colloquially called “neuralese” (Transformer News 2026).

What else can you use instead of, or in addition to, CoT?

Unsurprisingly, there are several reasons (OpenAI 2024) why CoTs, especially in Frontier models, are being suppressed (OpenAI 2024; Anthropic docs; Claude Code issue 49708) or degraded (Baker et al. 2025). So what can be done instead?

It depends on what you’re using the CoTs for. If you’re using them for signals, the easiest answer is to use multiple models in parallel. For example, using a Frontier model as your main processor and a second model showing traces (DeepSeek-AI 2025; DeepSeek API docs) as judge (e.g. an open weight LLM-as-judge) (Zheng et al. 2023; Panickssery et al. 2024; Verga et al. 2024; Goel et al. 2025). This can satisfy on some level a number of cases. Based on our experience, this is likely sound practice regardless, given the questions around a given model’s faithfulness (Lanham et al. 2023; Chen et al. 2025; Arcuschin et al. 2025) or honesty (Baker et al. 2025; Transluce 2026).

Additionally, having a sound evaluation layer is important. Note: not all evals need to be probabilistic. Symbolic or predictable tools are great checks (DeepSeek-AI 2025; Chen et al. 2021; Test-Hardening 2026; Huang et al. 2023) for fixed-output systems like certain types of classifiers or natural language tools. Good evals are suited to purpose and should be built with strong guarantees where possible (Angelopoulos and Bates; Conformal NLP survey 2024), reasonable guarantees where not, and stable, well-understood degradation patterns.

Finally, build your systems defensively. Don’t assume the presence of a given signal (Azure docs; OpenAI forum; Anthropic docs; Digital Applied 2026; Claude Code issue 49708; vLLM docs) if your upgrade cycle is frequent enough to have to factor in changes in the Frontier models.

🗓️ Upcoming Event

If you’re enjoying this article, Graham Toppin will be speaking alongside 40+ other speakers at the MLOps North conference in Toronto, Ontario.

See Tracks and Abstracts

Citations and Sources

  • CoT – Chain of Thought
  • CoTs – Chains of thought
  • DPO – Direct Preference Optimization
  • GRPO – Group Relative Policy Optimization
  • RLHF – Reinforcement Learning from Human Feedback
  • RLVR – Reinforcement Learning with Verifiable Rewards
  • Hiding – specifically w.r.t. CoT, this is where no policy, outcomes or preferences are trained on CoT. This potentially allows monitoring of the model (”reading its mind”) (see OpenAI (2024). Learning to Reason with LLMs)
  • Training time – where a model’s weights are updated, including pre-, mid-, and post-training.
    • Training-time compute – how much compute is spent during these phases. It’s paid once per model.
  • Pre-training – self-supervised next-token prediction on a large, broad corpus of data (e.g., web data, code, books, etc.). This phase produces the base model and uses most of the training-time compute budget.
  • Mid-training – continued training, but on smaller, curated data. This is typically where science, math, coding, reasoning and domain-specific data are introduced to the model, and can include learning-rate annealing and context-length extension. It’s the least agreed-upon term across labs and practitioners.
  • Post-training – focuses on adapting and refining the base model’s behaviour, perhaps via:
    • supervised fine-tuning (on demonstrations and instructions)
    • preference optimization, for example RLHF, DPO
    • CoT training, such as with GRPO or RLVR

Citations and Sources

Subscribe to TMLS Insights

Expert insights powered by the thought leader talks hosted by TMLS, a community of 16,000 AI/ML practitioners who connect through free virtual events and in person at ourTMLS Summit and MLOps World | GenAI Summit.

Subscribe
Table of Contents
Share This Post