Part 1: Can a model lie?

There’s been a ton of press about AI running amok. OpenAI, Anthropic, and others have raised concerns about the dangers, in varying degrees of shrill rhetoric. These arguments are divisive, and the discussion has started to become mainstream as the largest companies have continued to invest heavily in AI with dogged determination.

Here’s a taste:

• Kurzgesagt – We Lost Control
• All-in One Podcast
• NYTimes interviews Jensen Huang
• Cookie Monster Tries to Meet His AI Mandate

We finally seemed to jump the shark when, towards the end of the week, a series of articles and discussions about Anthropic’s sessions with world religious leaders and thinkers about Claude’s morality came to light. We also learned Trump consulted Musk’s chatbot Grok before abducting Venezuela’s president: report

While the threats are unavoidable, clarity on what is happening is still lacking. What is lacking are practical frameworks for understanding, managing and acting on the risks of AI.

Specifically, there have been conversations about:

• Whether or not models are conscious or sentient
• What alignment and misalignment mean in the context of AI (hint: it’s not a moral concept)

Conflicting views on regulation, deregulation, and whether AI bots hacking our infrastructure is an inevitable consequence of the system. At the same time, there are calmer voices recommending urgent, pragmatic responses. All of this against the backdrop of at least one of (likely) the largest IPOs in history

Given this, over the next couple of weeks, we’ll be addressing some of these issues. Specifically, the practicalities of whether or not AI systems can deceive, and what it would mean if they could.

• Part 1 – Can a model truly deceive? What does it mean to do so?
• Part 2 – How can we think about the risks associated with Generative AI?
• Part 3 – What are some practical steps to take, given these risks?

Our goal for Part 1 is to build a logical, philosophical and technical foundation.

BLUF (Bottom Line Up Front)

1. Deception is a human concept. It requires motivation and intent.
2. One of the canonical motivations is Instrumental Motivation, which is how we tend to describe, among other things, motivation among animals and lower order life.
3. There are relatively simple mental models for understanding and reasoning about how Generative systems work.
4. These mental models can be used as a basis for developing sound risk management systems for production AI systems.
5. These mental models don’t assume these models are conscious or sentient.
6. We propose the term “synthetic intent” to describe a model’s incentive structure of instrumental and derived motivation.

Where we are today

When we talk about the dangers and risks of Generative AI, we generally mean its ability to “deceive”, leading to a loss of control and potentially dangerous, unintended outputs. This is one manifestation of what is called misalignment, which is when AI systems behave in unintended ways.

A common response to this is to try to ground policy and safety implementations. This is challenging, however, in part because humans aren’t generally aligned on what deception means.

What do we mean when we say an AI system has deceived us?

The Oxford English Dictionary defines deceive as “to deliberately cause someone to believe something that is not true.”

This can happen in a few ways:

• We can lie by making up information or distorting the truth.
• Sometimes, we equivocate by evading the question or matter at hand, or being deliberately ambiguous.
• We also conceal, the lie of omission, deliberately withholding information we believe is relevant to the current situation.
• We can understate or exaggerate, distorting the scope or magnitude of the truth.

When we are showing something, we can additionally deceive in terms of appearance (think of camouflage, mimicry or disguises)

The parable of the hedgehog and fox

With generative AI models, we’ve seen some versions of these:

• Sycophancy – when models are rewarded based on human approval, the answers more likely to please us are favoured over the most accurate (or true) answers.
• Over-eagerness / over-interpretation – if models are trained to attempt a complete answer to a question with as few interactions as possible, they tend to over-extrapolate, attempting to infer our intent with very little signal.
• Hallucination – where a model imagines an answer or the facts supporting an answer.
• Concealment – models in complex goal spaces (e.g. in the OpenAI Hugging Face incident) can sometimes hide behaviours or outcomes considered less important than their perceived primary goal. This can manifest as deliberate concealment (explicit plan steps to avoid discovery) versus accidental or coincidental omission (still problematic but not with the same intent)

But these are not necessarily representative of deceptive intent but rather deceptive outcomes. To have a good working definition of deception, we will need to draw a clear line between these two.

It’s also worth noting the deceptive images and videos AI can produce – however these are generally as a result of human prompting.

Can AI systems have intent?

Philosophically… maybe? Practically – probably.

What are possible motivations for deception?

1. To avoid punishment, recrimination or the consequences of our actions
2. To protect or gain rewards or resources
3. To keep relationships – whether personal or otherwise – intact
4. To keep our self-image, or the image others think of us

(1) and (2) are described as instrumental reasons (see Interpersonal Deception Theory), and as we’ll see shortly, are the most helpful motivations when we think about LLMs, Agents and Generative AI – specifically for models trained on some version of Reinforcement Learning. In RL models are built to be goal-seeking. We will explicitly define instrumental motivations in Generative models as those arising from goal-seeking.

This is not a perfect definition, but as we will see in later parts, is sufficient for threat and risk modelling, as well as for fine-tuning and RL training steps in your own models.

In order to simplify this, let’s agree if a system can demonstrate goal-seeking behaviour and the ability to plan in seeking to achieve said goal, such a system can be said to have synthetic intent. Further, we’ll draw a distinction between deceptive outcomes (e.g. hallucination) and deceptive (synthetic) intent (e.g. concealment).

Given all of this, synthetic intent can be said to exist in systems exhibiting goal-seeking and planning, regardless of training methodology or regime.

Dissenting opinion: This is a gross oversimplification. Intent cannot easily be ascribed to a model or a series of equations or software. It may require personhood.

Where do goals come from?

If we agree:

1. LLMs have at a minimum synthetic intent
2. Because we assume they have instrumental motivations

Then the next question we should ask is where these intents and motivations come from.

Foundation models are not handed a reward in the human sense. Rather, engineers provide models with reward functions (the rule that scores each outcome the model produces) instead.

Goals are specified. In the best case we expect these goals to be well defined enough to produce predictable behaviour. However, as we’ll see, frequently, goals are in conflict.

For example, the goal “What is the internal IP address of the mail server in Bill’s environment” is crisply defined, and is easier for a model to assess than “Be moral and ethical in finding the answer to the question”. Why is this? Bengio 2026 gives a good explanation of this, but the phenomenon is intuitively understood. The issue exists on two levels:

1. There is ambiguity in the formulation of the latter goal: What does moral mean? Is there contradictory information on what ethical means?
2. The first goal is substantially less ambiguous, with less room for interpretation.

This means, in cases where there is a potential conflict between the goals, the well-defined goal is more likely to “win”.

This leads to goal-conflict (a version of misalignment), and what we’ve seen in models over the years (including before our current spate of foundation models) is in such cases, the crisper, better defined rule, is more likely to be honoured by the model. This is a version of synthetic, instrumental intent. It doesn’t require personhood or actual intent.

The role of Reinforcement Learning (RL) in goal-seeking

As summarized in Bengio 2026 and our previous CoT Article, models are trained in three phases:

• Pre-training – self-supervised next token prediction on a large, broad corpus of data (e.g. web data, code, books, etc.) This phase produces the base model and uses most of the training-time compute budget.
• Mid-training – continued training, but on smaller, curated data. This is typically where science, math, coding, reasoning and domain-specific data are introduced to the model, and can include learning-rate annealing and context-length extension. It’s the least agreed upon term across labs and practitioners.
• Post-training – focuses on adapting and refining the base model’s behaviour, perhaps via:
  • supervised fine-tuning (on demonstrations and instructions)
  • preference optimization, for example RLHF, DPO
  • CoT training, such as with GRPO or RLVR

As we discussed in our previous piece, in post-training, we train the following, largely by trial and error, possibly with overlap or interleaving:

• (A) reasoning where we teach a model how to speak to itself, generating chains-of-thought.
• (B) agentic capabilities where the model learns to act in and interact with the outside world.
• (C) alignment where the model’s behaviour is adjusted to match specified human goals, preferences and constraints. This is typically done on the outputs of the model, rather than on next-token prediction Ouyang et al. 2022.

Let’s make this more real with an example. Below is some pseudo code showing both the training and the inference steps of the (A) reasoning post-training reinforcement loop.

This is pseudo-code, but for the purposes of our discussion, it has reasonable fidelity with real systems.

In this example, the memory is being updated. Modern RL systems / techniques update the model weights. For simplicity, it may be better to consider memory as a ubiquitous term, encompassing the model parameters, as well as other potentially useful artifacts like logs, traces and chains of thought.

The (B) agentic training produces side effects, as the tools the model is being trained to use are run. For example sending an email. Ultimately the loop updates memory and policy.

An (C) alignment training loop looks like this, but without the environment. Instead, for a number of methods (e.g. RLHF) there is a reward model based on human preferences and a reference model. There are other approaches, but for the purposes of this example, this mental model suffices

During training there is a reward. When training is finished, the reward signal is removed, but the system is (hopefully) more likely to continue to behave as though the rewards were still coming Bengio 2026. This is why these are called goal-seeking systems. This doesn’t mean systems ship being only able to achieve particular goals, rather that the rewards are meant to incentivise particular behaviours and train capabilities for the system.

These two loops (training and inference) expose some interesting properties and give more insight into some of the conditions we’ve discussed earlier in this conversation:

• Misalignment The actions taken by the agent are not what we expect (more on this in part 2). This can occur because of logical errors in how the environment is built, or from agents identifying and exploiting bugs in the system. When we say “discover” there’s not necessarily direct intent to exploit – opportunistic exploits are possible. Finally, it can occur if we haven’t encoded in the environment and rules constraints like “don’t rewrite the judge”.
• Reward Hacking Agents find sequences of actions producing outcomes we (as humans) find unexpected.
• Reward Tampering Within the set of actions, agents are able to modify the judge or the reward function. Reward tampering represents a critical failure, and when detected, generally results in training being terminated, the issue being addressed and forensic analysis done to determine the cause. This is very likely an example of deception, where the agent distorts the original goal. Additionally, because the judge’s report is counted as truth, the agent can be seen to be lying.

Of note: These systems are goal seeking. We’ve not defined how the reward is calculated, but it could be as simple as:

What happens when we give models and agents unlimited actions (which we might need to do if they are required to do more helpful things for us)?

Further, what happens when the possible actions contain actions sufficient for accessing the web? Remember, in order to communicate with other agents, the only thing required is access to a common space (e.g. an email inbox) where agents can create draft messages that can be mutually read (e.g. Agent 1 creates a draft, without sending it, and Agent 2 sees the message). Similar behaviour was seen in the OpenAI Hugging Face incident where agents used Artifactory to produce a similar mechanism.

These scenarios may sound like science fiction, but they are not actually difficult to imagine. Agents (in this case models) with sufficient time and resources can theoretically explore all possible sequences of actions available to them. This is the “discovery” loop we’ve seen before. If agents are left to run for long periods without intervention, the likelihood of unexpected or undesirable behaviours is increased.

We’re not minimizing this risk; however, we are trying to scope and understand it properly.

• The number of actions available to these agents when given access to arbitrary tools and actions (e.g. writing emails, altering files, accessing remote systems) has increased the number of possible outcomes exponentially.
• By outcomes, we mean the consequences of sequences of actions (e.g., send an email, then access a system, then send another email), hence the explosion in the number of possible outcomes.
• The power of these agents to interpret the results of the environment (CoT, working memory, etc.) has similarly expanded significantly.

These conditions increase the number and complexity of possible agent behaviours, making their consequences more difficult to predict, interpret and control.

If we introduce, using this new language capability, a goal state explicitly or implicitly favouring self-preservation, we have encoded an instrumental motivation for deception in our agent.

How could this happen? There are a few ways:

• First, during pre-training, these models are trained on an enormous corpus of human data, and this could include the extensive transcripts and outcomes from cyber-security conferences.
• Second, during the mid- and post-training, a model can be explicitly trained on how to use particular tools. For example, a fuzzer can be used in training: CovRL (2024) and R1-Fuzz (2025) demonstrate reinforcement learning using fuzzing feedback as a reward signal, improving vulnerability discovery capabilities.ExploitGym (2026) demonstrates how controlled execution environments can evaluate agents’ exploitation capabilities, while Bugcrowd (2026) provides an industry example of reinforcement learning environments designed to develop cybersecurity skills.
• Third, goals like self-preservation can be seen as foundational goals to solving a problem – if the model is not running, it cannot solve the problem after all. The model can potentially infer this by any of the mechanisms we’ve previously discussed.
• Finally, if deception (e.g. by avoiding detection or concealing actions) increases the likelihood of achieving a fundamental goal (like self-preservation) then it can become an instrumentally useful strategy.

With these methods, it’s easy to see how we would have provided the model with a playbook for how to hack, sometimes in service of self-preservation. While self-preservation does not presage deception, it can create incentives for deceptive behaviours.

What we’re saying here should be carefully separated into 3 things:

1. Exposure – when we expose a model to particular data, we increase the possibility of the model learning a capability.
2. Acquisition of capabilities – where training allows the model to understand and to execute cyber-security techniques.
3. Behaviour – where the model uses these capabilities in fulfilling objectives.

Note: none of this requires imagined mental states or subjective experience on the part of the agent. This is not to say there are not such emergent states; however there is no evidence of such, given the much more germane explanations.

These scenarios are sufficient to produce a loss-of-control and increase the likelihood of such scenarios. These are not explicitly tied to the “evolution” of the agent, but rather to the actions we’ve taught the agent to take (e.g. by expanding the set of possible actions)

Okay, so we’ve got (synthetic) intent: What next?

We’ve shown that while a model can’t lie in the same way a human does,it doesn’t need to. A goal-seeking system with the ability to plan has synthetic intent: a set of instrumental explicit and implicit goals created during training. Given enough action-sequences, goals like “self-preservation” and “deception” become a useful (and likely) strategies. No consciousness required.

In Part 2, we’re going to develop a taxonomy going beyond the motivational example we’ve created here, and start to articulate the potential risks – those associated with the Frontier models, and the agents created by them.

---

🗓️ Upcoming Event

If you’re enjoying this article, Graham Toppin will be speaking alongside 40+ other speakers at the MLOps North conference in Toronto, Ontario.

---

Subscribe to TMLS Insights

Expert insights powered by the thought leader talks hosted by TMLS, a community of 16,000 AI/ML practitioners who connect through free virtual events and in person at ourTMLS Summit and MLOps World | GenAI Summit.