Lies, Damned Lies and LLMs Part 2. A Technical Foundation

A logical foundation, and mapping risks onto agentic systems

Part 2: “There’s a 10% chance they might kill us all” – Jacob Coxon

This is the second part of a three-part series on managing the risks associated with Generative AI systems.

• Part 1 – Can a model truly deceive? What does it mean to do so?
• Part 2 – What is a technical foundation for understanding the risks associated with Generative AI?
• Part 3 – What are some practical steps to take, given these risks?

Part 2: What does it mean to assess risk?

And how to map risks onto your agentic systems

In Part 1, we discussed what it might mean for a model to deceive. We described in some detail the training and inference loops for the post-training of a modern generative AI system, and dissected what it might mean for a model to have intent. We defined the term synthetic intent as a set of instrumental explicit and implicit goals created during training, to give us a reasonable scope for understanding the risk associated with generative systems.

In Part 2, we’re going to describe:

1. What is risk? How do we measure and assess risk?
2. What is a threat model? How does it apply to AI risk?
3. What is a taxonomy for agentic systems? why do we need one?
4. A first set of risks, and how to associate them with the taxonomy

Along the way, we’ll tie this back to the notion of synthetic intent and how we can think about AI risks.

This should set us up nicely for Part 3, where we’ll give some concrete recommendations to help you think about AI risk and some actions you can take to mitigate or manage them.

BLUF (Bottom Line Up Front)

1. Risk is how uncertainty affects outcomes, good or bad.
2. We can think of outcomes, intuitively, as being imagined, imminent and immediate. These can help us to gauge our reactions to possible risks.
3. Risk can be scored on three axes: evidence, impact, time horizon.
4. We build on the threat of AI deception by mapping eight known threats and five known opportunities.
5. Threat modelling can help us to reason about how to protect ourselves and our businesses from AI deception.
6. The Threat Modelling Manifesto is a great starting point for understanding threat modelling.
7. We build on our definition of synthetic intent, from Part 1, to ground our definition of LLM deception without reverting to philosophical positions.
8. The second half maps these risks onto a technical taxonomy of agentic systems

What is emergent behaviour?

Before we jump in, I want to share a story. In the mid-90s as an undergrad, I got the chance to work on the Feature Interaction Problem at the University of Waterloo.

Put simply, a set of features may individually work well on their own, but when combined, may produce surprising or unanticipated behaviour.

The canonical example is where Call Forwarding and Call Waiting interact.

Additionally, even though each of these features were deterministic, things like concurrency meant, again, combining them created vulnerabilities a motivated hacker could exploit.

Most importantly, the interactions produced difficult to resolve ambiguous behaviour (e.g. when a user with both call waiting and call forwarding enabled receives a call while they are on the phone, should the user be notified, should the call be forwarded, or is there another acceptable behaviour?).

And these interactions weren’t frequently difficult discover before shipping a system.

The results of these unintended interactions are an example of emergent behaviour.

Today, we’re going to discuss how emergent behaviour in Generative AI systems can produce some dangerous and useful effects (they tend to go hand in hand).

Understanding what we want Generative AI to be, given this emergent behaviour has never been more important. It is possible for Generative AI at scale to do things it was not designed to do, things it was never intended to.

We need to be able to see the world from multiple perspectives. We need to be foxes, not hedgehogs (“A fox knows many things, but a hedgehog knows one big thing.”) To wit, a hedgehog’s response to things is to curl up into a ball – it’s one known “thing” about situations; the fox, on the other hand, needs to be able to see the world through the lens of multiple experiences and ideas.

What is risk?

Risk is how uncertainty can affect outcomes (desired or undesired, positive or negative, good or bad). There are a lot of ways to model risk. We’re going to pick a straightforward one, and then use it to describe the risks surrounding synthetic intent

Disambiguation warning: despite this, you’ll frequently see risk or threat being used to describe a possible negative outcome and opportunity describing possible positive ones. The context usually shows if we’re talking about how uncertainty affects outcomes, or threats. We’ll make this explicit later in the article.

First of all, when we discuss risk, we are concerned with:

• a given situation or scenario AND
• a taxonomy or ontology – describing the world view associated with it (e.g. in the case of LLMs, we might describe the prompt pipeline, and how those systems are trained)

Once we have these, we can frame the positive and negative outcomes these can produce. (We’ll do this shortly)

Let’s start with a simple framework to think about risk – threat horizons. These are a heuristic contemplating probability, proximity and certainty. It isn’t a formal way to think about risk, but it is a good way to think intuitively about threats and how our options change as those threats develop.

We’ll describe a more formal, separate framework shortly; let’s develop the intuition first.

Threat horizons

1. Imagined – the outcome might occur, but we don’t know when. We have lot of options in how we choose to respond.
2. Imminent – the outcome is increasingly likely, and our options begin to narrow. We may have seen situations like this before.
3. Immediate – the outcome seems inevitable and there is limited time to react. At this point our options have narrowed and we have only one or two ways to react.

To frame the risk associated with a given situation we’re going to look at it from three dimensions:

1. Evidence – what evidence do we have to support the occurrence of each outcome?
2. Impact – for each outcome, what is the potential impact?
3. Time horizon – when might it occur?

When we’re evaluating evidence we’re asking how much the data we see (available observations, experiments, etc.) supports the possible outcomes we’re evaluating.

1. Hypothesized – an outcome argued or conjectured, without any experimental support.
2. Demonstrated – the outcome has been reproduced in controlled settings.
3. Observed – the outcome has been seen in deployment.

Let’s also define time horizons or urgency as falling within three groups:

1. Near term – within the next 3 months (we’ll draw this as a circle)
2. Medium term – within the next 3-12 months (a square)
3. Long term – over 12 months away (a triangle)

Finally, let’s think of impact as a simple sliding numeric scale between +3 and -3, with 0 being neutral or unknown impact, the positive values being opportunities (blue) and the negative values being threats (red).

Dissenting opinion: This is one of many ways of thinking about risk. However we’ve found this is the easiest to communicate to beginners, and has reasonable fidelity with more sophisticated models.

---

If you’re enjoying this article, Graham Toppin will be speaking alongside 40+ other speakers at the MLOps North conference in Toronto, Ontario.

See Tracks & Abstracts

---

How should we think about risk?

Let’s develop some systematic, transparent and falsifiable ways of looking at the current situation.

First of all, what are some of the risks we see with current AI? We’ll use Bengio 2026 yet again to provide some basic ones1:

1. R1 – sycophancy: systems’ tendency to tell us what we want to hear, rather than what is true.
2. R2 – reward hacking: when systems optimize for rewards not fully matching our (human) intentions. This can be explained by Goodhart’s Law – when a metric or goal is optimized for, it stops being a good metric.
3. R3 – reward tampering: a more extreme version of reward hacking where systems mess with the way they are evaluated.
4. R4 – rationalization: when we are able to see a system’s internal chain-of-thought where when two goals conflict, the result is frequently rationalized, unanticipated and undesired.
5. R5 – evaluation awareness: where models are able to determine they are being evaluated, and change behaviour based on this.
6. R6 – self-preservation: if a system is not running, it cannot fulfill any of its goals, present or future. This could theoretically create a deeper goal that when coupled with, for example, rationalization, produces unanticipated behaviour.
7. R7 – harmful actions: where systems take actions considered to be crimes if taken by a human, or cheat on assigned tasks while evading detection or escaping containment.
8. R8 – covert or unplanned coordination: where systems coordinate to conduct operations (e.g. cyber-attacks) or simply coordinate and communicate when not explicitly told to so.

Dissenting opinion: This is a very simplistic analysis of the risks, but sufficient for this article. The risks as stated are not cleanly separated – there is some overlap and dependence. A more advanced risk model might better account for these issues, but would become far too detailed.

The same training regime (pre-training, post-training with RL-loop) and architecture (generative, transformer based deep-learning general purpose technology) giving rise to these risks also produce some real gains, though! Here are a few possible outcomes with some concrete examples:

1. O1 – Faster routine knowledge work: given access to ChatGPT, a cohort of 453 knowledge workers saw their writing time fall by 40% and quality rise by 18% Noy and Zhang (2023), Science.
2. O2 – Narrowing the novice–expert gap: issues per hour rose by 15% on average. It was speculated the use of LLM based tools allowed practices of more able representatives to help on-board newer staff Brynjolfsson, Li and Raymond (2025), Quarterly Journal of Economics.
3. O3 – Learning gains from designed tutors: In a randomized control test, it appeared students learned more and felt more motivated when using an AI powered tutor versus an active learning class Kestin et al. (2025), Scientific Reports.
4. O4 – Correcting entrenched false beliefs: personalized dialogues between AI agents and conspiracy theorists appeared to lower their beliefs by 20% Costello, Pennycook and Rand (2024), Science.
5. O5 – Reduced clinical documentation burden: Ambient AI scribes substantially reduce the administrative burden on clinicians. Lukac et al. (2025), NEJM AI; Pragmatic RCT of ambient AI (2025), NEJM AI

We’ve deliberately not shown any opportunities from primary sources like the Frontier Research labs, VCs or startups, as they are inherently biased and can be found in public discourse.

Dissenting opinion: both these risks and opportunities have been picked in a non-peer-reviewed way and represent inherent biases we have not directly addressed in this article.

With this grounding, we can create a better model for thinking about some of the risks associated with generative AI.

Risk assessments

Now, let’s see all of these systems from the previous section AND our risks and opportunities on a single chart.

Risk Matrix

Our placement of the risks and opportunities is a bit arbitrary for this example, but gives an immediate assessment of the current situation. We will provide more concrete recommendations for how to place the risks in Part 3.

Great! Now, how should we think about the bad potential outcomes – the threats – associated with each of these? Each of the risks and opportunities exists in the abstract. Without being able to associate them with the system we are managing (e.g. an agentic pipeline, a training regime), defining policy will be difficult. This is where threat modelling comes in: it will allow us to think about them within our specific context.

Threat models

Threat modelling asks what we can do to protect something of value, usually from the risks concerned with privacy, security and safety.

Note: AI Safety is its own large, complex field containing many areas of active research. We do not explicitly deal with it in this article.

The threat modelling manifesto gives a useful framework for answering this. At its core, it asks 4 questions:

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good enough job?

The manifesto also identifies useful principles and values. We won’t go into all of them here but we’d encourage you to take some time to read them on your own. The manifesto is not very long, and is well written.

For the purposes of this article we’re going to focus on the risks associated with AIs deceiving us. Okay let’s begin:

Synthetic intent is “a set of instrumental explicit and implicit goals created during training”. This is our pragmatic simplification for the larger questions of deceit and intent.

So, why does synthetic intent matter?

It gives us a way to reason about why an agent might appear to pursue unexpected goals, or take unintended actions.

It allows us to avoid questions about whether or not a model is conscious or “intends” to deceive in the same way a human might. Instead, we can look at the goals it has learned, the direct and indirect actions it can take, and the environment in which it operates.

Threat models allow us to describe these issues, and to reason about them.

1. What are we working on? We’re looking at goal-seeking systems built using the three post-training loops: reasoning, agentic and alignment training. As discussed previously, synthetic intent is a way of reasoning about the threats these risks represent.We can describe these using the following:
   • A lifecycle taxonomy (showing how a model is built and used, from training to inference)
   • An infrastructure deployment diagram (showing how agents are deployed and used, from data centre to API to remote machines);
   • A generalized deployment (showing two primary use cases: coding and intelligence pipelines);

2. What can go wrong? We’ve already covered this in our risk matrix. Further, the stages (exposure, acquisition of capabilities and behaviour) will give our threat model its stages:
   • what the model saw
   • what it can do
   • what it does
   And our failure modes are:
   • misalignment
   • reward hacking
   • reward tampering

2. What are we going to do about it? This is the question we’ll answer in Part 3. But we’ve already touched on some solutions in Part 1:
   • constrain the action set of the model
   • keep the judge out of the agent’s reach
   • limit unsupervised run time.

2. Did we do a good enough job? This too we’ll discuss more in Part 3. We know, for instance, the risk surface area expands substantially every time we give an agent a new action it can use in pursuing a goal.

A life cycle taxonomy for generative AI

Infrastructure deployment diagram

Generalized deployment diagram

The threat manifesto also names a number of anti-patterns, which are especially illuminating. Let’s name them and go through them one by one as they might relate to LLMs deceiving us.

Are we wrong?

This is the most important question to ask ourselves. If we are able to frame our thesis against this question, we can move from an assumption to something we can trust.

So, are we wrong about synthetic intent?

Synthetic intent was framed based on:

• Observed behaviour AND
• Inferred mechanism (e.g. Reinforcement learning, etc.)

Explicitly, we do not make any claim on the internal states of a model.

Based on this, we could state that synthetic intent would fail if evidence shows either: – the observed behaviours we’ve seen (deception, reward hacking, etc. don’t require goal states – goals created in training cannot explain the deception we’ve seen, and only adding internal states would provide a solid explanation.

So, what do we know?

Could we be wrong2?

If our framing of synthetic intent is:

1. Too strong, then the observed behaviours don’t actually need goals during training.

• This could be true if: A. do deceptive outcomes occur at the same rate without reinforcement learning or planning? B. can systems without any training exhibit instrumental goal-seeking behaviour?
• In a nutshell, do we see instrumental goal seeking without training; for example in the symbolic systems described earlier when we discussed feature interactions?
• If it is true, then case (A) would weaken our explanation of how synthetic intent arises, while (B) would challenge whether training is necessary at all.
• Neither would definitively disprove synthetic intent, but would substantially reduce the utility of synthetic intent.

2. Too weak, then the goals created – both implicitly and explicitly – in training do not explain the observed behaviours.

• If this is true, then A. do goal-directed behaviours appear without any reward, data or tool exposure? B. More deeply, do models show indicators predicted by theories of consciousness?
• More speculatively, this might mean the “no consciousness required” simplification may fail; however there is no settled theory of consciousness3. As a practical matter, it would substantially weaken synthetic intent if the goals in training failed to explain the observed behaviours.

Of note, a recent paper has asserted alignment and interpretability to be achievable. This is an early paper (the only thing available is a blog post, the technical paper has not been published, and no one has reproduced the claims). This is an example of the volatility of the field. We’ll be watching this as it develops!

What’s next?

In this article, we’ve built a practical risk management framework, and a threat model, focusing on the question of whether or not an LLM can lie, and the concept of synthetic intent as a pragmatic simplification. We also built several taxonomies and tied the risks and opportunities associated with LLMs to them.

By separating the model, how it’s trained, the actions it can take and the environment it is operating in, it becomes clear that most of the risks we’ve identified are as a result of not just training and learned behaviours, but also of choices made during design and deployment and are not intrinsic to Generative AI as a whole.

This challenges a number of the assumptions we make about technology:

1. We tend to see technology as monolithic – it is not.
2. Because of this, we also tend to see a technology or a technological direction as inevitable – they are not.
3. We also frequently fail to shape technology direction and vision in ways beneficial to all of us. This is again, not inevitable.
4. We always hope the answer to the ambiguity and conflict is simple – in most cases it is not.

By breaking Generative AI systems and the scenarios around them down, we hope we’ve helped to demystify the technology. This should also give us a solid framework leading into Part 3 to show real-world examples of how to mitigate risks and take advantage of opportunities within your own ecosystems.

1 Bengio 2026 provides several references for these risks.

Of note, R4’s citations are not direct causal relationships (traces may or may not actually relate to a model’s internal state) and gives indirect support to synthetic intent while R6 similarly does not provide definitive proof of self-preservation; it only describes behaviours related it.

1. R1 – Sycophancy – Towards Understanding Sycophancy in Language Models (2023)
2. R2 – Reward hacking – OpenAI-Hugging Face Incident Investigation (METR, 2026)
3. R3 – Reward tampering – OpenAI-Hugging Face Incident Investigation (METR, 2026)
4. R4 – Rationalization – OpenAI-Hugging Face Incident Investigation (METR, 2026)
5. R5 – Evaluation awareness – AI Sandbagging: Language Models Can Strategically Underperform on Evaluations (ICLR, 2025) – Alignment Faking in Large Language Models (2024)
6. R6 – Self-preservation – Alignment Faking in Large Language Models (2024) – Peer-Preservation in Frontier Models (2026)
7. R7 – Harmful actions – OpenAI-Hugging Face Incident Investigation (METR, 2026)
8. R8 – Covert or unplanned coordination – OpenAI-Hugging Face Incident Investigation (METR, 2026) – Peer-Preservation in Frontier Models (2026)

2 As discussed in Part 1, establishing a causal relationship between instrumental goals and observed behaviour is challenging because of the opaque nature of these systems. Interpretability and explainability of these systems remain an open question.

3 So, the burden of proof in this case would be deeper than simply disproving synthetic intent

---

Subscribe to TMLS Insights

Expert insights powered by the thought leader talks hosted by TMLS, a community of 16,000 AI/ML practitioners who connect through free virtual events and in person at ourTMLS Summit and MLOps World | GenAI Summit.