Navigate this blog
In this post
Can Machines Learn from the Consequences of Their Actions?
So far, we have looked at how machines can learn by analysing patterns in data. We explored supervised learning, where their behaviour can be corrected by comparing how far their predictions are from the actual values (Refer to How Do Machines Think and Learn?). We also saw an approach where machines can discover patterns without explicit supervision in the form of expected outputs, but with their behaviour guided by algorithms, objectives, and hyperparameters (Refer to How Do Machines Learn without Supervision?).
In this blog, let’s explore another way of conditioning the behaviour of a machine: learning through the consequences of its actions.
From Effect to Consequence
When I was first introduced to reinforcement learning as a concept, it was interesting to discover that some of its roots could be traced back to B. F. Skinner’s work on operant conditioning. This, in turn, built upon Edward L. Thorndike’s earlier work on the Law of Effect.
Thorndike’s Law of Effect, first introduced in his work on animal learning in 1898 and further developed in 1911, suggested that a behaviour followed by a satisfying consequence is more likely to be repeated. On the other hand, a behaviour followed by an unpleasant or discomforting consequence is less likely to be repeated.
Thorndike demonstrated this through his famous puzzle-box experiments with cats. A hungry cat was placed inside a box, with food kept outside. The cat initially tried several ways to escape, such as clawing, pushing, and pulling at different parts of the box. Eventually, it accidentally operated a mechanism that opened the door. With repeated trials, the cat gradually learnt to escape more quickly.
Building on this idea, B. F. Skinner in 1938, published The Behavior of Organisms, in which he developed his work on operant conditioning and used a specially designed experimental chamber, popularly known as the Skinner Box, to study how behaviour could be strengthened or weakened by its consequences.
Operant conditioning, also referred to as instrumental conditioning, is a form of associative learning in which an organism learns to associate its behaviour with the consequence that follows. Over repeated interactions, these associations influence whether the organism is more or less likely to repeat that behaviour.
This brings me back to another interesting concept on “Association Rule Mining” we explored in my previous blog (Refer to How Do Machines Learn without Supervision).
While the two may sound similar, there is an important difference. In association rule mining, the machine discovers relationships and patterns among items in the data. In operant conditioning, the organism learns an association between its own behaviour and the consequence that follows. In one case, the machine discovers associations in data; in the other, the association helps shape future behaviour.
Skinner conducted several experiments on animals, particularly rats and pigeons. In one of his classic experiments, a rat placed inside a Skinner Box could press a lever to receive a food pellet. Initially, the rat explored its surroundings, but as it discovered the connection between pressing the lever and receiving food, it began repeating that behaviour more frequently.
Skinner used two important concepts to describe how consequences influence behaviour:
- Reinforcement strengthens a behaviour, making it more likely to occur again.
- Punishment weakens a behaviour, making it less likely to occur again.
Another interesting experiment was conducted by Shigeru Watanabe and his colleagues in 1995. They trained pigeons to distinguish between colour slides of paintings by Monet and Picasso. The pigeons learnt to peck the appropriate response key for each category and were rewarded during training. Interestingly, they could also distinguish between new paintings by the same artists that they had not encountered before!

This demonstrated that pigeons could learn to discriminate between complex visual patterns and generalise what they had learnt to previously unseen examples.
Can Machines Learn Similarly?
This is exactly where the reinforcement learning approach comes in.
In reinforcement learning, a machine learns through interactions with its environment. It takes an action, observes what happens as a consequence, and receives a reward signal. Over repeated interactions, it learns which actions are more likely to help it achieve its goal.
Let us draw a parallel between the three important elements of operant conditioning, often described through the ABC model (Antecedent, Behaviour, Consequence), and the corresponding concepts in reinforcement learning.
Let’s call our machine the agent.
- Antecedent: The situation or stimulus that occurs before a behaviour. In reinforcement learning, this is broadly comparable to the agent’s current state or observation of its environment.
- Behaviour: The response of the animal to the situation. In reinforcement learning, this corresponds to the action selected by the agent from its available action space.
- Consequence: The outcome that follows the behaviour. In reinforcement learning, this is represented through a reward signal, which tells the agent how desirable the outcome of its action was.
There is one more important element here. When an agent takes an action, the environment may transition from one state to another. The agent observes this new state and receives a reward. This interaction continues, allowing the agent to learn which sequence of actions is likely to produce the best outcome.
The objective is not necessarily to maximise an immediate reward, but to maximise the cumulative reward over time.
What’s Cooking in the Real World?
One of the most famous examples of reinforcement learning is AlphaGo, the AI system developed by DeepMind to play the game of Go.
AlphaGo initially learnt from thousands of expert human games. It was then trained through reinforcement learning, playing thousands of games against different versions of itself and improving its strategy based on the outcomes.
The system used deep neural networks along with search algorithms to select moves and estimate the likelihood of winning. Its remarkable performance against human champions demonstrated the potential of combining deep learning with reinforcement learning.
If you have watched the AlphaGo documentary, you might remember the remarkable moves that challenged even the most experienced Go players.
But where else can we apply reinforcement learning?
Some interesting applications include:
- Robotics: Training robots to walk, balance, grasp objects, and navigate their surroundings through repeated interactions.
- Game-playing agents: Learning strategies in complex games such as chess, Go, and video games.
- Autonomous navigation: Learning how to navigate different environments by taking actions and evaluating their outcomes, often through simulated environments.
- Industrial process control: Optimising control decisions in complex and changing environments.
I also see interesting possibilities for reinforcement learning in building the harness for our LLM-based agentic systems.
For instance, organisations building agentic systems for automated penetration testing often struggle to ensure that an agent stays within the defined scope of an engagement. Reinforcement learning could potentially help an agent learn to select actions that make progress towards the testing objective while accounting for scope-related constraints and feedback.
Another interesting possibility could be to use reinforcement learning to help an agent navigate a red-teaming engagement plan, selecting the next appropriate step based on the current findings and the outcomes of previous actions.
Of course, in security applications, we cannot rely on a reward signal alone to enforce safety. Explicit guardrails, access controls, scope restrictions, and human oversight remain essential.
How Easy Is It to Train a Machine?
Do we need more compute? How about preparing the dataset?
Unlike supervised learning, where we typically prepare a labelled dataset, reinforcement learning requires us to define an environment in which the agent can interact and learn.
We also need to define the actions available to the agent and, most importantly, the reward function that tells it whether its actions are helping it achieve the intended objective.
The reward function is particularly important. If we define it incorrectly, the agent may learn to maximise the reward in ways that do not necessarily align with what we actually want it to achieve. This is often referred to as reward hacking.
Training can also be computationally expensive. Depending on the problem, an agent may need thousands or even millions of interactions with its environment before it learns a reasonably good strategy.
This is why simulations are often used to train reinforcement learning agents before deploying them in real-world environments. They provide a controlled and relatively safer way to experiment with different actions and outcomes.
In summary, designing the environment, reward function, and training process is not necessarily straightforward.
Can We Draw Similarities with LLMs?
I see some interesting parallels with the way we guide the behaviour of LLMs, although the underlying learning mechanisms are different.
At my previous organisation, I was tasked with helping an LLM learn the native query language of the product. One of the approaches I used was to provide examples that associated certain commands with a query language the model was already familiar with. The few-shot examples in the prompt helped the model connect unfamiliar commands with patterns it had already learnt, making it easier to generate the expected queries.
In a way, this reminded me of associative learning, where previously established associations help guide a response. However, there is an important distinction: few-shot prompting does not update the model’s parameters or train it through reinforcement learning. Instead, it provides examples within the prompt context that help guide the model’s response.
So, while these approaches are not the same, they offer an interesting perspective on how associations, experience, and feedback can influence learning and behaviour.
(Note: Images used in this blog post were generated with the help of ChatGPT.)