Reinforcement learning, from a small agent to language models

Reinforcement learning trains an agent to choose actions from the consequences they produce. You will build a small agent that learns from rewards, then use the same concepts to understand how language models are trained with human feedback or verifiable answers.

Start with basic Python, averages and probabilities. The equations are explained beside concrete examples; neural networks enter after the first working agent. This complements Fine-tuning language models, hands-on, which covers model adaptation and memory-efficient training. The LangGraph article covers running agent workflows; this article covers learning their decision rules.

The Python lab below was executed on CPU. The language-model training experiment is a proposed lab and has not been run. Numerical teaching examples are original and explicitly labelled. Source documentation was checked on 30 September 2026.

• The essentials
• 1 · The learning loop
• 2 · Rewards and return
• 3 · Learning values
• 4 · A Python agent
• 5 · The algorithm map
• 6 · Policy gradients and PPO
• 7 · Language models and RLHF
• 8 · GRPO, with numbers
• 9 · Designing the reward
• 10 · Your first Unsloth experiment
• 11 · Knowing whether it worked
• 12 · A learning path
• Sources

The essentials

• A policy chooses actions. Learning changes that rule using feedback from what happened. Running a model repeatedly, or adding memory to an agent, does not itself perform reinforcement learning.
• The objective concerns accumulated reward. A useful action can have an immediate cost if it enables a better outcome later.
• Exploration supplies the evidence. An agent that never tries an alternative cannot learn its consequences from experience.
• A value predicts future return. It helps connect early decisions with rewards that arrive much later.
• Language-model training uses the same ingredients. Generated tokens are actions, and the model is a policy. A scorer supplies feedback.
• The reward needs its own evaluation. A rising score can reflect a useful skill, memorization or an exploited loophole.

1 · The learning loop

An agent makes decisions; its environment produces the consequences. The agent receives an observation, selects an action using its policy, and receives a numerical reward plus a new observation. Training uses these transitions to improve the policy. An episode is one run until a defined ending, such as reaching a destination. A rollout is a collected sequence of interactions, which may cover only part of an episode. See Spinning Up's core concepts.

Our original teaching environment is a short corridor. The agent starts in position 3, moves left or right, and stops at either endpoint. It gets +1 for reaching 6, −1 for reaching 0, and −0.1 for every move that ends in an interior position.

failure                 start                 success
   0 —— 1 —— 2 —— [3] —— 4 —— 5 —— 6
  −1               left / right                +1

The state describes the environment sufficiently to predict its next transition. Observations can hide relevant information: a game screenshot may omit velocity. A Markov decision process (MDP) models states, actions, transition probabilities and rewards under the assumption that the current state suffices. When the observation does not, an agent may need history. Definitions and formalism.

2 · Rewards, return and exploration

Write r_t for the reward produced by action a_t, which takes the environment from s_t to s_(t+1). The return adds subsequent rewards. A discount factor γ, pronounced gamma, controls how strongly later rewards contribute:

G_t = r_t + γ r_(t+1) + γ² r_(t+2) + …

For the corridor's shortest successful path, the rewards are −0.1, −0.1, +1. With γ = 0.95, its discounted return is −0.1 − 0.095 + 0.9025 = 0.7075. Its undiscounted return is 0.8. These are different measurements of the same trajectory. The lab trains with the former and reports the latter.

A discount of zero considers only the immediate reward; values closer to one give later consequences more weight. For continuing tasks, a discount below one also helps keep the sum finite when rewards are bounded. The objective is expected return across possible trajectories. Return and the RL objective.

Exploration tries actions to obtain information; exploitation uses what currently looks best. Our lab uses an epsilon-greedy policy: with probability ε it chooses randomly, otherwise it takes an action with the largest current value. Its chosen schedule starts at ε = 1 and decreases to 0.05. These are exercise settings, not universal recommendations.

A bandit is a useful simpler starting point: choose one option and receive a reward, without actions steering a sequence of future states. The corridor adds delayed consequences. In an LLM task, scoring one complete answer can resemble a contextual bandit, while generating that answer token by token is sequential. Multi-step tool use adds further transitions.

3 · Learning values with Q-learning

A value function predicts return. V(s) evaluates being in a state under a policy; Q(s,a) evaluates taking an action there and then following a policy. In the corridor, a table with two entries per position is enough. The decision rule selects the action with the higher estimate.

Q-learning moves an entry toward a target made from the observed reward plus the best estimated value at the next state. This is a sample-based version of the Bellman optimality relationship, which decomposes return into its immediate and future parts:

target = reward + γ × max_a Q(next_state, a)
Q(state, action) += α × (target − Q(state, action))

For a terminal transition: target = reward

Here α, the learning rate, controls the size of an update. The difference between target and current estimate is the temporal-difference error. Using a current estimate inside the target is called bootstrapping. Gymnasium explains this update and its terminal case.

Original numerical example: suppose Q(3, right) is 0.20, the reward is −0.10, and the highest Q at 4 is 0.80. With γ = 0.95 and α = 0.20, the target is 0.66 and the new entry is 0.20 + 0.20 × (0.66 − 0.20) = 0.292. One transition revises an estimate; it does not establish the true value.

Rewards propagate through repeated experience. A successful transition from 5 teaches that moving right can pay. Later updates at 4 use that estimate, and updates at 3 use what was learned at 4. This addresses credit assignment: connecting a later outcome with the earlier decisions that enabled it.

4 · Train a small agent on your CPU

Download lineworld.py and run python3 lineworld.py. It uses Python's standard library, with no packages, account or GPU. The full file includes the environment, random tie-breaking, training and a separate evaluation loop. These are the central training lines:

next_state, reward, terminated = step(state, action)
future = 0.0 if terminated else max(q[next_state])
target = reward + GAMMA * future
q[state][action] += ALPHA * (target - q[state][action])

The program trains for 600 episodes with α = 0.2 and γ = 0.95. It evaluates five independently trained tables on 1,000 episodes each, with learning disabled and no epsilon exploration. The random baseline gets its own evaluation. Training and evaluation use separate random-number generators.

The observed CPU result on 30 September 2026 was:

None of these evaluations hit the 100-step collection cap. For the last trained table, Q(3, left) was approximately 0.4435 and Q(3, right) was 0.7075. The second value matches the discounted shortest-path return calculated above. This is a deliberately easy, deterministic environment with a fixed start; perfect performance here establishes that this example works, not that it generalizes to a game or a language model.

Change one thing and predict the result

1. Change the interior reward from −0.1 to +0.2. An agent may learn to keep moving inside the corridor to collect rewards. Check endpoint success as well as return.
2. Set GAMMA = 0. Estimate which actions now look good from the starting position, then compare the learned behavior.
3. Start evaluation from 1, 2, 4 and 5. Explain what this probes and why it still says nothing about a different corridor layout.
4. Print one complete evaluation trajectory: state, action, reward and next state. Explain every update before replacing the table with a network.

Moving to Gymnasium

Gymnasium standardizes the interface to environments. Its current API returns observation, info from reset() and five values from step(). Calling the environment with random actions collects experience; a learning algorithm must still update the policy. Basic usage.

observation, info = env.reset(seed=42)
next_observation, reward, terminated, truncated, info = env.step(action)
episode_over = terminated or truncated

Termination means the task reached a terminal state. Truncation means collection stopped for an external reason, such as a practical time cap. Reset after either, but for the Q-learning target remove the future value only after termination. Use the final observation before resetting. If a deadline is part of the task itself, it is a finite-horizon problem: the remaining time belongs in the state and expiry is terminal. Handling time limits.

5 · Where the main algorithm families fit

Deep reinforcement learning uses neural networks to represent a policy, a value function or an environment model. It becomes useful when a table cannot represent all the observations, such as images or long text contexts.

These categories overlap. Another distinction concerns the data: on-policy learning uses data from the current or recent policy; off-policy learning can learn from a different behavior policy. Q-learning's target is greedy even when its data comes from epsilon-greedy exploration. Offline RL restricts learning to a fixed dataset; it adds the problem of evaluating actions poorly represented in that dataset.

The distinctions above follow Spinning Up's algorithm taxonomy. For a next implementation exercise, use a maintained library on a small standard environment. Stable-Baselines3's practical guide discusses algorithm selection and why multiple runs matter. The David Silver course offers a longer path through control, approximation, planning and exploration.

6 · Policy gradients and PPO

A neural policy π_θ(a | s) assigns action probabilities using weights θ. A policy gradient estimates how changing those weights changes expected return. In a basic episodic implementation, a sampled action's log-probability is weighted by its later return. Subtracting a state-dependent baseline can reduce noise without changing the expected gradient. The advantage measures how much better an action is than the state's expected outcome:

A(s, a) = Q(s, a) − V(s)
training signal ≈ gradient of log π_θ(a | s) × estimated advantage

A positive estimate encourages the sampled action; a negative estimate discourages it relative to the baseline. A critic estimates values to help calculate this signal. The reward scorer and the critic have different jobs: one grades outcomes, the other predicts expected future return. Policy-gradient derivation.

Proximal Policy Optimization (PPO) controls updates using the probability ratio between the policy being trained and the policy that generated the rollout. PPO-Clip uses a clipped surrogate objective:

ratio = π_new(action | state) / π_old(action | state)
objective = min(ratio × A,
                clip(ratio, 1 − ε, 1 + ε) × A)

The clipping ε here is a PPO parameter, separate from epsilon-greedy exploration. The objective reduces the incentive to push a sampled probability too far in a favorable direction. It is not a hard guarantee that every probability ratio stays within the interval. Schulman and colleagues' PPO paper.

Original example: with A = +2, old probability 0.10, new probability 0.14 and clipping ε = 0.2, the two terms are 2.8 and 2.4; the minimum is 2.4. For this sample, raising the ratio further offers no additional benefit through that clipped term.

In language-model RL, a separate frozen reference policy can anchor behavior through a KL-divergence penalty, which penalizes a distribution moving away from the reference. The old rollout policy and the reference policy are distinct: the former changes as new data is collected, while the latter is often the initial model. PPO can be used outside language models and does not inherently require a human-feedback reward.

7 · The same loop inside a language model

A language model's policy chooses the next token, a unit of text. Its observation includes the prompt and generated prefix. The completed answer receives a score; training assigns that signal back to sampled tokens. For a tool-using agent, tool results also enter the next observation. More generated text supplies more actions, not automatically better reasoning.

The classic InstructGPT pipeline first trained on demonstrations, then trained a reward model on human rankings, then optimized the policy with PPO. Humans supplied the comparison dataset; they did not manually score every gradient update. The reward reflected those annotators' preferences, and the resulting models still made factual and other mistakes.

DPO derives a direct preference-training objective from a KL-regularized reward formulation. It connects mathematically to RLHF but its standard training loop does not need a separately fitted reward model or fresh generations at each step. It remains useful to distinguish these computational procedures even when both are called “alignment.”

DeepSeek-R1-Zero explored RL from an already pretrained base model without preliminary SFT. It did not start from random weights. The full R1 system added supervised stages and further RL to address issues including readability and language mixing. SFT is therefore a common practical starting point, not a mathematical prerequisite for all RL.

8 · GRPO, with numbers you can change

Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath, samples several answers to the same prompt. Their scores provide a relative baseline, avoiding a separately learned critic. It can still use a learned reward model: replacing that model with a programmatic verifier is a separate choice.

For an illustrative group of four answers with rewards [0, 0, 1, 1], the mean is 0.5 and the population standard deviation is 0.5. Standardizing gives advantages [−1, −1, +1, +1]. Successful samples get a positive signal compared with this group's baseline.

mean = average(rewards for this prompt)
advantage_i = (reward_i − mean) / (standard_deviation + small_constant)

The interactive example uses population standard deviation and a small constant of 10⁻⁸. Libraries can use a different variance correction, scaling rule or loss normalization. The calculation explains the relative signal; it is not a complete trainer.

Try “all fail” and “all succeed.” Both produce zero relative reward advantage because there is no within-group contrast. The optimizer may still have other terms, such as KL regularization. With a uniformly failing task, more identical failures do not provide a useful ranking; improve task difficulty, sampling or the starting model. With uniform success, try harder examples or measure whether further training is needed.

GRPO uses this advantage in a policy-update objective related to PPO. A positive advantage supplies a direction of improvement; it is not an instruction to add that number to a token probability. Gradients, clipping, shared weights and optional regularization determine the actual update.

The current TRL implementation exposes several normalization and loss variants. Its documented default KL coefficient is zero, so a reference penalty is not automatic. Record the actual configuration instead of assuming that a class named GRPOTrainer reproduces every detail of the original paper.

9 · Designing a reward that measures the task

Reward hacking or specification gaming occurs when a policy earns the specified reward through behavior that misses the intended outcome. DeepMind documents a boat-racing agent collecting repeated rewards instead of completing the race, among other examples. A more capable optimizer can make a faulty objective's consequences more visible. Krakovna and colleagues.

For an original learning task, ask a model to answer generated arithmetic problems using exactly one JSON object, such as {"answer": 17}. Let a checker parse the output and compare the integer against the computed result. The reward is 1 only when both the output contract and the answer are correct; otherwise it is 0.

def reward(text, expected):
    try:
        obj = json.loads(text)
    except (ValueError, TypeError):
        return 0.0
    valid = (type(obj) is dict and set(obj) == {"answer"}
             and type(obj["answer"]) is int)
    return float(valid and obj["answer"] == expected)

This is an explanatory fragment requiring import json; the proposed LLM experiment has not been executed. Test the checker independently on correct answers, incorrect answers, booleans, extra keys, malformed JSON and trailing prose. If unique keys are part of the contract, reject duplicate JSON keys explicitly; Python's default decoder keeps the last occurrence. Never evaluate generated expressions with Python's eval.

The rule is intentionally strict. If it produces all-zero groups, measure format accuracy and answer accuracy separately to diagnose why, then consider a brief supervised warm-up. Giving partial reward for formatting changes the objective: a model can learn valid-looking wrong answers. Adding arbitrary bonuses is an experiment that needs its own comparison.

A verifier checks properties; the reward maps those checks into a score. A unit-test suite verifies the cases it covers, not every possible input. A model-based judge estimates quality and can be biased or fooled. Use the term “verifiable” only for the property actually checked. The Unsloth guide is a useful introduction to these roles.

10 · Plan your first Unsloth experiment

After the CPU lab, follow Unsloth's maintained GRPO tutorial in its supported GPU environment. Use its linked notebook as the executable installation recipe. This avoids mixing snippets from incompatible versions. The sequence below is a proposed experiment, not a reported training run.

1. Fix one task and a baseline. Use synthetic arithmetic with the JSON contract above. Generate training and held-out problems separately, deduplicate them and save the splits. Also reserve a harder split, such as larger operands, to test a defined distribution change.
2. Choose a starting model that sometimes succeeds. Measure its format accuracy and answer accuracy before training. If no sampled answer ever passes, first investigate the prompt, checker, token budget and task difficulty. A pretrained or instruction-tuned model provides useful starting capabilities.
3. Verify one batch before training. Inspect prompts, sampled completions, raw rewards and group membership. Confirm that the expected answer is available to the scorer without being accidentally inserted into the model's prompt.
4. Freeze the setup. Record notebook revision, model/tokenizer revisions, package versions, random seed, group size, generation settings, maximum prompt/completion lengths, batch sizes, loss variant and KL coefficient. LoRA trains small adapter matrices; QLoRA also quantizes frozen base weights. These reduce memory use but do not define the reward.
5. Run a small, bounded trial. Save checkpoints, meaning model states you can restore. Track each reward component, held-out correctness, output length, truncated completions, group reward variation and GPU memory. Compare at the same inference settings.
6. Explain the result before increasing the budget. Inspect improvements and regressions. Resume only if the scorer and held-out evidence support a useful next experiment.

Generation is a substantial part of the workload: more answers per prompt and longer completions increase the work and often memory demand. Unsloth's published memory examples depend on model, precision, context, batch and runtime choices. They do not establish what will fit on a particular laptop or GPU. Check the selected notebook's current requirements before provisioning hardware. Tutorial and setup.

How to read the Unsloth guide critically

Its intuitive framing is useful, but several statements need qualification. GRPO does not inherently eliminate the reward model; its group compares completions of one prompt. Dataset row count is not generally optimizer-step count because batching, accumulation and generation scheduling intervene. A training run has no universal minimum step count or guarantee of improving with more time. Compare the guide against the original method and current trainer documentation.

Likewise, producing thinking tags or longer explanations is not evidence that answers became more reliable. Measure correctness and failures on unseen tasks. The R1 paper reports a particular training system and evaluation; it does not promise the same result from a small notebook run.

11 · Know whether it worked

Training reward measures what the optimizer sees. Useful evaluation also needs outcomes on held-out data, repeatability and an explicit baseline. Stable-Baselines3 recommends separate evaluation environments and multiple runs because exploration and random seeds affect results.

For the proposed LLM lab, report baseline and trained-model correctness, valid-format rate and generated-token count on the same problems. If you report pass@k, explain that it concerns success among k attempts; keep k fixed across comparisons. Selecting the best of more samples spends more inference compute.

Keep a small collection of inspected failures beside the aggregate numbers. A reward improvement accompanied by new failures on ordinary prompts may be a poor trade. Repeatedly tuning against a test set turns it into development data; reserve a final set for the final comparison.

12 · A learning path with concrete outputs

This is a suggested sequence, not a course-completion estimate. Move on when you can produce and explain the output.

For a deeper theory sequence, use David Silver's lecture materials: Markov decision processes, dynamic programming, model-free prediction, model-free control, function approximation and policy gradients. The course page was checked; its linked videos and lecture PDFs were not all reviewed for this article.

Choosing a follow-up project

These are proposed follow-up experiments. A small game environment gives you explicit actions and rewards. A specialized language-model task gives you a possible training target when a checker can grade a useful outcome. A tool-using agent adds longer trajectories and delayed credit, making evaluation harder.

A practical starting point is a synthetic, verifiable subtask adjacent to the fine-tuning guide, followed by a comparison with SFT. For an agent built from the LangGraph article, first record reliable trajectories and task outcomes. Orchestration and persistent memory can support data collection, but learning requires a separate training process.

Sources and reading scope

The sources below separate original methods, maintained implementation documentation and teaching material. “Undated” means no reliable page publication date was established; it does not mean the page has never changed. Paper dates identify their original release, with the reviewed revision linked.

Sources checked 2026-09-30: 16 sources, including the course index. Paper text and appendices were read; research figures were not independently inspected and linked videos were not watched. The corridor, its measured results, the four-answer calculator and the study sequence were created for this article. The GPU experiment and suggested applications remain untested.