Nick Ustaran-Anderegg
&
Michael Pratt
June 30, 2026
If you are training an LLM to hold multi-turn conversations, use tools, or interact with environments, you need to make two choices that are often mashed together into one: which RL algorithm, or optimiser, computes your policy updates, and how you score the trajectories it learns from. This post walks through both choices, the trade-offs specific to multi-turn settings, and how we handle them at AgileRL.
PPO is an actor-critic method. You run two models: the policy (your LLM, producing token probabilities) and a critic (a separate model estimating expected future reward from any given state). The critic provides per-token advantage estimates via GAE, and the policy gets updated with a clipped surrogate objective that keeps updates from being too large.
For multi-turn, the critic is what matters. It learns which intermediate states lead to good outcomes. A search turn that found the right document gets a different advantage than one that did not, even when the whole trajectory scores the same at the end. This is fine-grained credit assignment, and it's the reason PPO handles long conversations better than the alternatives. The cost is roughly 2x memory (the critic is typically policy-sized), and more hyperparameters to get right - incorrect hyperparameters can lead to value function collapse or policy oscillation.
A recent variant called Turn-PPO (Li et al., Dec 2025) reformulates the MDP at the turn level instead of the token level. Each complete agent response between environment interactions is one "action." The critic estimates value per-turn rather than per-token, which avoids the extremely long horizons of token-level PPO, and in benchmarking environments it outperformed both token-PPO and GRPO.
GRPO drops the critic entirely. For each prompt, it samples multiple completions (a 'group', typically 4 to 64), normalises their rewards within the group, and uses that as the advantage signal. Every token in a given completion gets the same advantage. The policy update still uses a clipped objective with a KL penalty against a reference policy.
Compared to PPO, GRPO has the advantage of requiring half the memory, has a simpler implementation, and involves fewer hyperparameters which need optimising. However, you need multiple rollouts per prompt, which is expensive when each rollout involves several rounds of environment interaction. Additionally, because advantage is uniform across a trajectory, you cannot tell which turns actually mattered; if all your sampled trajectories for a prompt score similarly, the learning signal disappears.
MT-GRPO (Wei et al., May 2025) partially addresses the uniform-advantage problem by adding per-turn rewards and per-turn advantage computation. It helps, but the critic-free approach still lacks PPO's ability to learn a value function over intermediate states.
REINFORCE is the vanilla policy gradient baseline. It does not use a critic or group normalisation. For some tasks it is a reasonable starting point, though it tends to have higher variance than either of the other two options.
CISPO (Clipped Importance Sampling Policy Optimisation) like GRPO, drops the critic and makes use of the group-relative advantage however the mechanism behind the weight update is a little different. CISPO clips the importance sampling weight rather than the token update, so every token contributes to the gradient instead of being switched off when it leaves the trust region. This matters most for rare, low-probability "reflective" tokens (however, wait, recheck) that often fork a reasoning path. Under PPO/GRPO, their ratios overshoot the clip and they sit frozen for the rest of the updates in the batch despite being the tokens that stabilise entropy and drive long CoT behaviour. The same property has a second consequence. Because the clipped weight still performs the importance-sampling correction, rather than discarding the term outright as PPO's update-clip effectively does, it remains well-behaved as the policy drifts from the distribution that generated the batch. This buys tolerance to off-policy data, letting the learner take many updates per batch and consume rollouts that are several steps stale without the gradient variance blowing up. The clip on the weight bounds that variance; the survival of the gradient preserves the signal.
So, which algorithm is best for your problem? PPO handles long interactions (5+ turns), high-variance environments, and situations where you need to figure out which turn in an episode actually caused success or failure. Group-based methods present a faster path when you are getting started, when interactions are short, or when memory is tight. REINFORCE works as a baseline or for simple tasks where the extra machinery of PPO or group-based methods are not justified. CISPO allows for some level off-policy-ness, making it an ideal candidate for async-RL allowing the separation of rollout and training.
Colocated and asynchronous training push on the same two levers: policy freshness and hardware utilisation. In a colocated setup, rollout generation and gradient updates run on the same GPU(s) in alternating phases. Every batch is generated by the current policy, so the data stays on-policy and the update path stays simple: one active set of weights, no sync protocol, and no replay lag to debug.
The downside is idle compute. During weight updates, generation sits still; during rollout generation, trainers wait. That hurts most in long chain-of-thought or multi-turn workloads, where rollout time dominates each step and expensive training hardware spends too much time stalled.
Async systems split those phases into separate worker pools that run continuously. Rollout workers keep generating while trainers keep updating, with weights pushed over periodically instead of every step. You can scale each side independently, so if generation is the bottleneck, you add rollout capacity without touching trainer capacity.
The tradeoff for this additional throughput is stale data and system complexity. The learner now updates on trajectories produced by older policy versions, so policy lag becomes an issue to contend with. Async training also requires reliable weight sync, bounded replay windows, and safeguards for lag spikes.
This is why architecture and objective function cannot be chosen separately. Async only works when the algorithm can tolerate controlled off-policy drift. CISPO is useful here because clipped importance weighting keeps stale samples usable instead of dropping all drifted tokens and wasting the extra rollout throughput.
For most teams, colocated is the better default for debugging, smaller runs, and clean credit assignment. Async becomes worth it once generation is clearly the bottleneck and the optimiser can spend that off-policy budget without destabilising training.
To train an agent, our chosen RL algorithm needs a reward signal. The most appropriate method depends on the nature of our task, and the data and resources available during training.
RULER (Relative Universal LLM-Elicited Rewards) uses an LLM-as-judge to rank multiple trajectories against each other. You generate N trajectories for the same prompt, present all N to a judge model, and the judge scores each one from 0 to 1 based on relative quality. This method presumes that relative ranking is easier than absolute scoring. Asking "which of these 4 conversations is best?" can give you a more reliable signal than "rate this conversation 0 to 1".
RULER does not need labelled data, but requires an extra inference call to a (typically expensive, proprietary LLM API) per group, which is costly in time and money. It naturally fits GRPO since GRPO already generates groups, but it works with any optimiser.
Hand-crafted reward functions are programmatic scoring functions written to evaluate trajectories using domain-specific logic. They can combine rule-based checks (regex for PII detection, format validation), heuristic scoring (turn count penalties, tool-use efficiency), and even LLM-as-judge components for qualitative dimensions like tone. A customer service reward function might give +1.0 for resolving the issue, -1.0 for leaking PII, +0.3 weighted by tone quality, and -0.02 per turn beyond five. This approach enables full control, but it takes domain expertise and iteration to get right.
Trained reward models are the classic RLHF approach. Train a separate model on human preference pairs, then use its forward pass as the reward signal at inference time. This is relatively fast at inference (no judge LLM call), but requires preference data collection, and the reward model can drift from true preferences over time (reward model overoptimisation).
Any RL algorithm can be combined with any reward signal, and the combinations have different strengths.
CISPO + RULER can be a fast path to a working system. Generate trajectories, rank them, normalise, update. This combination is good for prototyping, short multi-turn tasks (2 to 5 turns), and situations where you do not have labelled data. This has applications in customer service, creative tasks, open-ended agents. Its limitation is no per-turn credit assignment, meaning that learned trajectories are usually not optimal, even if the final outcome is desirable.
PPO + RULER is a combination we find interesting for multi-turn qualitative tasks. RULER solves the "what does good look like?" problem through relative comparison. PPO's critic then solves the "which turn was responsible?" problem by learning to decompose the trajectory-level RULER signal into per-turn value estimates. You lose PPO's usual sample efficiency advantage (you need N rollouts per prompt because RULER needs a group to rank, so compute looks more like GRPO), but you gain PPO's credit assignment advantage. This matters most for longer trajectories with 5+ turns.
PPO + hand-crafted rewards gives you the most control. One trajectory per prompt, dense per-turn rewards if you can design them, the critic learns value estimates. This approach is best for long-horizon tasks with clear intermediate objectives: search agents, tool-use agents, dialogue systems.
GRPO (or better: CISPO) + hand-crafted rewards rewards is the DeepSeek R1 pattern. Generate many trajectories, score each with your function, normalise and update. This method works well when outcomes are verifiable (code passes tests, mathematics answer is correct), and episodes are short.
Orthogonal to both the algorithm and the reward signal source, you choose reward granularity.
Sparse means one scalar at the end of the trajectory. This can be a calculated episode score, a binary outcome (issue resolved or not), or a RULER score. Sparse rewards are easier to define, but harder to learn from because they introduce uncertainty over which turns contributed to a success or failure. PPO's critic partially compensates for this by learning intermediate values. GRPO and CISPO have no mechanism for compensation.
Dense means a reward at each turn. Did this turn acknowledge the customer's concern? Did it make progress toward resolution? Was the tone appropriate? Dense rewards provide better credit assignment, but are harder to design.
For verifiable final outcomes (mathematics, code) with short trajectories, sparse is fine. The answer is right or wrong. For long interactions, such as in customer service, you want both: dense per-turn rewards plus a sparse outcome bonus. The outcome reward should be weighed heavily so the agent does not game the intermediate metrics (reward hacking) at the expense of actually solving the problem.
Wei et al. (2025) showed that this combined approach with MT-PPO achieved the best results: faster convergence, higher final accuracy, and near-perfect format correctness compared to trajectory-level-only baselines.
All of this assumes you have a reliable and performant RL implementation, and can find good hyperparameters. In practice, many multi-turn RL projects get stuck at this stage. The standard RL workflow is: pick hyperparameters, train for N steps, evaluate, go back to step 1 with different hyperparameters, repeat this 50 to 500 times. When training LLMs, engineers typically use less ablations (if any) because of the prohibitively high costs involved. As a result, performance is always left on the table, and oftentimes agents do not even show any learning because of an incorrect combination of hyperparameters.
PPO is especially sensitive to hyperparameters. The critic learning rate relative to the policy learning rate, GAE lambda, and value loss coefficient all interact. GRPO's group size and KL coefficient trade off against each other. Multi-turn tasks amplify all of this, because longer horizons and noisier environments make training more fragile.
At AgileRL, we replace manual hyperparameter search with evolutionary hyperparameter optimisation (HPO). A population of agents with different hyperparameter configurations train simultaneously, sharing experiences but learning independently. The population naturally shifts towards what works, and the optimal hyperparameters can change over the course of training (higher learning rates and more exploration early on, more stability later) without requiring manual schedules.
Benchmarks show 10x faster convergence compared to standard approaches, as well as higher rewards. We have been able to match the finetuned performance of 8B parameter models with 2B parameter models, just by using our evolutionary HPO and RL framework.
AgileRL implements PPO, GRPO, GSPO, CISPO, REINFORCE, DPO, and SFT for LLMs under one framework. A common pattern is to use SFT first (fine-tune on demonstration data to get the model into the right distribution), then RL optimisation (PPO, GRPO, CISPO, or REINFORCE against your reward signal) to push beyond what the demonstrations could teach.
Both the RL algorithm and the training stage are configuration choices. You can use SFT on customer service transcripts, optimise that adapter with GRPO for fast iteration, then move to PPO or CISPO when multi-turn credit assignment and async stability become the bottleneck, all with the same infrastructure and checkpoints.
Arena, our platform, handles the distributed training infrastructure: parallelised rollout generation, async replay buffering, checkpoint management, and hyperparameter evolution across GPU clusters. For teams building production multi-turn agents, the path looks like this: define your reward signal, pick a starting algorithm, choose sync or async rollout mode based on workload variance, let evolutionary HPO find the optimal hyperparameters, iterate as you identify failure modes or gather more data, and scale training on Arena as you move to production. Arena's Pipelines feature lets you coordinate these loops so agents can continue training or deploy from checkpoints of previous runs.
AgileRL gives you algorithm flexibility and automated hyperparameter tuning together. You do not have to bet on an algorithm or a hyperparameter configuration upfront, and typically the best performance is found by using these techniques sequentially, always building on the learning of the previous stage.
Multi-turn LLM training research is moving fast. The papers cited in this post are mostly from the last 12 months, and best practices are still being established. Agents fine-tuned with RL can achieve incredible, superhuman performance at a large variety of tasks, but rely on adequate algorithm selection, reward design, hyperparameter tuning, and knowing how to diagnose things when training goes sideways. We work with customers on all of that: reward function design, algorithm selection, training diagnostics, environment design, and live onboarding sessions tailored to specific use cases.
If you are building multi-turn agents and want to talk through what the right setup looks like for your problem, get in touch.
Check out Arena, and get started training agents for free.
