GRPO and Evolutionary HPO

Reinforcement Learning (RL) has demonstrated remarkable potential in achieving superhuman performance across diverse domains, from agents conquering Atari games to DeepMind's AlphaGo mastering the ancient game of Go, self-driving vehicles, and robots learning complex physical tasks. Reinforcement learning has proven to be a powerful tool in these situations, allowing agents to develop never-before-seen strategies to approach tasks in novel ways, leading to often super-human performance.

With RL, agents are not limited to mimicking human behaviour through supervised learning, but are instead able to explore and generate their own data, removing the ceiling of human performance. This has again been observed recently as RL was used to finetune DeepSeek's R1 large language model, resulting in the emergence of powerful reasoning behaviour and subsequently superior performance.

At AgileRL, we believe that reinforcement learning will not only play a key role in the path towards artificial general intelligence but also that it has the potential to provide immense practical value in solving real-world business challenges. The ability of RL agents to learn optimal behaviours through trial-and-error interaction with their environment, combined with their capacity to generalize and adapt, makes them uniquely suited for tackling complex decision-making problems across industries. Additionally, RL in large language model finetuning will allow businesses to finetune small and relatively basic LLMs into powerful, specialized agents, which will be cheaper than the general models provided by big players in the market.

In this article, we delve into the implementation of GRPO in the AgileRL framework, which combines powerful evolutionary hyperparameter optimization with the DeepSeek algorithm to produce an expert reasoning agent, with models as small as 3B parameters. To demonstrate this capability, we utilize a Countdown-inspired dataset that emulates the mathematical challenge from the popular television show. In this task, participants are presented with 3-4 numbers and must craft an equation using basic arithmetic operations to reach a target number.

GRPO - Group Relative Policy Optimization

GRPO (Group Relative Policy Optimization) is an elegant simplification of PPO (Proximal Policy Optimization) that makes reinforcement learning more computationally efficient, especially for large language models.

The two key innovations are:
‍
- Eliminating the critic network: Instead of training a separate value function to estimate expected rewards (which requires additional compute and memory), GRPO normalizes rewards across a batch of samples. It calculates advantage by subtracting the mean reward from each sample's reward and dividing by the standard deviation.
- Group-based evaluation: GRPO generates multiple outputs using the same policy, evaluates them as a group, and then updates the model. This approach reduces variance in the training signal by smoothing out the randomness inherent in probabilistic environments.

These changes are particularly valuable for LLM training because they reduce computational overhead by removing the need for a separate critic model, provide more stable gradient updates in environments with sparse or noisy rewards, and simplify implementation while maintaining or improving performance.

Evolutionary Hyperparameter Optimization
‍
Traditionally, hyperparameter optimization (HPO) for reinforcement learning (RL) is particularly difficult when compared to other types of machine learning. This is for several reasons, including the relative sample inefficiency of RL and its sensitivity to hyperparameters.

Evolutionary algorithms have been shown to allow faster, automatic convergence to optimal hyperparameters than other HPO methods. At regular intervals, after learning from experiences, a population of agents can be evaluated. Through tournament selection, the best agents are selected to survive until the next generation, and their offspring are mutated to further explore the hyperparameter space. Eventually, the optimal hyperparameters for learning in a given environment can be reached in significantly fewer steps than are required using other HPO methods.

Tournament selection

Tournament selection is used to select the agents from a population that will make up the next generation of agents. The best agent from a population is automatically preserved and becomes a member of the next generation. Then, for each tournament, k individuals are randomly chosen, and the agent with the best evaluation fitness is preserved. This is repeated until the population for the next generation is full.

Mutations

Mutation is periodically used to explore the hyperparameter space, allowing different hyperparameter combinations to be trialled during training. If certain hyperparameters prove relatively beneficial to training, then that agent is more likely to be preserved in the next generation, and so those characteristics are more likely to remain in the population.

In this article, we train a population of 4 3B Qwen2.5 models using GRPO with evolutionary hyperparameter optimization (HPO) and compare the performance with a single 3B Qwen2.5 model without HPO.

Code

In this section, we will outline how we used the AgileRL framework to train a population of LLMs using GRPO and evo-HPO, taking you through each step of the process.

Defining Hyperparameters

Before we commence training, it's easiest to define all of our hyperparameters in one dictionary. Below are the initial hyperparameters and mutation parameters that we used for this experiment. Additionally, we also define a mutations parameters dictionary, in which we determine what mutations we want to happen, to what extent we want these mutations to occur, and what RL hyperparameters we want to tune. Additionally, we also define our upper and lower limits for these hyperparameters to define search spaces. It is worth noting that, unlike the rest of the AgileRL framework, we can only tune the RL hyperparameters and not the architecture hyperparameters.

Defining our base model and dataset

In this tutorial, we use the open-source transformers and datasets libraries from
Hugging Face to download our pretrained model weights and training data. There are a huge number of models and datasets hosted on Hugging Face, and different ones can easily be substituted in.

In this tutorial, to keep things simple, we will use a 3 billion parameter Qwen model, and the Countdown dataset, and initialise them as follows:

Create the Reasoning Environment

From model to agent: In reinforcement learning, models are called agents. This is because they are trained by taking actions, receiving rewards, and learning from this feedback. This enables them to become very good at taking actions to solve tasks - to develop agency. Since we are training our model with reinforcement learning, it becomes an agent through this process.

We must create a reinforcement learning environment in which our agent can explore possible solutions and learn to optimise rewards. AgileRL provides a HuggingFaceGym class that wraps a Hugging Face dataset and converts it into a reinforcement learning, gymnasium-style, environment.

So, how does the environment know how to reward an agent for its outputs? We must define a reward function that the agent learns to optimise. Following the techniques used in the DeepSeek reasoning paper, we will define our reward function as the sum of two rewards:
‍
- Accuracy rewards: Verifying answers against ground truth. In this tutorial, we will reward the model +1 if the final answer it produces is correct, otherwise 0.
- Format rewards: Encouraging structured reasoning with explicit steps. In this tutorial, we will reward the model +1 if it puts its thinking process between '<think>' and '</think>' tags, otherwise 0.

Therefore, the maximum score an agent can receive is 2, by producing the correct answer in the correct format. The key here is that we never tell the agent which answer it should produce or which format it should use. By giving it rewards for displaying these behaviours, the agent itself discovers the best way to achieve high rewards and learn the behaviour we desire.

Now that we have defined our reward functions, we must also design our prompt. This forms the input given to the agent and provides the context necessary to complete the task. This is a task-specific feature, and different reasoning problems will require different chat templates, although they can follow a similar
format. We must also define a function to collate our questions and answers, and standardise their length. Combining all these components, we can now initialise the HuggingFaceGym object.

Create a population of GRPO Agents

To allow our model to become an agent and learn through reinforcement learning, we can use the GRPO class. This class follows the same structure as the other reinforcement learning algorithms in the AgileRL library.

We also define the RL hyperparameters that we want to tune in this training run. For this experiment we selected three: beta (the KL penalty coefficient), learning rate, and group size.

Creating Tournament Selection and Mutations objects

As discussed in the Tournament Selection section, tournament selection is used to select the agents from a population that will make up the next generation of agents. If elitism is used, the best agent from a population is automatically preserved and becomes a member of the next generation.

The class TournamentSelection defines the functions required for tournament selection. TournamentSelection.select() returns the best agent and the new generation of agents.

The Mutations class is used to mutate agents with pre-set probabilities. The available mutations for GRPO currently implemented are:
‍
- No mutation
- RL algorithm mutation - mutation of learning hyperparameters, such as learning rate or batch size.

Mutations.mutation() returns a mutated population. Tournament selection and mutation should be applied sequentially to fully evolve a population between evaluation and learning cycles.

Training and Saving an Agent

The simplest way to train an AgileRL agent is to use the finetune_llm function.

Configuring Accelerate and DeepSpeed

An important part of training an LLM to display reasoning behavaiour is distributed training. They are called Large Language Models for a reason, and are often too large to train on a single GPU. If you want to train a larger, more powerful model, then this becomes even more infeasible.

Instead, we can leverage distributed training, to share the workload across multiple devices and speed up training. To enable distributed training in this tutorial, we use deepspeed and accelerate.

To generate an accelerate file, run the command accelerate config in your terminal, following the instructions on screen to outline the details of the compute you intend to use for your finetuning, saying yes to the question "Do you want to use DeepSpeed?" and no to the question "Do you want to specify a json file to a DeepSpeed config?" if you want an auto-generated DeepSpeed config file. More information on the deepspeed configuration can be found in their docs. The accelerate config will handle the details of the distribution, and the GRPO class handles how the accelerator is used during training. You can then launch a training
run using accelerate with the following command:

Alternatively, you can avoid accelerate config by defining your own accelerate-deepspeed config file and pass it as an argument to accelerate launch:

Example config file:

Results

For our benchmarks, we did two training runs; one with a population of 4 GRPO agents with hyperparameter optimization (HPO) and a single GRPO agent without HPO.
‍
To compare the performance of these two runs, we used three metrics: reward, accuracy, and completion length. The reward indicates the raw reward obtained from a given response, the accuracy is the percentage of responses that resulted in the maximum reward, and the completion length is the number of tokens used in a given response.

The benchmarks below show that the population-based training with evo-HPO resulted in significantly larger rewards and accuracies compared to the single-agent training run without it. Furthermore, the population-based evo-HPO approach also led to far greater reasoning capabilities, with completion lengths at the end of training converging to 800 tokens compared to 200 tokens for the single-agent case.

Exploration vs exploitation

The HPO graphs indicate that higher learning rates were favoured in these early stages, allowing the agents to update parameters more aggressively, resulting in more effective exploration of the solution space and therefore higher scores and accuracies. It is also worth noting that towards the end of the training run, smaller learning rates were favoured as exploitation was prioritized over exploration.

Evo-HPO as a regularization method

The dramatic difference in completion length (200 tokens vs 800 tokens) presents an interesting empirical observation that evo-HPO acts as a regularizer, preventing overfitting on early reward signals, which, in the single agent case, led to convergence to a local optimum at the expense of reasoning depth. The population of agents were able to effectively explore the hyperparameter space and avoid falling into the same local minima that the single agent did. This allowed agents to reason through problems completely. The population diversity that emerged from evo-HPO prevented both early convergence to sub-optimal solutions and the maximization of reward metrics without satisfying the underlying intent (reward hacking).

Below we can see a comparison in the solutions to the same question of the elite agent from the population vs the single-agent at the end of training. The elite agent demonstrates a systematic approach, showing its step-by-step reasoning process and verifying each attempt, while the single agent provides a minimal explanation before jumping to the answer. This difference in reasoning depth and thoroughness highlights how the population-based approach with evo-HPO encourages more comprehensive problem-solving strategies.

Conclusion

Our experiments with GRPO and evolutionary hyperparameter optimization have demonstrated several key findings in the context of LLM finetuning for reasoning tasks:

1. Superior Performance: The population-based approach with HPO consistently outperformed single-agent training, achieving higher rewards and accuracy. This was particularly evident in the early stages of training, where the dynamic adaptation of learning rates enabled more effective exploration of the solution space.
2. Enhanced Reasoning Depth: The population-based approach maintained significantly deeper reasoning processes (800 tokens) compared to the single-agent approach (200 tokens), demonstrating that evo-HPO acts as an effective regularization method. This prevented premature convergence to sub-optimal solutions and encouraged more thorough problem-solving approaches.
3. Adaptive Learning Dynamics: The evolutionary approach showed remarkable adaptability in balancing exploration and exploitation. Higher learning rates early in training facilitated better exploration, while gradually decreasing rates later enabled more focused exploitation of successful strategies.
4. Efficient Model Scaling: The combination of GRPO with evo-HPO demonstrates that smaller language models can achieve sophisticated reasoning capabilities when properly optimized. This suggests a promising path toward creating efficient, specialized language models that can be trained and deployed with fewer computational resources.

To obtain these results, we used an x4 A100 GPU instance. This experiment, is a prototype for what we intend to bring to Arena, where we can leverage our reinforcement learning engine, built on top of our open-source framework, to further distribute and speed up the training process. These results validate our approach of using evolutionary hyperparameter optimization to enhance the performance of smaller language models through reinforcement learning.

What's next for AgileRL LLM finetuning?

This experiment has shown that hyperparameters significantly impact the training process when it comes to LLM finetuning with RL, and we believe that HPO could play a role in superior LLM performance. Over the coming weeks and months, we look to introduce LLM finetuning to Arena - our platform for all things RL. Our RL engine will make the process more distributed, allowing us to significantly reduce the time and cost to tune hyperparameters for LLM-based agents. We also look to enable creation of RL datasets from business data, comprehensive LLM finetuning experiment tracking and monitoring for team collaboration, deployment with built-in guardrails and monitoring, and the ability to retrain models on real-world deployment data. These features will streamline the entire LLM finetuning workflow from data preparation through to production deployment, allowing users to create expert LLMs using much smaller models than some of the existing solutions offered.

Combining GRPO with evolutionary hyperparameter optimization to squeeze the most out of small models