Reading a Reward Curve: What Six Months of Training Logs Taught Me

For the first couple of months of training my portfolio agents, I judged almost everything by one question: Is the reward curve going up?

If it was rising, I assumed things were working. If it flattened out, I assumed the agent had stopped learning. If it dropped, something was probably wrong.

It was a reasonable way to start looking at training. It was also, as I eventually learned, far too simplistic.

What a reward curve actually tells you

If you don't work with machine learning, here's the basic idea.

In reinforcement learning, an agent makes decisions and receives a reward based on the outcome of those decisions. In a portfolio management setting, that might mean changing the allocation across assets, holding the existing portfolio, or rebalancing the weights.

As training progresses, we can track the rewards the agent receives and plot them over time. The resulting reward curve gives us a visual sense of how the learning process is progressing.

The textbook picture is fairly intuitive: the curve rises during the early stages of training, then gradually levels off as the agent learns a policy and further improvements become smaller.

• A rising curve looks like learning.
• A stable curve looks like convergence.
• A declining curve looks like a problem.

That's the theory.

Then I started paying attention to what happened beyond the curve.

The curve that looked perfect — and wasn't

About three months into my experiments, I had a training run that looked almost textbook.

The reward climbed quickly at the beginning, the rate of improvement gradually slowed, and eventually the curve settled into a relatively high, stable plateau.

It looked exactly like what you would want to see from a successful training run.

I was pleased with it.

For about a day.

Then I ran the backtest on a period that contained a sharp increase in market volatility.

The strategy struggled.

What had happened?

The agent had learned a policy that worked very well under the conditions it had predominantly encountered during training. Those conditions were relatively calm, low-volatility periods. When the market environment changed, the policy did not generalize nearly as well as the reward curve had led me to believe.

The important point is that the reward curve itself wasn't wrong.

The agent really had improved on the training data. The problem was that I had been interpreting that improvement as evidence that the learned policy would generalize to conditions outside the training experience.

Those are two very different things.

That was my first real lesson:

A high and stable reward curve can tell you that an agent has learned. It cannot, by itself, tell you that it has learned something that generalizes.

The curve that looked broken — and wasn't

A few months later, I made the opposite mistake.

This time, the reward curve looked terrible.

It was noisy and jagged, with no obvious upward trend. Every so often, the reward would drop sharply before recovering. My immediate reaction was the obvious one: something must be wrong.

Before stopping the run, though, I decided to look at what was happening underneath the aggregate reward.

The answer turned out to be quite different.

Those sharp drops were occurring around periods where the agent was being exposed to substantially more volatile market conditions. These regimes had deliberately been included in the training process, and lower rewards during difficult market conditions were not necessarily evidence of a broken training process.

Once I separated the rewards by market regime instead of looking at one blended curve, the picture became much clearer.

What initially looked like instability was actually telling me something about how the agent was responding to different environments.

The curve wasn't broken.

My interpretation of the curve was.

That became the second lesson — and probably the more important one:

A single aggregated reward curve can hide almost as much as it reveals.

If an agent is learning across very different market environments, the same average reward can be produced by very different underlying behaviours.

What I look at now

Those two experiences changed how I read training logs.

I still look at the reward curve. I just don't treat it as the final answer anymore.

Here are some of the things I check now.

• First, I break the reward down by market regime. Instead of looking only at one aggregate training curve, I examine how the agent behaves under different volatility or market conditions. An apparently stable average can hide substantial differences between regimes.
• Second, I look at the composition of the training data. Before training even begins, I want to know what kind of market environments the agent is actually seeing. How much of the training period is relatively calm? How much represents stressed or highly volatile conditions? An agent cannot learn how to respond to environments that it rarely encounters.
• Third, I care much more about genuinely unseen data. A separate test period is more informative than another attractive-looking training curve. In financial applications especially, I prefer a chronological out-of-sample period that the agent has never encountered during training over a randomly shuffled validation split. The real question isn't whether the agent can perform well on the market it has already seen. It's whether the learned strategy still behaves sensibly when the market changes.
• And finally, I stopped assuming that smooth is always good. A smooth reward curve can mean that the agent has converged to a robust policy. It can also mean that the training environment simply wasn't challenging enough.

So now, when I see a smooth curve, my first question isn't "Is this good?"

It's:

"Why is it smooth?"

Why this matters beyond reinforcement learning

This lesson isn't really unique to deep reinforcement learning.

It isn't even unique to machine learning.

Quantitative researchers have been dealing with the same problem for decades. A backtest can produce an impressive performance number while telling you very little about how a strategy will behave when the market moves outside the conditions that produced that number.

The danger isn't necessarily that the metric is wrong.

Sometimes the metric is perfectly correct.

The danger is that the metric is measuring something that isn't the thing you actually care about.

For a portfolio agent, a high training reward is interesting. But what I ultimately care about is whether the strategy can navigate different market environments without falling apart.

That distinction changed how I think about training.

A reward curve is useful. It can tell you whether something is happening during learning, whether the agent appears to be converging, and whether a training run deserves further investigation.

But it isn't a certificate of a good strategy.

In the end, a reward curve is a backtest with extra steps.

And like any backtest, it deserves a healthy amount of skepticism.

That is also why I became interested in studying portfolio performance across different volatility regimes. My research on volatility regime-dependent portfolio optimization takes that idea further by examining how deep reinforcement learning agents behave when market conditions change.

You can read the research here.