Understanding Test Plan Results

The “Test Plan” section of Income Lab offers a way to explore the ways a plan might progress through a huge number of possible futures. However, by providing such a broad reach, it can be somewhat difficult to interpret these large-scale results. Those looking for easier-to-follow single-example versions of what Test Plan does should use the Retirement Stress Test instead.

This article covers how to interpret the results you find in Test Plan. There are two main sections of Test Plan results:

1. Income: Examples of and statistics on how income could develop over the life of the plan, including the effects of adjustments.
2. Legacy: Examples of and statistics on how the investment portfolio balance could develop over the life of the plan, including the effects of investment returns and changes in withdrawals.

The Income Section

The Income section of Test Plan includes three different views:

1. Lifetime Experience: Overall “30,000 foot” summary of how the simulated plan experiences play out.
2. Income Adjustments: An understanding of the size and frequency of income adjustments found in the simulations.
3. Example Scenarios: Seven representative income scenarios across a range of experiences (good to bad) and some statistics on these examples.

In the “Income Experience” section, we find an easy-to-use overall Test Plan statistic: how often did the simulated plan experiences meet or exceed the originally planned income (Scenarios Above Plan), and how often did these experiences fail to deliver what was originally planned for (Scenarios Below Plan)? Here above and below plan are calculated using the total mortality-adjusted income for the plan or the specific scenario. That means higher income is better than lower income, and higher income earlier in the plan is better and higher income later in the plan (when the clients may no longer be alive).

This section also gives some information on the average and extremes of those simulations that are above or below plan. The extremes are just that - extreme - and so they are not especially useful as plausible outcomes. However, the extreme downside scenario can be helpful for understanding how bad things could get. However, the average (median) is certainly helpful. 

The Income Adjustments section shows how often, on average, the plan saw an increase or decrease in nominal income across all simulations. These include inflation and deflation adjustments, as well as adjustments for Income Path (e.g., the retirement smile). The reason all adjustments are included here is that this view is meant to provide clients with a feel for how often an advisor might recommend a change of any type.

The one type of adjustment not included in this summary is a downward adjustment from a place above the originally planned income level to another place above the originally planned income level. The reason behind excluding these downward adjustments is that, though they are indeed decreases in income, they are adjustments within a very positive experience. Including them could paint the downward adjustment statistic in an inaccurately negative light.

Finally, the Income Example Scenarios show seven examples of income outcomes. These are ranked best to worst. This ranking is based on an overall income experience measure that prioritizes higher income, income early in the plan, and avoidance of major reductions in income. The reason we must use this sort of overall measure for ranking is that even a very good experience might have some months where its income is below an otherwise much worse experience. So, simulation scenarios can't be ranked just based on whether they are always above or below another scenario. Scenarios whose income never cross may not even exist!

For example, in the example above, the 90th percentile scenario spends some time below the 75th percentile scenario. And both the 75th and 90th percentile scenarios spend some time with their income above the “Best” scenarios.

The summary table at the bottom of this view is only available in the “real” (i.e., constant purchasing power or “today's dollars”) view setting. That's because in the nominal view includes the effects of inflation. Since the Test Plan runs the plan through a variety of scenarios with varying inflation, including periods of particularly high inflation, some of the scenarios will include high inflation. And higher inflation periods will naturally have higher income (all else equal) due to inflation adjustments. But that could be misleading when viewed in this table view. For example, higher inflation periods would have higher average, highest, and even lowest incomes. But that doesn't necessarily mean they had great real income or great purchasing power. 

Even for the “real” view, the “Average Income” in this table is most useful. The highest and lowest income rows are interesting, but because they represent only a single month in the scenario, they could be higher or lower than their peers in ways we might not initially expect. For example, in the table above, we see that the lowest income for the Median scenario is lower than the lowest income for the 25th percentile scenario. But when viewing these scenarios in the income graph above, we see that the 25th percentile scenario's income is lower than the Median scenario through almost all of the plan, except for in the final few years. The intuition is that anyone, if faced with the decision between the Median scenario and the 25th percentile scenario, would clearly choose the Median, despite those final few years being better in the otherwise-worse 25th percentile scenario.

A final thing to note about the Example Scenarios for Income is that the scenarios shown in the “real” setting are not the same as those shown in the “nominal” setting. Instead, the 7 representative scenarios (best, 90th percentile, 75th percentile, median, 25th percentile, 10th percentile, and worst) are independently selected from the real and nominal income scenarios. So, the “best” scenario for nominal will be generally quite high, as will the best scenario for the real scenarios, but these are not the same scenario shown in two ways - they are two potentially very different scenarios, with two very different start dates in history (if you are using the historical analysis type) and two very different sequences of returns and inflation. If we instead showed the same seven scenarios, in nominal and real versions, the results could be very confusing. In fact, if the worst scenario involved high inflation and the worst scenario had low inflation, their nominal counterparts might show the best below the worst! Because of this complication, the “real” version of the Examples Scenarios for Income is much easier to understand.

The Legacy Section

The Legacy section of the Test Plan is very simple. It shows seven example scenarios of how the portfolio would have developed when following this plan, including the effects of different sequences of returns and different amounts of portfolio withdrawals (which depend on that scenario's income, including income adjustments). These scenarios are ranked by final portfolio balance. As with the Income Example Scenarios, the legacy scenarios are picked separately for the real and nominal settings - they are not real and nominal versions of the same scenarios or values; they are different scenarios.

One thing to note about the legacy scenarios is that, if the plan being tested has guardrails, these portfolio balances do not go off in all directions, like a wide cone. Instead, if a balance is heading for the heights, it will come back down (because in that situation, the upper guardrail was hit and income went up). If a balance was heading down, it eventually corrects and stabilizes or comes back up (because the lower guardrail was hit and income was cut). So, the portfolio balance experiences are tighter. This range of examples is much more realistic than what you might see in legacy planning software that shows an unrealistically wide range of final outcomes from a Monte Carlo analysis.

The Legacy section also contains a table showing, for each example scenario, the final balance, the first point during the plan when the balance was below the legacy goal (or N/A if the balance never went below the legacy goal), and the amount of time the balance spent below the legacy goal during the plan.