Resource Allocation and Optimization · Vol. 1 No. 1 (2026)
Original publication: October 31, 2026
Cameron Richardson — Switzerland; Trevor Gardner — Ireland; Benjamin Lewis — New Zealand
Observational comparison of archived learning protocols. Source training-step counts are compared directly; wall time, total actor work, and fresh policy evaluation are unavailable.
Can a longer run over a small level set catch up with a shorter run sampling broadly? Public easy-mode Procgen logs offer a descriptive comparison, although their worker roles differ. Unrestricted-level online returns near twenty-five million training steps exceeded the unrestricted evaluation returns of fifty-million-step, two-hundred-level training in all sixteen game averages. The mean normalized contrast was 0.2531, with a working interval of 0.1322 to 0.3741. The analogous fifty-million-step contrast was 0.3415. These source-protocol differences caution against treating additional repeated exposure as an automatic replacement for breadth; they do not identify the causal benefit or computational price of level diversity.
Training resources are often summarized by a single count of environment steps. That count is important, but it does not say how many distinct situations the learner encounters. Fifty million interactions with a recurring set of layouts need not provide the same experience as twenty-five million interactions drawn broadly from a procedural environment. A resource statement that omits the exposure distribution can therefore conceal a substantial difference in what the learner has had an opportunity to practice.
This study asks how the online return of unrestricted-level training near twenty-five and fifty million recorded training steps compares with unrestricted-level evaluation after fifty million steps of fixed-two-hundred-level training. We use publicly released Procgen logs and treat the comparison as an empirical description of two source protocols. We do not assume the protocols differ only in level diversity, and we do not convert their step counts into an estimate of computational efficiency.
The underlying reinforcement-learning framework in Sutton and Barto, 2018 makes the environment distribution part of the task. Cobbe et al., 2019b supplies sixteen procedurally generated games with source protocols for broad sampling and finite-level generalization. Cobbe et al., 2019a previously examined how limited training environments relate to broader performance. Together, these sources motivate asking whether the amount of repeated interaction can stand in for the breadth of interaction, rather than treating every environment step as informationally interchangeable.
For an interactive rendering system, the issue is especially concrete. Meng et al., 2026 propose an interactive game-rendering paradigm. Such a system may make new environments inexpensive to present, but its rendering function alone does not establish that a learner actually sees a broad distribution. Our analysis concerns the latter exposure question. It contains no rendering experiment, no implementation of the target method, and no claim about its frame quality or response latency.
An unrestricted training protocol can encounter new layouts as interaction proceeds. A finite protocol repeatedly samples from its assigned set and may improve substantially within that set. Additional repetitions could strengthen transferable skills, strengthen configuration-specific behavior, or do both. The balance is an empirical question. A larger nominal step budget alone does not identify which route dominates.
The proposition examined here is that broad exposure at an earlier training budget can be associated with better broad-distribution recorded performance than a longer finite-exposure run. Association is the appropriate wording because source worker roles and return collection differ. We test neither a universal sample-complexity ordering nor a counterfactual statement about what the same fitted policy would have achieved under an alternative history.
Algorithms such as Schrittwieser et al., 2019 integrate learning and planning in ways that change the informational value of interaction. Ye et al., 2021 explicitly addresses limited-data performance, while Hafner et al., 2023 studies broad control competence through learned world models. Their differing designs reinforce why a step count is only one resource measure. None is the learner rerun in this manuscript, and their published achievements are not imported as evidence for our particular source contrast.
Environment breadth also differs from importing prior data or language-guided knowledge. Guss et al., 2019 supplies a benchmark built around human demonstrations, Fan et al., 2022 uses a broader open-ended task ecosystem, and Wang et al., 2023 studies an agent that accumulates reusable skills. These are alternative sources of experience structure, not interchangeable replications of a procedural-layout experiment. They help locate the exposure question without expanding the empirical population beyond the sixteen games analyzed here.
The local source is the archived repository in OpenAI, 2026, fixed to commit 1a2ae2194a61f76a733a39339530401c024c3ad8. We acquired all one hundred forty-four easy-mode CSV files: separate training and evaluation logs for fixed-two-hundred-level runs, and online training logs for unrestricted-level runs. Each family covers sixteen games and three source run labels. The acquisition also preserves the license, explanatory files, source constants, and file-level cryptographic checks.
The fixed-level regime uses a training worker and an evaluation worker. Source code gives the evaluation worker an unrestricted level parameter and zero weight in the PPO gradient update. The broad regime instead trains online with unrestricted level sampling. Thus, both outcomes concern the broad source level distribution operationally, but one comes from a noncontributing evaluation worker and the other from a worker whose experience contributes to learning.
This distinction is a material limitation. Online returns can reflect exploration behavior, contemporaneous policy updates, and the episode history of the training worker. The broad training log is not a separately collected fixed-policy test suite. Conversely, the finite regime's evaluation worker is not a complete list of independently held-out level identifiers: the acquired files do not supply those identities. We describe the source roles exactly rather than recast both series as identical test-set evaluations.
The source curves contain 3,051 updates, with recorded training-step increments of 16,384. Every series used here shares that grid. The twenty-five-million window consists of one hundred updates ending at 25,001,984 steps; its first record is at 23,379,968. The final window spans 48,365,568 through 49,987,584 steps. These fixed windows reduce dependence on one last logged value without pretending to remove all dependence from the rolling episode buffers.
Each game's return is transformed by subtracting the lower reference value in the source easy-game constants and dividing by that game's supplied reference range width. We do not clip the result. The anchors facilitate aggregation across unequal raw reward scales; they are neither estimated extrema nor a claim that a normalized score of one has the same utility for all games.
The measurement distinctions in Stevens, 1946 are directly relevant to that transformation. An affine reward normalization changes units without establishing a common substantive value scale. Carroll et al., 2006 further motivates caution where noisy measured quantities stand in for latent constructs. Here, return is an observed task score, while breadth of useful competence is a broader interpretation that the records cannot measure completely.
Shannon, 1949 offers a general distinction between the quantity of transmitted symbols and information relative to a distribution. We use that distinction conceptually, without calculating entropy or claiming that a novel level contains a fixed number of informative bits. The empirical variables remain source step counts and logged returns. A statement about information gained per step would require additional measurements absent from the archive.
For each regime, game, and source run, we calculate the normalized window mean. We average the three runs within game before comparing regimes. The run labels identify files; they do not prove that identical random seeds generate matched trajectories across protocols. The final analysis therefore uses differences between game-level regime means rather than a claimed paired-seed randomized design.
The first contrast subtracts the finite regime's final unrestricted evaluation mean from the unrestricted regime's twenty-five-million online mean. It asks whether the broad regime was already ahead at roughly half the source training-step budget. The second contrast makes the analogous comparison using the unrestricted regime's final window. Both contrasts retain exactly the same sixteen games and the same final finite-regime reference.
We estimate each equal-game mean through intercept-only OLS. HC3 standard errors are used with t15 intervals, following the heteroskedasticity-robust logic of White, 1980. N is sixteen game rows. No controls, time trends, or fixed effects are included; adding covariates after aggregation would not recover a randomized diversity effect. R-squared is zero because each regression estimates a mean rather than a predictive relationship.
The cluster-inference discussion in Cameron and Miller, 2015 helps explain why many correlated observations do not automatically provide many independent comparisons. Our thousands of log records are reduced before estimation. Newey and West, 1987 addresses serial covariance in time-series settings, but we do not add a time-series correction to a regression whose observations are already game summaries. Temporal dependence remains embedded in each summary and is part of the conditional scope of the working intervals.
Figure 1. Two source protocols, three fixed window summaries, and sixteen game-level contrasts. Different worker roles prevent interpreting the arrows as a causal intervention on level diversity.
At the final finite-regime budget, the equal-game unrestricted evaluation mean was 0.389590. The unrestricted regime's online mean was 0.642715 near twenty-five million training steps and 0.731106 near fifty million. These are normalized source-record means. They are not independent re-evaluations of the original policies, and they do not include uncertainty from a fresh episode sample.
The half-budget contrast averaged 0.253125, with a standard error of 0.056741 and a working interval from 0.132185 to 0.374066. The equal-budget contrast averaged 0.341516, with a standard error of 0.067481 and an interval from 0.197685 to 0.485348. Both contrasts were positive in every one of the sixteen game averages. That consistency is descriptive evidence about the supplied collection, not a guarantee for any newly generated game.
Table 1. Unrestricted-regime online return minus finite-regime unrestricted evaluation return, after game-specific normalization. Sixteen game rows; intercept-only OLS, HC3 covariance, t15 intervals, no controls or fixed effects. The reference is the finite regime's final hundred-update window in both columns.
| Model | Term | Estimate (SE) | 95% CI | N |
|---|---|---|---|---|
| M1 | Mean outcome/contrast | 0.253125 (0.056741) | [0.132185, 0.374066] | 16 |
| M1: fit | R-squared = 0.0000 | - | - | 16 |
| M2 | Mean outcome/contrast | 0.341516 (0.067481) | [0.197685, 0.485348] | 16 |
| M2: fit | R-squared = 0.0000 | - | - | 16 |
Model labels: M1 = Unrestricted regime near 25 million steps minus finite-regime evaluation near 50 million steps; M2 = Unrestricted regime near 50 million steps minus finite-regime evaluation near 50 million steps.
The difference between the two reported mean contrasts is about 0.088391 normalized units, reflecting further progress within the unrestricted regime. We have not fitted a separate interval for that increment and do not infer its uncertainty by comparing the two displayed intervals. More importantly, a favorable half-budget comparison does not show where the finite regime might eventually catch up. The source archive ends near fifty million steps and supplies no observation of longer finite-level training.
Because every supplied game has a positive contrast at both budgets, replacing equal-game weights with any fixed nonnegative game weights that sum to one would preserve the observed direction. This is a statement about the finite table, not an estimated robustness probability. The magnitude would still depend on the weights and normalization anchors, and uncertainty would require an analysis matching that alternative target. It is useful to separate this limited directional stability from broader claims that the result would survive new games, different algorithms, or matched fresh evaluation episodes. Those extensions change the evidence rather than merely reweight it.
The records show that more source training steps on the finite level menu did not erase the observed difference from unrestricted online training. They do not show that broad training required half as much computation. Williams et al., 2009 distinguishes hardware performance constraints from high-level operation counts. Mytkowicz et al., 2009 likewise demonstrates how apparently innocuous measurement conditions can distort computational comparisons. We have no matched hardware session, energy meter, aggregate actor-work count, or comparable wall-time accounting for these protocols.
Source training steps exclude the informational and computational role of evaluation workers as a comparable deployment cost. A broad worker contributes to learning; a zero-gradient worker still runs environments to produce an evaluation stream. Equal logged step budgets therefore do not imply equal total sampling expenditure. The result should be described through its explicit budget label instead of upgraded into a claim of measured computational savings.
Forecast-evaluation guidance in Hyndman and Koehler, 2006 is useful in requiring the score and horizon to be named. Gneiting and Raftery, 2007 similarly stresses alignment between a measure and its decision target. Our two windows answer a budget-indexed descriptive question. They cannot establish which protocol is preferable for an application with a different exposure distribution, reward utility, or cost structure.
Levine et al., 2020 distinguishes learning from fixed data from ongoing interaction. Fu et al., 2020 provides benchmark datasets for the former, and Chen et al., 2021 studies sequence modeling of trajectories. Our learner histories were generated through online source training; reading their logs later does not convert the original learning into offline reinforcement learning. That distinction matters when describing what kind of experience breadth was available.
For generated environments, Yu et al., 2025 and Valevski et al., 2024 concern game-oriented world modeling. Micheli et al., 2022 investigates transformer world models, and Bruce et al., 2024 addresses generative interactive environments. These studies motivate questions about which situations a model can generate or simulate. The present comparison instead records a difference in policy outcomes under two exposure protocols. It cannot tell whether a generative environment preserves the same task distribution or whether its visual diversity creates useful control diversity.
The benchmark is a supplied set rather than a probability sample of games, and three source runs per condition leave considerable training variability unexplored. The study is secondary and exploratory; no preregistered causal experiment is claimed. The working intervals condition on the source curves and on the chosen normalization. They do not encompass original training reproducibility, alternative algorithm settings, or the scientific independence of multiple possible analyses of the archive.
Preserving those conditions follows the documentation approach in Gebru et al., 2021 and the reuse principles in Wilkinson et al., 2016. The analysis files retain every game-level mean, source window, and contrast. A stronger experiment would randomize or systematically vary level counts while matching worker roles, total sampling, architecture, and fresh evaluation episodes. Until then, these results support a bounded observation: additional repetition over the finite menu did not substitute for broad exposure in the captured source records, even when the latter was assessed at an earlier training-step budget.