More Training Steps Do Not Substitute Automatically for Broader Level Exposure: A Procgen Protocol Comparison

Authors

  • Cameron Richardson Author
  • Trevor Gardner Author
  • Benjamin Lewis Author

Abstract

Can a longer run over a small level set catch up with a shorter run sampling broadly? Public easy-mode Procgen logs offer a descriptive comparison, although their worker roles differ. Unrestricted-level online returns near twenty-five million training steps exceeded the unrestricted evaluation returns of fifty-million-step, two-hundred-level training in all sixteen game averages. The mean normalized contrast was 0.2531, with a working interval of 0.1322 to 0.3741. The analogous fifty-million-step contrast was 0.3415. These source-protocol differences caution against treating additional repeated exposure as an automatic replacement for breadth; they do not identify the causal benefit or computational price of level diversity.

Observational comparison of archived learning protocols. Source training-step counts are compared directly; wall time, total actor work, and fresh policy evaluation are unavailable.

Original publication: October 31, 2026.

References

  1. Gneiting, Tilmann; Raftery, Adrian E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association. https://doi.org/10.1198/016214506000001437.
  2. Hyndman, Rob J. and Anne B. Koehler. “Another look at measures of forecast accuracy.” International Journal of Forecasting, vol. 22, no. 4, 2006, pp. 679-688. https://doi.org/10.1016/j.ijforecast.2006.03.001.
  3. Ye, Weirui, et al. "Mastering Atari Games with Limited Data." arXiv preprint arXiv:2111.00210, 2021. https://arxiv.org/abs/2111.00210.
  4. Schrittwieser, Julian, et al. "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model." arXiv preprint arXiv:1911.08265, 2019. https://arxiv.org/abs/1911.08265.
  5. Wang, Guanzhi, et al. "Voyager: An Open-Ended Embodied Agent with Large Language Models." arXiv preprint arXiv:2305.16291, 2023. https://arxiv.org/abs/2305.16291.
  6. Hafner, Danijar, et al. "Mastering Diverse Domains through World Models." arXiv preprint arXiv:2301.04104, 2023. https://arxiv.org/abs/2301.04104.
  7. Shannon, C.E.. (1949). Communication in the Presence of Noise. Proceedings of the IRE. https://doi.org/10.1109/jrproc.1949.232969.
  8. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317-372. https://doi.org/10.3368/jhr.50.2.317.
  9. Cobbe, Karl; Klimov, Oleg; Hesse, Chris; Kim, Taehoon; Schulman, John. (2019a). Quantifying Generalization in Reinforcement Learning. Proceedings of the36th International Conference on Machine Learning, PMLR97:1282-1289. https://proceedings.mlr.press/v97/cobbe19a.html.
  10. Fu, Justin, et al. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning." arXiv preprint arXiv:2004.07219, 2020. https://arxiv.org/abs/2004.07219.
  11. Cobbe, Karl, et al. "Leveraging Procedural Generation to Benchmark Reinforcement Learning" arXiv preprint arXiv:1912.01588, 2019. https://arxiv.org/abs/1912.01588.
  12. Yu, Jiwen, et al. "GameFactory: Creating New Games with Generative Interactive Videos." arXiv preprint arXiv:2501.08325, 2025. https://arxiv.org/abs/2501.08325.
  13. Guss, William H., et al. "MineRL: A Large-Scale Dataset of Minecraft Demonstrations." arXiv preprint arXiv:1907.13440, 2019. https://arxiv.org/abs/1907.13440.
  14. OpenAI. (2026). train-procgen: released easy-mode training and evaluation logs, commit1a2ae2194a61f76a733a39339530401c024c3ad8 [Archived repository; snapshot acquired8November2026]. https://github.com/openai/train-procgen/tree/1a2ae2194a61f76a733a39339530401c024c3ad8.
  15. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817-838. https://doi.org/10.2307/1912934.
  16. Bruce, Jake, et al. "Genie: Generative Interactive Environments" arXiv preprint arXiv:2402.15391, 2024. https://arxiv.org/abs/2402.15391.
  17. Chen, Lili, et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling." arXiv preprint arXiv:2106.01345, 2021. https://arxiv.org/abs/2106.01345.
  18. Meng, Zijie; Che, Jinming; Wei, Bingcai; Cao, Xixin. (2026). Make a game: A novel paradigm for interactive game rendering. ICASSP 2026, 1026-1030. https://doi.org/10.1109/ICASSP55912.2026.11461927.
  19. Stevens, S. S.. (1946). On the Theory of Scales of Measurement. Science. https://doi.org/10.1126/science.103.2684.677.
  20. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
  21. Mytkowicz, Todd; Diwan, Amer; Hauswirth, Matthias; Sweeney, Peter F.. (2009). Producing wrong data without doing anything obviously wrong!. Proceedings of the 14th international conference on Architectural support for programming languages and operating systems. https://doi.org/10.1145/1508244.1508275.
  22. Williams, Samuel; Waterman, Andrew; Patterson, David. (2009). Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM. https://doi.org/10.1145/1498765.1498785.
  23. Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
  24. Valevski, Dani, et al. "Diffusion Models Are Real-Time Game Engines" arXiv preprint arXiv:2408.14837, 2024. https://arxiv.org/abs/2408.14837.
  25. Fan, Linxi, et al. "MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge." arXiv preprint arXiv:2206.08853, 2022. https://arxiv.org/abs/2206.08853.
  26. Micheli, Vincent, et al. "Transformers are Sample-Efficient World Models" arXiv preprint arXiv:2209.00588, 2022. https://arxiv.org/abs/2209.00588.
  27. Carroll, Raymond J., David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu. “Measurement Error in Nonlinear Models.” 2006. https://doi.org/10.1201/9781420010138.
  28. Gebru, Timnit, et al. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  29. Newey, Whitney K., and Kenneth D. West. “A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix.” Econometrica, vol. 55, no. 3, 1987, pp. 703-708. https://doi.org/10.2307/1913610.
  30. Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem; da Silva Santos, Luiz Bonino; Bourne, Philip E.; Bouwman, Jildau; Brookes, Anthony J.; Clark, Tim; Crosas, Mercè; Dillo, Ingrid; Dumon, Olivier; Edmunds, Scott; Evelo, Chris T.; Finkers, Richard; Gonzalez-Beltran, Alejandra; Gray, Alasdair J.G.; Groth, Paul; Goble, Carole; Grethe, Jeffrey S.; Heringa, Jaap; ’t Hoen, Peter A.C; Hooft, Rob; Kuhn, Tobias; Kok, Ruben; Kok, Joost; Lusher, Scott J.; Martone, Maryann E.; Mons, Albert; Packer, Abel L.; Persson, Bengt; Rocca-Serra, Philippe; Roos, Marco; van Schaik, Rene; Sansone, Susanna-Assunta; Schultes, Erik; Sengstag, Thierry; Slater, Ted; Strawn, George; Swertz, Morris A.; Thompson, Mark; van der Lei, Johan; van Mulligen, Erik; Velterop, Jan; Waagmeester, Andra; Wittenburg, Peter; Wolstencroft, Katherine; Zhao, Jun; Mons, Barend. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. https://doi.org/10.1038/sdata.2016.18.

Downloads

Published

2026-10-31

Issue

Vol. 1 No. 1 (2026): Inaugural Issue

Section

Articles

License

Copyright (c) 2026 Cameron Richardson, Trevor Gardner, Benjamin Lewis

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.