Counterfactual Traffic Interventions for Generative Driving Models

Authors

  • Lucas Green Author
  • James Harrison Author
  • Adrian Hall Author

Abstract

Changing one traffic element in a generated scene can alter many others. A useful intervention test must therefore verify both the intended change and the properties meant to remain stable. This article develops a counterfactual evaluation design for driving-world generation, motivated by OmniDrive. It distinguishes a controlled request to a model from a causal intervention in the real world. No traffic scenes are generated here. The proposed protocol records scene identity, intervention compliance, spillovers, and assessment uncertainty before interpreting a paired output difference.

References

  1. Meng, Zijie, et al. "OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation." arXiv preprint arXiv:2606.17536 (2026).
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Geiger, Andreas, Philip Lenz, and Raquel Urtasun. "Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite." 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354-3361. https://doi.org/10.1109/CVPR.2012.6248074.
  15. Caesar, Holger, et al.. "nuScenes: A multimodal dataset for autonomous driving" arXiv preprint arXiv:1903.11027, 2019. https://arxiv.org/abs/1903.11027.
  16. Yu, Fisher, et al.. "BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning" arXiv preprint arXiv:1805.04687, 2018. https://arxiv.org/abs/1805.04687.
  17. Sun, Pei, et al.. "Scalability in Perception for Autonomous Driving: Waymo Open Dataset" arXiv preprint arXiv:1912.04838, 2019. https://arxiv.org/abs/1912.04838.
  18. Luiten, Jonathon, et al.. "HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking" arXiv preprint arXiv:2009.07736, 2020. https://arxiv.org/abs/2009.07736.
  19. Zhang, Yifu, et al.. "ByteTrack: Multi-Object Tracking by Associating Every Detection Box" arXiv preprint arXiv:2110.06864, 2021. https://arxiv.org/abs/2110.06864.
  20. Weng, Xinshuo, et al.. "AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics" arXiv preprint arXiv:2008.08063, 2020. https://arxiv.org/abs/2008.08063.
  21. Unterthiner, Thomas, et al.. "Towards Accurate Generative Models of Video: A New Metric & Challenges" arXiv preprint arXiv:1812.01717, 2018. https://arxiv.org/abs/1812.01717.
  22. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  23. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  24. Hafner, Danijar, et al.. "Dream to Control: Learning Behaviors by Latent Imagination" arXiv preprint arXiv:1912.01603, 2019. https://arxiv.org/abs/1912.01603.
  25. Ha, David and Jürgen Schmidhuber. "World Models" arXiv preprint arXiv:1803.10122, 2018. https://arxiv.org/abs/1803.10122.
  26. Ye, Hu, et al.. "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" arXiv preprint arXiv:2308.06721, 2023. https://arxiv.org/abs/2308.06721.
  27. Huang, Ziqi, et al.. "VBench: Comprehensive Benchmark Suite for Video Generative Models" arXiv preprint arXiv:2311.17982, 2023. https://arxiv.org/abs/2311.17982.
  28. Radford, Alec, et al.. "Learning Transferable Visual Models From Natural Language Supervision" arXiv preprint arXiv:2103.00020, 2021. https://arxiv.org/abs/2103.00020.
  29. Gebru, Timnit, et al.. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  30. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.

Downloads

Published

2026-09-28

Issue

Vol. 2 (2026): Archive issue of 2026-09-28

Section

Articles

License

Copyright (c) 2026 Lucas Green, James Harrison, Adrian Hall

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.