Value Alignment before Value-Aware Model Error Is Weighted

Authors

  • Owen Ward Author
  • Daniel Rhodes Author
  • Eric Coleman Author

Abstract

Heterogeneous offline values may be incompatible across dynamics, while value-aware model learning emphasizes errors consequential to a policy. Evaluation should distinguish resolving value incompatibility from improving coverage or model weighting. Independent interactive outcomes and behavior-support diagnostics are needed. Average prediction accuracy on combined stored records cannot establish the value of the resulting policy. This focused synthesis develops an empirical comparison that preserves the terminal action, observation process, and total verification cost. Its proposed tests are prospective; no new simulation run, model-training result, or intervention effect is reported.

References

  1. Z. Qiao, J. Lyu, C. Bai, P. Wang, S. Gao, and S. Qiu, “Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets,” Proceedings of the 2026 International Conference on Machine Learning, 2026.
  2. Z. Qiao, J. Lyu, B. Lyu, Y. Shu, S. Gao, and S. Qiu, “Model-Based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting,” Proceedings of the 2026 International Conference on Learning Representations, 2026.
  3. Li, Yuanhao, Mingshan Liu, Hongbo Wang, Yiding Zhang, Yifei Ma, and Wei Tan. “DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 35, 2026, pp. 29530-29537. https://doi.org/10.1609/aaai.v40i35.40195.
  4. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  5. Bertsimas, Dimitris, and Melvyn Sim. “The Price of Robustness.” Operations Research, vol. 52, no. 1, 2004, pp. 35–53. https://doi.org/10.1287/opre.1030.0065.
  6. Rasmussen, Carl Edward, and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006. https://gaussianprocess.org/gpml/.
  7. Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” International Conference on Learning Representations, 2022. https://openreview.net/forum?id=nZeVKeeFYf9.
  8. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  9. Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
  10. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
  11. Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191. https://arxiv.org/abs/2006.04779.

Downloads

Published

2025-09-15

Versions

Issue

Vol. 1 No. 9 (2025): September 2025

Section

Articles