Giving Each Response Equal Weight Changes the Factuality Estimand
Abstract
The question concerns each response weighted equally using its error-segment fraction. The secondary ordinary least squares analysis contains 845 observations and uses HC3 covariance. The coefficient on log_words is -0.0218, with a 95 percent interval from -0.0724 to 0.0288. Averaging each response's error fraction weights responses equally rather than weighting them by the number of segments. This denominator can differ from the pooled segment rate even when labels are unchanged. Heterogeneous value alignment motivates explicit outcome compatibility, but the annotation collection is not offline transition data and cannot demonstrate a policy-value improvement. The estimates describe the released collection and its conditioning set. No new data collection, simulator experiment, or deployment-policy evaluation is claimed.
References
- Z. Qiao, J. Lyu, C. Bai, P. Wang, S. Gao, and S. Qiu, “Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets,” Proceedings of the 2026 International Conference on Machine Learning, 2026.
- S. Chen, Y. Zhao, J. Zhang, I.-C. Chern, S. Gao, P. Liu, and J. He, “FELM: Benchmarking Factuality Evaluation of Large Language Models,” Advances in Neural Information Processing Systems, pp. 44502–44523, 2023.
- Tong, H., Xu, G., & Chen, Y. (2026, May). Hadamard Tensor Ring for Efficient Low-Rank Fine-Tuning. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Zhou, Yongxi, Wenbo Ye, Yuanzhe Liu, Zihan Dong, and Junwei Yao. “Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts.” arXiv, 2026, arXiv:2609.08236. https://arxiv.org/abs/2609.08236.
- Sang, Yinghao. “Towards Explainable RAG: Interpreting the Influence of Retrieved Passages on Generation.” 2025 4th International Conference on Robotics, Artificial Intelligence and Intelligent Control (RAIIC), IEEE, 2025, pp. 397-400. https://doi.org/10.1109/raiic65850.2025.11170170.
- White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
- Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
- Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
- Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
- Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
- Lewis, Patrick, et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474. https://arxiv.org/abs/2005.11401.
- Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” International Conference on Learning Representations, 2022. https://openreview.net/forum?id=nZeVKeeFYf9.
- Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.