Resource Allocation and Optimization · Vol. 1 No. 1 (2026)
Original publication: October 31, 2026
Alexander Stewart — Poland; Aaron Knight — Greece; Paul Lewis — Croatia
Keeping one hundred rated images from each generator changes representation, but does it improve prediction at a fixed total budget? Two 1,500-image menus from AIGIQA-20K were used to fit the same metadata regression, with exact prompt groups held out. Generator-balanced selection and uniform image selection produced almost identical equal-generator mean squared errors, 0.21765 and 0.21757. Seven generators favored balancing and eight favored the uniform menu. Balanced counts alone therefore supplied no clear predictive gain in this fixed comparison, although they changed which generators received the retained ratings.
An evaluation team retaining a limited number of rated images must decide both how many to retain and where those images should come from. Those are distinct resource decisions. Increasing a total budget can reduce sampling variability while leaving a generator underrepresented; balancing generator counts can alter representation without adding a single rated image. This article isolates the latter question: at a fixed number of retained source images, does generator-balanced selection improve prediction when each generator is given equal weight during evaluation?
The distinction matters for generated-image quality assessment because source collections often contain unequal numbers of images from different systems. A model trained on such a collection may receive considerably more information about one generator than another. Meng, 2026 addresses quality assessment through a separation of semantic and distortion information. We examine an upstream allocation choice rather than that architecture. The prediction model uses only released configuration metadata, and no target neural model, image pixels, semantic embedding, or new rating exercise enters the experiment.
The outcome is the released composite MOS in AIGIQA-20K, whose database is described by Li et al., 2024. It should not be relabeled as a pure distortion score. The related Li et al., 2023 resource separately highlights quality and alignment assessment, illustrating why an aggregate label's construction affects what a predictor can claim to learn. Our resource question is consequently narrow: how two subsets of already rated images support prediction of the source's composite outcome. Neither subset represents a measured financial annotation cost or a fresh sample of human judgments.
Balanced allocation has an intuitive appeal: every generator receives a minimum amount of information. But equal training counts do not automatically minimize equal-generator prediction loss. Generators may differ in residual variation, predictor support, and how much their coefficients benefit from the common slopes. The sampling perspective in Neyman, 1934 makes allocation an explicit design choice instead of a synonym for representativeness. The proposition tested here is therefore two-sided. Balance might help poorly represented categories, yet it could also remove useful observations from categories with more varied settings.
The role of evaluation weights is equally important. Solon et al., 2015 discusses the interpretation of weighted and unweighted analyses, while Kitagawa, 1955 separates composition from within-group quantities in comparisons of rates. Here the primary evaluation averages mean squared error across fifteen source generator categories with equal category weight. A secondary descriptive summary weights all held images equally. These targets need not agree because held generator counts are unequal. We do not choose the preferred weighting after inspecting which menu has the lower loss.
This allocation question differs from visual model design. Talebi and Milanfar, 2017 learns image-assessment outputs, Ke et al., 2021 uses a multiscale architecture, and Ying et al., 2019 addresses picture-quality assessment from visual inputs. Our deliberately simple regression makes the retained-data decision easier to inspect, but its simplicity also narrows generalization. A generator-balanced menu could interact differently with a high-capacity visual learner, particularly if visual feature distributions vary across systems in ways that configuration labels and prompt length fail to represent.
Nor are preference and prompt fidelity interchangeable with the present outcome. Wu et al., 2023 and Xu et al., 2023 concern human preference-related evaluation, while Ghosh et al., 2023 and Huang et al., 2023a organize tests of content correspondence and compositional abilities. Those perspectives motivate preserving the source label's meaning. A menu that predicts composite MOS well has not necessarily learned which objects appear in an image, whether a spatial relationship matches a prompt, or which image a particular user would prefer. Equal generator coverage does not resolve those construct differences.
The starting table contains 20,000 image records from fifteen generator categories and 19,259 distinct exact prompt strings. Prompts are ordered by the SHA256 digest of a fixed prefix, budget20261107: followed by the exact prompt. The first eighty percent of distinct groups form the training pool and the rest form the holdout. This produces 16,015 pool images and 3,985 held images representing 3,852 prompt groups. Every source generator occurs in both sets. Exact repeated strings remain together, while semantic near-duplicates are not identified or merged.
This grouping responds to the validation concerns in Roberts et al., 2017: records sharing a prompt are not equivalent to unrelated new inputs. The leakage account in Kapoor and Narayanan, 2023 further motivates keeping held outcomes out of both retention menus and fitting. The held set is frozen before either model is trained. Nothing about a held rating, a held loss, or a favorable generator result is used to change the allocation. It remains possible for semantically similar prompts with different strings to occur across sets, so the split should not be described as a complete semantic-content separation.
For uniform selection, pool images are sorted by a second fixed SHA256 rule using the prefix allocation20261108: and the image name, and the first 1,500 are retained. For generator-balanced selection, that same ordering is applied separately within each category, retaining one hundred images from every generator. The resulting menus overlap on 1,165 image records. This overlap is intentional: the comparison changes allocation while holding the source pool and deterministic ordering principle constant. It is not a comparison between independent newly acquired rating samples.
Uniform-menu counts range from 28 for DALLE 3 to 166 for Pixart XL 2. These unequal counts are an observed consequence of the declared image-level selection rule, not a claim that either generator merits that allocation. The balanced menu replaces the count distribution with exactly one hundred per category. Both menus contain 1,500 existing MOS labels. Differences in the expense of obtaining those labels, the number of individual raters behind them, or the practical cost of identifying a generator were not observed and are not inferred from the retained counts.
Each menu fits the same ordinary least-squares formula. It contains an intercept, fourteen generator-category indicators, raw guidance configuration, log one plus iteration count, log image area, log one plus prompt word count, and log aspect ratio. The design has twenty parameters and full numerical rank in both fits. Source-coded zeros remain as recorded configuration values alongside the category indicators. We do not treat those zeros as common physical doses across all generators or interpret the fitted slopes as causal effects of changing a generation setting.
The modeling choice is intentionally modest. The dimensional-representation ideas of Pearson, 1901 and Tipping and Bishop, 1999 suggest alternatives for reducing richer inputs, but no principal components or latent image representation are fitted here. Hoerl and Kennard, 1970 offers a regularized alternative when correlated predictors threaten stability; we do not tune a ridge penalty in this experiment. Keeping one ordinary regression specification limits the degrees of freedom through which an allocation comparison could become an implicit model-selection exercise. Full rank establishes estimability, not suitability for every generator's response surface.
The error-estimation concern in Varma and Simon, 2006 explains why we do not optimize a menu or model specification against the held ratings. Predictions are generated once from each fitted model and remain unbounded, with no clipping based on holdout extrema. The same held image then has a paired squared-error difference, balanced minus uniform. Those differences are averaged within generator, giving fifteen category-level contrasts. An intercept-only OLS model estimates their equal-generator average. Figure 1 shows how the menu choice enters the analysis through fitting rather than through a changed evaluation population.
Figure 1. Equal retained-image budgets produce different generator coverage. The two frozen metadata regressions face the same prompt-grouped holdout, and the primary outcome weights the fifteen held generator categories equally.
We use HC3 covariance and a t reference with fourteen degrees of freedom for the fifteen generator-level rows. White, 1980 supplies the broader heteroskedasticity-robust perspective, but the small category count warrants restraint. The dependence discussion in Cameron and Miller, 2015 identifies limitations that this working summary does not eliminate: all categories share fitted common slopes, and exact prompts can cross generator categories. The interval therefore conditions on one pair of fitted menus and does not incorporate retraining variation, menu randomness, or all cross-category dependence.
Table 1. Balanced minus uniform held mean squared error. Each row in the intercept-only OLS analysis is one generator's paired mean loss difference. N is fifteen source generators, covariance is HC3, and the reference distribution has fourteen degrees of freedom. No controls or fixed effects enter this contrast model; the underlying prediction regressions each use the same twenty parameters.
| Model | Term | Estimate (SE) | 95% CI | N |
|---|---|---|---|---|
| M1 | Mean outcome/contrast | 0.000075 (0.001300) | [-0.002713, 0.002864] | 15 |
| M1: fit | R-squared = 0.0000 | - | - | 15 |
Model labels: M1 = Balanced minus uniform held MSE, equal generator.
Equal-generator mean squared error is 0.217571 for uniform retention and 0.217646 for balanced retention. The estimated difference is positive 0.000075, with a standard error of 0.001300 and a 95% interval from negative 0.002713 to positive 0.002864. The point estimate slightly favors the uniform menu, but the interval includes improvements and deteriorations of considerably greater magnitude than the estimate. The result supplies no clear average prediction advantage for either allocation in this fixed comparison; it does not demonstrate equivalence.
Seven generators have lower loss under the balanced fit and eight have higher loss. This heterogeneity matters because identical category counts do not imply identical predictive demands. The shared model may gain from more observations in one part of the configuration space while losing fit elsewhere. The observed direction count is descriptive and is not a sign test over independent generator draws. All fifteen categories are source-defined and share data-generation and model-fitting structures, so they do not constitute a random sample of every future image-generation system.
The image-weighted summary also slightly favors uniform retention: mean squared error is 0.208642 against 0.209943 for balancing. That summary has a different target from the table and is not substituted for it. Its lower numerical level reflects the distribution of held images across categories, rather than necessarily better prediction for every generator. Reporting both summaries makes the evaluation weights visible and prevents the apparently benign phrase overall error from concealing a change in which source categories receive the most influence.
The loss itself is a choice. Gneiting and Raftery, 2007 provides a general basis for specifying what a predictive score rewards, while the error-measure discussions in Willmott and Matsuura, 2005, Chai and Draxler, 2014, and Hodson, 2022 show why absolute and squared-error criteria should not be casually interchanged. We use squared error throughout the executed primary comparison. We do not add an absolute-error result that was not computed, or infer that the ranking of the menus would be identical under a different loss, a pairwise preference objective, or a content-correctness score.
This experiment studies source-category balance rather than population representativeness. The generators are known, their metadata labels are available, and the holdout includes every category. It does not estimate performance on an unseen generator. A generator-balanced menu also uses information that a fully anonymous image pool might not provide. Its one-hundred-per-category constraint is feasible in the observed source but could be infeasible or expensive for a rare generator in practice. No conclusion about the optimal allocation follows from comparing just these two fixed rules.
The source corpus is likewise distinct from other visual-quality resources. Hosu et al., 2019 concerns an ecologically motivated image-quality collection, Fang et al., 2020 addresses smartphone photographs, and Murray et al., 2012 supplies aesthetic vote distributions. Their category meanings and rating protocols differ from the generated-image source used here. A balanced allocation over devices, distortion types, photographers, or semantic categories would be a new experiment. The present findings should not be generalized to such designs simply because each can be described informally as balanced sampling.
The retained records and processing rules are documented in the spirit of Gebru et al., 2021 and the reuse aims of Wilkinson et al., 2016. They include the two image menus, generator counts, exact prompt grouping, fitted formula, per-image held losses, and generator summaries. Those records make it possible to separate this allocation study from a larger-budget experiment on the same database. Reusing a frozen holdout also creates an accumulation risk across successive analyses: the present question was specified before its result, but the collection is not an untouched external validation resource for unlimited future decisions.
The equal-generator target also does not require the training set to have equal counts as a matter of mathematics. Different fitting weights, category-specific models, or explicitly optimized allocations could target that loss in other ways. We did not search those alternatives. The comparison thus evaluates a familiar coverage heuristic against one uniform menu, with a common inexpensive regression. Treating the small observed gap as proof that representation never matters would extend the evidence as far as treating balance itself as a predictive guarantee.
Generator-balanced retention substantially changed source representation while leaving the total retained-image budget unchanged. In the executed metadata-only comparison, that change yielded virtually no average equal-generator MSE difference, with uncertainty spanning benefits and harms. The finding separates a legitimate representation objective from a predictive guarantee. Teams choosing a retention policy should state whether they seek coverage, a particular weighted loss, or measured annotation efficiency, because equal category counts alone establish only the first of those outcomes.