Journal of Research and Discovery
Original publication date: 2026-08-12
Archive migration: 2026-10-30
Brian Rogers — Iraq; Cody Walker — Israel; Julian Hoffman — Grenada
Averaging generated-image scores by prompt changes the relationship observed between quality and alignment. In AGIQA-3K, the correlation between the 300 prompt-level means is 0.545, with a rank correlation of 0.529. The average of the ten within-condition correlations is 0.500. This secondary analysis shows that aggregation units carry substantive meaning. It does not identify a cause for the remaining disagreement or validate a new image-quality model. The results support keeping semantic and perceptual outcomes separate when evaluating prompts and generators.
A row-level correlation describes how individual released image scores move together in the observed table. A prompt-level correlation instead compares averages across the generation conditions available for each prompt. These quantities need not agree because they assign variation to different units. A strong relationship at one level cannot automatically be transported to another. This study asks what the public score record shows when the prompt, rather than the individual image row, becomes the descriptive comparison unit.
Decoupling Semantics from Distortions motivates a distinction between semantic and perceptual evidence in generated-image assessment (Meng, 2026). The present study uses that distinction to interpret human quality and alignment means. It does not compute the focal architecture's predictions or reproduce its multiscale streams. Its empirical contribution is a transparent aggregation analysis using released scores, with attention to which prompt-condition observations exist and how their averages define the resulting relationship.
AGIQA-3K supplies observations of human judgments about generated images (Li et al., 2023). Its released score table can support a secondary analysis without downloading the images, but that analysis has a defined information boundary. It does not observe new image features, rerun a quality model, or recover each participant's response from a mean and standard deviation. Dataset documentation should connect rows to prompts, generation conditions, collection decisions, and intended use (Gebru et al., 2021). Reproducible software records also matter when constructing splits or transforming variables (Pedregosa et al., 2011). The analyst should retain source labels rather than reinterpret a filename as a fully controlled experimental intervention. A prompt repeated across conditions creates useful pairing, yet the published table may still contain missing cells or unmeasured generation differences. A transparent report states which records are present, how conditions were identified, and which outcome was actually analyzed. Those details make a modest score-based result independently understandable without expanding it into an unperformed assessment of image content or model performance.
The released table contains 2,982 rows and 300 unique prompts across ten condition labels derived from filenames. For each prompt, the analysis takes the arithmetic mean of the observed quality means and the arithmetic mean of the observed alignment means. It also records the number of contributing rows. No missing prompt-condition score is imputed. The resulting summaries therefore describe available observations rather than an assumed complete generation experiment with every cell filled.
There are 287 prompts with ten rows, nine with nine rows, three with eight rows, and one with seven rows. These counts show that most prompt means have complete condition coverage while a small subset represents a different mixture. A complete-prompt analysis would define another descriptive population. The present calculation retains all 300 prompts and makes their unequal support explicit. It does not treat the missing rows as measured failures or infer why those combinations are absent from the source release.
Pearson correlation across prompt-level quality and alignment means is 0.545. Spearman rank correlation is 0.529. Separately, the analysis computes a Pearson correlation within each of the ten condition groups and takes their unweighted arithmetic mean, yielding 0.500. That latter summary gives each condition equal weight; it is not a pooled correlation and should not be interpreted as one. Neither statistic by itself measures the frequency of consequential disagreement on individual images.
Precision and identification are separate properties of a result. A narrow interval can surround an association whose scientific target is unclear. Robust covariance calculations allow particular departures from simple error assumptions, including unequal variance (White, 1980) and dependence within declared clusters (Cameron and Miller, 2015). Their role is to improve uncertainty calculations under those assumptions. They do not establish a causal assignment, guarantee a correct measurement, or expand coverage to conditions missing from the data. The analyst should therefore explain both the averaging population and the source of replication. An evaluation repeated many times on one history may provide information about stochastic variability while supplying little evidence about transfer to another history. Those two sources of uncertainty should remain distinguishable in the final interpretation.
An assessment instrument should be chosen for a stated target. CLIP learns visual-language correspondences (Radford et al., 2021), whereas MUSIQ addresses image quality using multiscale processing (Ke et al., 2021). NIMA's learned assessment setting (Talebi and Milanfar, 2017) and NIQE's natural-image-statistics approach (Mittal et al., 2013) introduce further distinctions in what evidence enters a score. Their relevance is not that one number can replace all the others. It is that perceptual appearance, aesthetics, and instruction correspondence need explicit definitions and validation populations. An image may rank differently under those requirements for meaningful reasons. A report should preserve the component outcomes before considering any combined utility. If only published human means are analyzed, the study can examine the relationship among those means but cannot claim a new benchmark result for an automated architecture. A further experiment would need actual predictions, appropriate splits, and independent outcome information. Keeping those requirements separate allows a score-based empirical result to remain useful without borrowing performance claims from methods that were never run.
An evaluation protocol should state what remains unfamiliar at assessment time. Leaving out an observation while retaining its close relatives can test interpolation rather than the broader transfer suggested by the study's wording. Validation methods for structured data emphasize matching the split to temporal, spatial, or hierarchical dependence (Roberts et al., 2017). Separately, parameter selection can bias an evaluation if the same observations guide both choices and reporting (Varma and Simon, 2006). These issues are not solved by calling a split a test set. The record should identify shared sources, the information accessible during development, and the stage at which each design choice was fixed. A result can be informative within a narrow boundary while providing no direct evidence about a more distant deployment population.
Prompt-level averaging can reduce some image-specific variation while preserving differences in the conditions contributing to each prompt. It can also combine favorable and unfavorable outcomes that cancel in the mean. A prompt with similar average quality and alignment may still include individual images where the two judgments diverge sharply. The present result therefore describes relative prompt summaries, not universal agreement between perceptual quality and semantic success within every generated example.
The remaining disagreement should not be assigned a unique cause. It could relate to prompt content, generation behavior, assessment conventions, or unobserved characteristics of individual images. The table-based analysis has not processed image pixels or participant-level responses. An adjusted or experimental follow-up could investigate specified explanations, but a correlation below one does not identify which mechanism produced it. The useful observation is the coexistence of positive association and incomplete correspondence under a clear aggregation rule.
For model evaluation, prompts should remain grouped when they define the intended assessment boundary. A predictor assessed on new prompts faces a different task from one assessed on new images of familiar prompts. The present descriptive calculation is not a prediction experiment, yet it helps motivate that design choice. It also suggests reporting separate quality and alignment outcomes at the level relevant to the application. A single combined score would require an explicit preference rule and would conceal information about their observed disagreement unless the components remain available.
A metric's original evaluation setting matters when interpreting a generated-image result. KonIQ-10k (Hosu et al., 2019) and KADID-10k (Lin et al., 2019) offer different quality-assessment populations and distortion structures. Structural similarity (Wang et al., 2004), learned perceptual correspondence (Zhang et al., 2018), and feature-distribution comparison through FID (Heusel et al., 2017) provide further examples of distinct measurement units. Some compare paired images; another concerns a collection's distribution. Treating them as equivalent quality labels can obscure the information needed to compute and validate each. A generated-image study should identify whether it has an appropriate reference image, an independently sampled comparison distribution, or only released human judgments. The answer determines which empirical claims are available. A table-based regression may reveal a useful relationship between human outcomes while saying nothing directly about an uncomputed metric. A later experiment can assess such metrics under matched data and declared splits, but its proposed procedure should remain separate from the observations already present in the released score table.
Describing a contrast accurately can be more valuable than giving it an unjustified mechanistic name. A causal interpretation needs assumptions about how exposure and outcome arose, including selection into the analyzed sample. Causal diagrams provide a language for those assumptions (Greenland et al., 1999). If several alternative metrics or partitions are examined, the investigator should also account for the freedom to select whichever result appears strongest. False-discovery control is relevant to explicit collections of statistical tests (Benjamini and Hochberg, 1995), while a descriptive sensitivity analysis should be presented as such. A model can fit the observed rows well and still leave the principal explanation unresolved. Reporting that boundary preserves the value of the observation without overstating what adjustment or a significance threshold has established.
Replacing an expensive judgment with an automated score changes the measurement process. A positive correlation is evidence of co-movement, but it does not establish that the two procedures can be used interchangeably (Bland and Altman, 1986). A calibrated forecast has a different requirement: its stated probabilities should match relevant observed frequencies (Guo et al., 2017). Neither property follows merely from a favorable overall ranking. A useful comparison preserves paired measurements, identifies the conditions in which disagreement matters, and keeps scale transformations explicit. If the intended judgment concerns recognizability, playability, or physical compatibility, the proxy should be assessed against that judgment directly. Agreement on easy cases cannot supply evidence about the difficult conditions absent from the comparison.
Human preference introduces another target beyond quality and prompt alignment. Pick-a-Pic supplies comparative preference data (Kirstain et al., 2023), and Human Preference Score investigates preference-oriented assessment (Wu et al., 2023). DINO's self-supervised visual representation (Caron et al., 2021) can support another view of image content, but its training objective does not define the user's complete utility. Reference-conditioning methods such as DreamBooth (Ruiz et al., 2022) and IP-Adapter (Ye et al., 2023) also change what information generation receives. A quality comparison should record that input boundary and avoid treating a reference-rich workflow as equivalent to one with less subject evidence. Preference, recognizability, prompt execution, and perceptual appearance can disagree. The useful scientific response is to retain the relevant outcomes and state any application-specific weighting. A released mean-score analysis can motivate these distinctions without collecting preferences or running conditioning models. Its contribution should remain the observed relationship among available judgments, while further claims about user choice and reference-driven generation require the corresponding independent measurements.
Coverage should be reported whenever the system or its evaluator can avoid difficult cases. An abstention policy changes the population on which an accuracy or fidelity score is calculated. Selective classification formalizes the joint reporting of coverage and conditional performance (Geifman and El-Yaniv, 2017). Distribution-shift research adds that estimated uncertainty can behave differently outside the development population (Ovadia et al., 2019). A benchmark should therefore retain which observations were eligible, accepted, rejected, or unavailable. A result on a carefully limited subset can be useful, provided the subset is visible. The challenge is to prevent a narrow success claim from becoming a broad capability claim through omission of the denominator or the circumstances in which the system declines to operate.
The unit of provenance should be fine enough to reconstruct an important disagreement. Reusable identifiers can connect a reported score to its input evidence and the relevant model and evaluator versions (Wilkinson et al., 2016). Pipeline dependencies should be treated as part of the experiment because apparently minor changes can alter output eligibility or metric behavior (Sculley et al., 2015). A report that preserves only aggregate numbers prevents this reconstruction. Keeping per-case results and documented transformations enables alternative weighting and error analysis without repeating expensive generation. Such a record also helps identify whether two papers compare the same scientific objects or merely use similar names for different releases. Versioned evidence makes progress easier to assess across successive evaluations.
Quality and alignment have a positive but incomplete relationship across prompt-level averages in the released AGIQA-3K table. The result depends on the aggregation unit and observed condition coverage. It neither validates an automated architecture nor explains why particular prompts diverge. Its contribution is a reproducible prompt-level description that keeps semantic and perceptual outcomes distinct and clarifies the additional measurements needed before their disagreement receives a mechanistic or predictive interpretation.