Resource Allocation and Optimization · Vol. 1 No. 1 (2026)
Original publication: October 31, 2026
Simon Hayes — Spain; Liam Hall — Portugal; Evan Anderson — Austria
Exploratory research draft. Fixed public program pairs and recorded execution events; no wall-clock speedup, newly generated repair, or fine-tuning result.
Choosing short tests can save work while removing useful evidence. On 26 QuixBugs tasks, selecting the three cases with the fewest corrected-reference execution lines reduced recorded pair-audit work from a uniform-policy expectation of 429,531 line events to 326,749. Mean bug detection fell from 92.7% to 76.9%. The geometric mean task-level work ratio was 0.311, illustrating how a typical proportional saving can differ from the total-work saving. These finite-benchmark results expose a workload-detection tradeoff rather than an unconditional testing optimization.
Limited evaluation resources create pressure to retain a small set of fast tests. A simple heuristic is to prefer cases whose reference implementation executes few source lines. Such cases may be convenient for constructing or checking many candidate comparisons. Yet a short reference execution need not expose the behavior that distinguishes a buggy candidate. The heuristic uses one kind of information to allocate effort toward an outcome that it does not directly measure.
We ask how bounded testing workload and bug detection change when three cases are selected by reference execution count rather than uniformly retained. The comparison holds the number of tests constant within every program. Its purpose is to inspect the tradeoff between a measurable work proxy and the evidence retained by the tests. The public QuixBugs pairs Lin et al., 2017 provide fixed candidates and expected outputs, while regression-test prioritization research Rothermel et al., 2001 motivates treating test order and selection as consequential decisions rather than incidental implementation details.
The question is relevant when tests help construct training or evaluation preferences for coding systems. Tao and Kamata, 2026 study fine-tuning coding assistants from pairwise preferences about what to change. We do not train an assistant or use its preference data. Instead, we examine a possible upstream choice: which available cases are retained when a fixed source pair is audited? If a cost heuristic removes discriminating tests, the resulting relation can become less informative even though its construction requires fewer recorded execution events.
The analytical proposition is a tradeoff, not a presumption of free efficiency. Reference-cost selection may lower pair-audit work, but it can also lower the probability of detecting the captured buggy implementation. The two outcomes must therefore be reported together. A fixed-count heuristic is not automatically a fixed-work policy, and reduced measured work is not automatically improved decision quality. The experiment uses finite recorded executions to make each component of that statement explicit.
The starting benchmark execution record contains thirty-one JSON-backed Python tasks. We restrict this study to the twenty-six whose repository-designated corrected implementation passed every captured case under the declared compatibility oracle. The retained tasks contain 198 cases. That eligibility rule is inherited from the prior execution audit, rather than introduced to improve the cost heuristic's outcome. Tasks failing the reference gate remain outside this comparison, and their exclusion limits the claim to the compatible finite cohort.
The underlying calls use reviewed pure functions, copied JSON arguments, explicit standard-library helpers, and a limit of one hundred thousand executed source lines per candidate call. Generator outputs are materialized, tuple and list containers normalized for comparison, and numeric values checked with the recorded tolerance. The broader oracle problem Barr et al., 2015 explains why those conventions belong in the evidence record. A reference-pass gate does not establish complete semantic correctness beyond the supplied cases; it defines a reproducible comparison cohort.
Executed lines are recorded through the Python tracing interface Python Software Foundation, 2026. Only events in the reviewed benchmark function source contribute to the count. Operations inside compiled helpers are not assigned a corresponding number of source-line events. A single traced line can also do very different amounts of computation depending on its expression and input. Cyclomatic complexity McCabe, 1976 describes another structural property, but it is neither our selection variable nor an inferred replacement for these measured execution counts.
The resulting workload unit is pair-audit source-line events: the sum of reference and buggy-candidate events for selected cases. It is not elapsed time, processor instructions, memory consumption, or a monetary cost. Measurement cautions illustrated by Mytkowicz et al., 2009 and rigorous benchmarking discussion Kalibera and Jones, 2013 motivate keeping those quantities separate. No timing experiment is performed here, and no numerical line-count reduction is converted into an advertised runtime speedup.
The cost-guided policy orders each program's cases by corrected-reference execution-line count, breaking ties by the captured source-case index. It keeps the first three. The ranking does not inspect buggy execution status, buggy work, or whether a particular test detects the bug. This information restriction matters: a policy that chose cases after examining detection would be an oracle-guided selection exercise rather than a usable reference-cost heuristic. The selected case identifiers and their recorded statuses are saved.
The comparator retains three cases uniformly without replacement from the same program's available cases. We compute its exact finite-record detection probability from the number of passing and failing buggy cases. Its expected additive pair work is three times the program's average observed per-case pair work. No Monte Carlo sample of test subsets is needed for either expectation. The deterministic cost-guided selection is compared with those exact uniform expectations for each task.
Both policies therefore retain seventy-eight cases across the twenty-six tasks, counting the three selected or expected-retained cases per task. They do not receive the same realized line-event budget. This is an equal-count policy comparison whose workload is an outcome. A separate optimization under a common event budget would answer another question and could allocate different numbers of cases to different programs. We do not silently reinterpret the current design as that budget-constrained optimum.
Figure 1. Reference-only test ranking, fixed three-case retention, and workload-detection outcomes.
Figure 1 shows the available information and the two downstream summaries. Reference trace counts determine cost-guided retention; buggy outcomes enter only evaluation. The same captured source pairs and oracles underlie both policies. The diagram includes no assistant-generation or fine-tuning stage. Direct preference optimization Rafailov et al., 2023 and human-feedback instruction learning Ouyang et al., 2022 provide downstream contexts in which evidence quality can matter, but the present mechanical detection outcomes are not certified human preferences or measured training benefits.
For every program, the first summary is the log of cost-guided pair work divided by uniform expected pair work. Averaging this log quantity gives an equal-task proportional comparison; exponentiating the average gives the geometric mean work ratio. The second summary is cost-guided detection, which is zero or one, minus the exact uniform three-case detection probability. Both are fitted with intercept-only OLS across twenty-six task rows, using HC3 standard errors and normal-reference intervals.
Robust covariance reasoning White, 1980 motivates reporting the variance estimator, while the dependence discussion in Cameron and Miller, 2015 cautions against treating all test cases as independent tasks. The regression N is twenty-six, not 198 or seventy-eight. These intervals are conditional summaries for hypothetical task heterogeneity around the captured cohort. They do not represent Monte Carlo error, since the uniform expectations are calculated exactly, and they do not establish the frequency of the effect in an independently sampled population of future software problems.
We also report total recorded work because it answers a different aggregation question. Equal-task log averaging gives each program one proportional contribution, while summing line events lets programs with large absolute traces dominate. Neither summary should be relabeled as the other. Scoring and estimation principles Gneiting and Raftery, 2007 favor declaring that target before interpreting a performance number. Cost-sensitive learning Elkan, 2001 would additionally require consequences for missed bugs and resource use before deciding whether the observed tradeoff is acceptable.
No noninferiority margin is specified for detection. We therefore cannot declare the heuristic harmless simply because it saves recorded work or because a particular confidence interval might include zero in another sample. The model-selection concerns described by Varma and Simon, 2006 also apply to searching among many ranking features. Here the one declared ranking uses reference lines only. The analysis does not choose the most favorable heuristic from an unreported set of cost predictors or retention budgets.
The uniform three-case policy has equal-task mean detection probability 0.9265123. The cost-guided policy detects the buggy implementation in twenty of twenty-six tasks, giving 0.7692308. Their mean difference is negative 0.1572815, equivalent to about 15.7 percentage points. The standard error is 0.0640839 and the 95% interval extends from negative 0.2828837 to negative 0.0316793. This is a measurable loss of retained detection evidence under the declared comparison.
The mean log work ratio is negative 1.1673025, with standard error 0.3520949 and interval from negative 1.8573958 to negative 0.4772091. Exponentiating the point estimate gives geometric mean ratio 0.3112053. That ratio describes the task-level proportional summary. The total cost-guided work is 326,749 line events, compared with a uniform expectation of 429,530.7101 events. Fractional expected work is legitimate because it averages over possible uniform subsets, while the deterministic retained set has an integer event count.
| Model | Term | Estimate (SE) | 95% CI | N |
|---|---|---|---|---|
| M1 | Mean outcome/contrast | -1.1673 (0.3521) | [-1.8574, -0.4772] | 26 |
| M1: fit | R-squared = 0.0000 | - | - | 26 |
| M2 | Mean outcome/contrast | -0.1573 (0.0641) | [-0.2829, -0.0317] | 26 |
| M2: fit | R-squared = 0.0000 | - | - | 26 |
Model labels: M1 = Reference-cost-selected versus uniform log pair-work ratio; M2 = Reference-cost-selected minus uniform detection probability.
Table 1 reports two intercept-only OLS models, HC3 standard errors, normal-reference intervals, N26, no controls, and no fixed effects. M1 estimates the mean log pair-work ratio; M2 estimates the mean detection difference. Zero R-squared is a property of the mean-only specification. Work counts source-line trace events for both candidates, including bounded operational failures, and does not measure wall-clock speed. Detection uses the same captured expected outputs and eligibility cohort for both policies.
The task direction counts add detail. Cost guidance has lower recorded pair work than the uniform expectation in twenty-three tasks and higher work in two; one is equal. Detection is better in two tasks, worse in six, and equal in eighteen. Thus a reference-cost ranking does not guarantee a lower buggy-execution cost, and its detection loss is concentrated rather than uniform across the cohort. The saved task rows identify those cases without filtering them from the aggregate result.
The geometric work ratio and total-work comparison differ markedly because they weight the cohort differently. A large proportional reduction in a small task contributes to the log average but may save few events in the total. A bounded long-running buggy case can dominate total work even if most programs are cheap. We do not remove such operational failures as inconvenient outliers. Their presence is part of the declared bounded audit, and the limit is retained in the interpretation of every workload figure.
The finding is consistent with a selection mismatch. Reference execution length measures how the designated corrected implementation traverses a case; it does not directly measure how strongly that case distinguishes the captured bug. A short path can omit relevant boundary behavior, while a cheap reference case can still cause an expensive buggy execution. The analysis does not identify which structural feature explains each loss. An influence examination following Cook, 1977 could inspect dominant tasks, but no unreported diagnostic is used to delete them or redefine the selected cases.
The comparison also omits the cost of obtaining reference profiles. In the source audit, all relevant cases had already been executed. A new workflow would need those measurements or estimates before applying the heuristic. That setup expense could exceed the saving from retaining three cases later, particularly for a one-off task. The study therefore reports conditional selection-stage work, not a net end-to-end efficiency gain. Production workflows with repeated candidate audits might amortize profiling differently, which would require its own measurement.
Coding benchmarks vary in how such evidence is organized. MBPP Austin et al., 2021, Codex-associated code evaluation Chen et al., 2021, and EvalPlus Liu et al., 2023 supply different task and test constructions. LiveCodeBench Jain et al., 2024 introduces another evaluation setting. Their cases are not included in the current line-event totals. The observed tradeoff should motivate testing a selection policy in its intended benchmark, rather than assuming that a favorable or unfavorable result transports automatically across different problem interfaces and test suites.
Repository agents make the distinction still broader. SWE-agent Yang et al., 2024 and Agentless Xia et al., 2024 operate in issue-resolution settings that differ from these short pure functions. CodeRL Le et al., 2022, DeepSeek-Coder Guo et al., 2024, and InCoder Fried et al., 2022 provide other coding-model contexts. No model from those works is run here. Their relevance is to define possible downstream users of execution evidence, while avoiding an unsupported claim that our simple retention heuristic explains their reported performance or should replace their evaluation procedures.
Structured evaluation principles Roberts et al., 2017 favor keeping source overlap and cohort restrictions visible. This study reuses the same bounded program execution evidence as the uniform-retention analysis, but changes the policy from varying a random case count to selecting a fixed count by reference cost. It is not an independent new benchmark replication. The contextual interpretation urged by Wasserstein and Lazar, 2016 also favors the paired workload and detection magnitudes over a single label of efficient or inefficient.
The retained records connect selected case indices to source-case counts, both implementations, event totals, execution statuses, uniform expectations, and task contrasts. Such construction details belong in dataset documentation Gebru et al., 2021. The tracing and production-failure topic in Yan, 2026 supplies a useful comparison of evidential context: a cheap execution summary is not automatically a diagnosis of the defect that matters. No microservice tracing system is evaluated here. Undocumented selection criteria can still become technical debt Sculley et al., 2015 because they alter which failures an evaluation signal is able to reveal.
For this finite cohort, reference-cost selection reduced recorded audit work while sacrificing bug detection. The result does not reject all cost-aware testing, and it does not endorse uniform sampling as universally optimal. It shows why a resource heuristic should be evaluated against the evidence objective it is meant to preserve. Fewer execution events are one outcome; retaining useful distinctions between candidate programs is another. Both are necessary to assess this three-test policy honestly.