Pith. sign in

REVIEW 4 major objections 7 minor 29 references

A statistics-first gate that routes only ambiguous SFT-data procurement cases to expensive debate reaches 90% routing accuracy, approaching an oracle that sees ground truth, while spending less than escalating every case.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:47 UTC pith:VBK5IWPE

load-bearing objection Honest negatives and a genuinely new composition, but the closed-loop benchmark and missing error bars leave the headline routing claim unsupported. the 4 major comments →

arxiv 2607.18960 v1 pith:VBK5IWPE submitted 2026-07-21 cs.LG cs.AIcs.CR

SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

Arther Tian , Alex Ding , Simon Wu , Aaron Chan This is my paper
classification cs.LG cs.AIcs.CR
keywords SFT data procurementcost-aware routingstatistics-first gateLLM-as-judgedebate biasdata quality axescalibrationinterval estimation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that deciding whether to buy a corpus of supervised fine-tuning data is a cost-aware routing problem: cheap interval estimates of diversity, utility, and redundancy should settle most cases, and expensive LLM adjudication should be reserved for the cases cheap statistics cannot settle. The proposed gate commits to a verdict only when at least two-thirds of the axes are decisive (adequate sample, tight interval), none are borderline, and the decisive axes agree; everything else goes to a one-round buy/reject debate before a chair. On a controlled synthetic bench of 60 units, the gate reaches 0.90 accuracy and 0.83 F1 at $0.017 per unit, beating the always-verify floor (0.75) and the always-escalate policy (0.83 at higher cost) and approaching the oracle (0.98). The same evaluation exposes the debate path's biases — the reject advocate wins 80% of debates and half the verdicts flip under position swapping — which the paper reads as evidence that a cheap-first design should keep that path off clear-cut cases. A sympathetic reader would care because procurement commitments happen before any training, under budget, and the claim is that routing, not scaling, captures most of the value of expensive adjudication.

Core claim

The central claim is that a statistics-first gate over interval-valued estimates of three corpus-level quality axes — diversity, utility, and redundancy — can route procurement decisions near an oracle without seeing the gold label, and that the gate's value is the routing rule itself, not any new estimator. In the paper's controlled evaluation, the gate sends only 10 of 60 units down the cheap path (the gate escalated 50), reserves debate for statistically ambiguous cases, and reports that escalating every case is worse on both axes because it drags a biased judge onto cases statistics had already settled. The paper also asserts that the debate path is useful but unreliable: its scores trac

What carries the argument

The load-bearing object is the gate rule: a decision rule that combines per-axis point estimates with confidence intervals, thresholds on sample size (n_min=20), interval width (w_max=0.25), a decisive-fraction requirement (f=2/3), a borderline band (m=0.05), and a unanimity condition across decisive axes; only when all conditions hold does it take the cheap verify path, otherwise it escalates to a two-advocate debate with a chair. The position-swap diagnostic — running the debate with the advocates' roles swapped in presentation order — is the mechanism that measures the debate path's positional and negativity biases.

Load-bearing premise

The gate's 0.90 routing accuracy rests on the assumption that cheap prefix measurements (entropy, usability pass-rate, near-duplicate rate) faithfully recover the true quality axes that define correctness; on this bench the gold label is a deterministic function of the same injected knobs the measurements recover, and on real corpora that recovery is untested.

What would settle it

Run the gate on real, non-injected corpora where correctness is defined by a downstream fine-tuning evaluation: if gate-approved corpora do not consistently yield better held-out models than gate-rejected ones, the routing value claimed here disappears. A less expensive check: compare the gate's verdicts against expert human buy/reject labels on a random sample of corpora; if accuracy falls toward the random-cost-matched baseline (0.82), the closed loop was carrying the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A data buyer can capture most of an oracle's routing accuracy at a lower cost than always-escalating, by spending LLM adjudication only on ambiguous cases.
  • The gate's calibrated buy-probability (ECE 0.241) is better calibrated than both the cheap floor (ECE 0.400) and the gold-routing oracle (ECE 0.413), so the confidence a buyer acts on is more trustworthy than either path alone.
  • Escalating every case does not just cost more; it is actively worse on accuracy (0.83 vs 0.90) and F1 (0.69 vs 0.83), because the biased debate path is applied to cases statistics had already settled correctly.
  • The measured negativity (0.80 reject win) and positional (52% flip) biases imply that debate verdicts should be treated as weak evidence and that the calibrated probability, not the bare label, should drive the acquisition decision.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • On real corpora there are no injected knobs, so the gate's side-classification thresholds (tau_div=0.15, tau_util=0.55, tau_red=0.25) have no ground-truth anchor; a testable extension is to measure how well the blind measurements track independently assessed quality axes, since the gate's routing decisions would collapse into random routing if those measurements are disassociated from true quality
  • The paper's closed-loop design suggests a strong falsification path: fine-tune identical models on gate-approved versus gate-rejected corpora and compare held-out task performance; the current claim predicts a clean rank separation that downstream training may not actually produce.
  • The debate bias results imply a simple remedy worth testing: averaging verdicts over swapped orderings (an ensemble over the 52% flip-prone pairs) may cancel much of the positional bias and narrow the remaining gap to the oracle, at approximately double the debate cost.
  • The same statistics-first gating pattern could transfer to other pre-training decisions — licensing text, purchasing prompt sets, or choosing evaluation suites — wherever a cheap signal can gate access to an expensive and potentially biased judge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SFGA, a two-stage decision system for SFT data procurement. Stage 1 computes interval estimates of three intrinsic axes (diversity, utility, redundancy) from a fixed prefix of the candidate corpus; Stage 2 is a gate that commits to a cheap statistics-only verdict only if at least 2/3 axes are decisive (n≥20, CI width≤0.25), non-borderline, and unanimous, and otherwise escalates to an adversarial LLM debate between buy- and reject-advocates, resolved by a presiding chair. The evaluation is a controlled 2×3×2 synthetic grid on Dolly-15k with 5 seeds, giving 60 routing units, where the gold label is a deterministic function of injected knobs. Headline results: 0.90 routing accuracy, 0.83 F1, $0.017/unit, sitting between always-verify (0.75) and oracle (0.98), and below always-escalate cost ($0.020). The paper also reports honest negative diagnostics of the debate path: a reject-advocate win rate of 0.80 and a 52% position-flip rate under advocate swapping. The authors explicitly frame the study as a measurement-fidelity and routing-calibration benchmark, with external validity as future work.

Significance. The architecture is plausible and the paper is exemplary in reporting negative results: the closed-loop limitation, the near-identity measurement recovery, and the debate-path biases are all stated prominently. If, after proper uncertainty quantification and a re-examination of the gate's decision rule, the routing advantage holds, the paper would make a useful contribution as a controlled benchmark and an honest case study in cost-aware evaluation. The debate diagnostics are a contribution in themselves. However, the current evidence does not support the headline routing claim as stated, because of the threshold-identity issue and the absence of error bars; the contribution is presently closer to a consistency check than to a validated procurement method.

major comments (4)
  1. [§5.2, Table 2] The headline 0.90 accuracy is a point estimate over 60 routing units (12 configs × 5 seeds) with no confidence intervals or significance tests. The benchmark is class-imbalanced: only 2 of 12 configs are buy cases (10 of 60 units), so an always-reject rule attains 0.833; the Wilson 95% CI for 54/60 is [0.79, 0.96], which includes 0.833. On the only stratum with both labels (d_high), 0.80 (24/30) has CI [0.61, 0.92], overlapping both the majority-reject rate (0.667) and the cost-matched random point estimate (0.82). Report cluster-level bootstrap by the 12 configurations, or per-config results, and add the majority-class baseline to Table 2. Without this, the margins over random and always-escalate are not established.
  2. [§3.3, §4, §5.1] The gate thresholds (τ_div=0.15, τ_util=0.55, τ_red=0.25) are identical to the thresholds defining the gold label, and the gold is a deterministic function of the same injected knobs the measurements recover. The paper itself says in §5.1 that near-perfect measurement recovery is 'close to an identity by construction.' Consequently, the routing-accuracy result largely restates the fidelity of the blind measurements; it does not test the procurement criterion against any external ground truth. The limitations section acknowledges this, but the abstract and Section 5.2 still present 0.90 accuracy as the central result. This framing must be corrected, and the paper should show the gate's performance on cases where the measurements disagree with the true knobs, if any.
  3. [§3.3, §5.2] The gate's cheap path requires the decisive axes to be unanimous, but the gold label is a conjunction. On this grid, a corpus with one failing axis and two passing axes is a 'conflict' and is escalated, even though the gold label is a clear reject. As a result the gate escalates 50 of 60 units and the cheap path handles only 10; the cost saving over always-escalate is $0.0025/unit (~13%). The 'statistics-first' claim is therefore weaker than it appears: the router routes most cases to the expensive path. Report the breakdown of escalation reasons (weak CI / borderline / conflict) and compare the gate to a simple cheap conjunctive rule (buy iff all three point estimates pass the same thresholds). This comparison is necessary to justify the architecture's routing decision.
  4. [§5.5] The debate-path diagnostics show a 52% position-flip rate and cross-seed std 0.098, which means the adjudication path is heavily order-dependent and noisy. Because 50 of the 60 units take this path, the 0.90 accuracy is largely attributable to this unstable component. The paper should report the accuracy and CI of the escalate path alone, and the escalated-case error rate, so the reader can see how much of the headline result is carried by the debate rather than by the statistics. As written, the honesty of the diagnostics and the claim that the gate keeps this path off the critical route stand in tension: on this benchmark the path is on the critical route for 83% of units.
minor comments (7)
  1. [Abstract, Table 2] The abstract says '$0.017 vs $0.020' but Table 2 reports always_escalate cost as $0.019660; round consistently or give exact values.
  2. [Table 3] F1 is listed as 0.000 for d_low, but the text says it is undefined by construction; use '—' and a note.
  3. [Eq. (2)] The borderline condition uses 'or' with overlapping conditions; clarify when an interval containing the threshold but with point estimate outside ±m is classified.
  4. [Figure 4] The '=1.00' labels and Spearman values should be in the caption or text; as printed the axes are unreadable.
  5. [Table 2] random_cost_matched has no F1 in Table 2; report it or explain why it is omitted.
  6. [References] Reference [3] is an anonymized placeholder; restore it before publication.
  7. [Table 2, §4] stats_only and direct_llm have n=36 rather than 60; explain why 24 units are excluded.

Circularity Check

1 steps flagged

The routing accuracy is a self-consistency check: the gold label is a conjunction of the same thresholds that parameterize the gate, and the measurements recover the injected knobs the gold is defined on; the paper itself says recovery is 'close to an identity by construction.'

specific steps
  1. self definitional [§3.3 (Eq. (2), Table 5), §4 (Gold label), §5.1 (Measurement fidelity)]
    "We use nmin=20, wmax=0.25, f=2/3, m=0.05, thresholds τdiv=0.15, τutil=0.55, τred=0.25, and z=1.96 intervals (all in Table 5). /// a corpus should be bought iff its true diversity, utility, and (low) redundancy all clear fixed thresholds, div_true ≥ 0.15 ∧ util_true ≥ 0.55 ∧ red_true ≤ 0.25. /// because the measurements and the gold label are both functions of the same injected knobs, near-perfect recovery is close to an identity by construction."

    The gate (side(a), Eq. (2)) classifies each axis with thresholds τdiv=0.15, τutil=0.55, τred=0.25, and the gold label is exactly the conjunction of those same thresholds applied to the true knob values. The blind measurements are functions of the same injected knobs (entropy, usability pass-rate, near-dup rate), and §5.1 reports near-perfect recovery (Spearman ρ≈1.00). Therefore the headline 0.90 routing accuracy measures agreement between two encodings of the same threshold rule, not an independent prediction. The paper discloses this and frames the result as measurement fidelity and routing calibration, but the central routing claim is nonetheless scored against a label constructed from the system's own decision thresholds, so it reduces by construction to a consistency check.

full rationale

The central routing result is circular by the paper's own equations: the gold label is defined by the same thresholds that parameterize the gate, and the blind measurements are functions of the same injected knobs the gold is defined on. §5.1 explicitly concedes that near-perfect recovery is 'close to an identity by construction.' Consequently, SFGA's 0.90 accuracy and its margin over baselines are comparisons against a label built from the system's own classification rule; they demonstrate internal consistency of the gate on a closed-loop benchmark, not external predictive validity. The debate diagnostics (negativity skew, positional flips) are independent, honest negative findings and are not circular. The self-citation to prior author work [3] is not load-bearing. The score is high because the principal quantitative claim—the routing accuracy against the gold—is forced by construction; the paper's transparency and explicit limitation statements mitigate but do not remove this circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The system introduces no new physical or model-level entities; the buy-advocate, reject-advocate, and presiding chair are role assignments to an existing LLM family. The free-parameter list is dominated by hand-set gate thresholds that coincide exactly with the gold-label thresholds, creating a strong circularity burden: the benchmark's headline accuracy is largely a consistency check between measurements and injected knobs. The remaining hyperparameters (n_min, w_max, f, m) set the escalation rate and are not justified by any risk or power analysis.

free parameters (7)
  • diversity threshold τ_div = 0.15
    Gate side-classification threshold identical to gold-label threshold div_true≥0.15 (§3.3, §4); hand-set rather than estimated, making routing accuracy on the bench partly circular.
  • utility threshold τ_util = 0.55
    Identical to gold-label threshold util_true≥0.55 (§3.3, §4); hand-set, circular with gold.
  • redundancy threshold τ_red = 0.25
    Identical to gold-label threshold red_true≤0.25 (§3.3, §4); hand-set, circular with gold.
  • minimum sample size n_min = 20
    Hand-chosen; no power analysis; affects which axes are 'decisive' and thus the escalation rate.
  • maximum CI width w_max = 0.25
    Hand-chosen decisive-interval width; impacts how often cases escalate.
  • decisive fraction f = 0.67
    Hand-chosen; directly sets the fraction of axes that must be decisive (Equation 3), and thus the gate's 50/60 escalation rate.
  • borderline margin m = 0.05
    Hand-chosen half-width of the borderline band in Equation 2; gates which cases are considered unambiguous.
axioms (5)
  • standard math Wilson interval for binomial proportions provides valid confidence intervals
    Used in the statistics engine for per-axis interval estimates (Section 3.3).
  • standard math Bootstrap interval estimation is valid for the empirical entropy and duplicate-rate measurements
    Invoked for interval estimation in Section 2; standard nonparametric tool.
  • domain assumption Blind prefix measurements (domain-bucket entropy, usability pass-rate, near-duplicate rate) are unbiased proxies for true diversity, utility, and redundancy
    Central measurement-fidelity premise (§3.1, §4); on real corpora there are no injected knobs to verify this.
  • ad hoc to paper The gold label 'buy iff div_true≥0.15 ∧ util_true≥0.55 ∧ red_true≤0.25' is the correct procurement criterion
    Defined in Section 4 as a deterministic function of the knobs; not derived from downstream training quality, and identical to the gate thresholds.
  • domain assumption LLM debate verdicts carry signal about true quality beyond the statistics (Spearman 0.94) and the calibrated buy-probability is meaningful
    Needed for the escalation path to add value (§5.5); assumption is measured only on this synthetic setup with one unnamed model family.

pith-pipeline@v1.3.0-alltime-deepseek · 9612 in / 11451 out tokens · 102393 ms · 2026-08-01T13:47:50.016319+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement." pith.science (2026). https://pith.science/paper/VBK5IWPE

@misc{pith2026260718960,
  author       = {Pith},
  title        = {Pith review of: SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBK5IWPE}},
  note         = {Machine review of arXiv:2607.18960}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present \sys{}, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes -- diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets ($2{\times}3{\times}2$ grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 $F_1$ at \$0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate (\$0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 ($p\approx3{\times}10^{-6}$) and a 52\% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work.

Figures

Figures reproduced from arXiv: 2607.18960 by Aaron Chan, Alex Ding, Arther Tian, Simon Wu.

Figure 1
Figure 1. Figure 1: Positioning. Existing procurement practice trades cost against reliability; SFGA spends [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end architecture. Cheap blind measurements feed a statistics engine whose [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Gate decision flow. The single most load-bearing figure: the gate is a branch, not a [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Measurement fidelity on the controlled bench: blind estimates vs. injected ground truth [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Routing accuracy vs. mean per-unit cost. The gate (star) sits near the oracle in accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Calibration error (ECE and Brier; lower is better) per strategy. The gate (highlighted) is [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Diagnostics of the escalation path: (a) the reject advocate wins most debates; (b) verdicts [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 21 linked inside Pith

  1. [1]

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. SemD- eDup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    A survey on data selection for language models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Prior work by the authors on trustworthy, cost-aware evaluation (anonymized for review), 2024

    Anonymous. Prior work by the authors on trustworthy, cost-aware evaluation (anonymized for review), 2024. Full citation withheld for double-blind review; to be restored in the camera-ready version

    2024

  4. [4]

    Glenn W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78(1):1–3, 1950

    1950

  5. [5]

    Instruction mining: High-quality instruction data selection for large language models.arXiv preprint arXiv:2307.06290, 2023

    Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models.arXiv preprint arXiv:2307.06290, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    AlpaGasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training a better alpaca with fewer data. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2307.08701

    Pith/arXiv arXiv 2024

  7. [7]

    Free dolly: Introducing the world’s first truly open instruction-tuned LLM

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned LLM. Databricks Blog, 2023. databricks-dolly-15k dataset

    2023

  8. [8]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. arXiv:2305.14325

    Pith/arXiv arXiv 2024

  9. [9]

    Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7 (1):1–26, 1979

    Bradley Efron. Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7 (1):1–26, 1979

    1979

  10. [10]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. InProceedings of the 36th International Conference on Machine Learning (ICML), pages 2242–2251, 2019. arXiv:1904.02868

    Pith/arXiv arXiv 2019

  11. [11]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017. arXiv:1706.04599. 12

    Pith/arXiv arXiv 2017

  12. [12]

    AI safety via debate.arXiv preprint arXiv:1805.00899, 2018

    Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate.arXiv preprint arXiv:1805.00899, 2018

    Pith/arXiv arXiv 2018

  13. [13]

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. Towards efficient data valuation based on the shapley value. InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1167–1176, 2019

    2019

  14. [14]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. InProceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (ACL), 2022. arXiv:2107.06499

    Pith/arXiv arXiv 2022

  15. [15]

    From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.arXiv preprint arXiv:2402.00530, 2024

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.arXiv preprint arXiv:2402.00530, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In International Conference on Learning Representations (ICLR), 2024. arXiv:2312.15685

    Pith/arXiv arXiv 2024

  18. [18]

    G- Eval: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2511–2522,

    2023

  19. [19]

    #InsTag: Instruction tagging for analyzing supervised fine-tuning of large language models

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #InsTag: Instruction tagging for analyzing supervised fine-tuning of large language models. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2308.07074

    Pith/arXiv arXiv 2024

  20. [20]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    Pith/arXiv arXiv 2022

  21. [21]

    Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models.arXiv preprint arXiv:2408.02085, 2024

    Yulei Qin, Yuncheng Yang, Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models.arXiv preprint arXiv:2408.02085, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Sentence-BERT: Sentence embeddings using siamese BERT- networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019. arXiv:1908.10084. 13

    Pith/arXiv arXiv 2019

  23. [23]

    Sequential tests of statistical hypotheses.The Annals of Mathematical Statis- tics, 16(2):117–186, 1945

    Abraham Wald. Sequential tests of statistical hypotheses.The Annals of Mathematical Statis- tics, 16(2):117–186, 1945

    1945

  24. [24]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 9440–9450, 2024. arXiv:2305.17926

    Pith/arXiv arXiv 2024

  25. [25]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. InProceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 13484–13508, 2023. arXiv:2212.10560

    Pith/arXiv arXiv 2023

  26. [26]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations (ICLR), 2022. arXiv:2109.01652

    Pith/arXiv arXiv 2022

  27. [27]

    Edwin B. Wilson. Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22(158):209–212, 1927

    1927

  28. [28]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. JudgingLLM-as-a-judgewithMT-benchandchatbotarena. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2306.05685

    Pith/arXiv arXiv 2023

  29. [29]

    LIMA: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.11206. A Hyperparameters and gate configuration Table ...

    Pith/arXiv arXiv 2023