Pith. sign in

REVIEW 3 major objections 4 minor 41 references

Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bottleneck-first trace compression preserves diagnosis

desk verdict A well-scoped systems paper whose central mechanism holds up; the main caveat is that Eq. (4) renders 'direct target truth' through source-built ECDFs, an unproven transfer assumption that should be tested or removed in revision. read the letter →

arxiv 2608.00423 v1 pith:5JO6LGHJ submitted 2026-08-01 cs.AI

Liming Liu , Chao Hu , Mingfei Lu , Cong Tan , Yiwei Ge , Chijin Zhou , Yongjun Xie , Runzhe Wang
show 2 more authors
Xiaohai Shi Heyuan Shi
This is my paper · ORCID
classification cs.AI
keywords LLMservingtracereductionbottleneckdiagnosisworkloadselectionreplaysuitecoverageprioritydirectverificationtwo-witnessgate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes BPW, a framework for choosing a small set of workloads to replay when testing LLM serving systems, so that every major bottleneck component—scheduler, prefill, decode, and KV-cache—has at least two direct, verified witnesses. It argues that representativeness-based trace reduction is insufficient because bottleneck-revealing workloads can be rare and evidence for one component cannot replace missing evidence for another. Instead, BPW nominates candidates from source-side measurements, orders them to prioritize weak or uncovered components, and derives bottleneck labels exclusively from direct target-system measurements. On three public traces, the earliest prefix satisfying the two-witness gate uses few workloads, and BPW reports the best diagnostic quality among 16 policies. This matters because it makes diagnosis-before-compression feasible for production-scale serving evaluation.

What carries the argument

The central mechanism is the three-stage pipeline: response-blind pressure coordinates (log concurrency, log input tokens, log output tokens, and a prefix-reuse-adjusted KV pressure term) feed an Extra-Trees classifier trained only on source data to nominate multi-component bottleneck hyperedges; a coverage-priority score orders workloads by weakest-component gain, uncovered-component gain, and reusable multi-component gain; and direct target verification applies source-built empirical CDFs to target responses relative to a target anchor, producing prediction-independent bottleneck labels. The load-bearing identity is the anchor-relative ECDF gap, where a target workload is labeled a bottleneck witness when its ECDF-transformed response exceeds the target anchor's by at least a pre-specified threshold under at least one serving configuration. This identity couples the source calibration to the target measurement scale, and the two-witness gate requires two distinct positive witnesses per component per restart.

What would settle it

A concrete check is to run BPW's selected suite on a target system whose GPU or batching configuration differs substantially from the source (for example, a different hardware generation or model), then compare the labels produced by the anchor-relative ECDF rule against an exhaustive label derived from direct latency or throughput degradation relative to the same anchor. If the ECDF-gap labels disagree with the direct degradation ranking for a non-trivial fraction of workloads, the source-calibration assumption fails and the verified gate is not actually verified.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that by separating candidate proposal from truth verification, a compact replay suite can satisfy a non-compensatory diagnostic gate for all four bottleneck dimensions. The paper defines the gate as requiring at least two distinct positively verified workloads per component, stable across restarts, plus thresholds on Macro-F1 and worst-dimension recall. BPW claims to reach this gate with three to five workloads across BurstGPT, ServeGen, and Mooncake, improving Mean prefix Macro-F1 by 2.3 percent and WBRC-AUC by 16.3 percent relative to the strongest of 16 reference policies. The key separation is that source predictions order workloads but never define target labels, so the evaluation is not circular.

Load-bearing premise

The load-bearing premise is that the empirical CDFs built from source-system responses remain a valid calibration for the target system's response scale, so that an anchor-relative gap in ECDF units means the same thing on the target as it did on the source.

Editorial extensions

If this is right

  • Replay suites can be built from closed source history without fitting on target responses, so target labels remain prediction-independent and the evaluation avoids circularity.
  • Weakest-component prioritization prevents strong evidence in one dimension, such as the scheduler, from hiding missing evidence in another, such as KV-cache.
  • A suite that satisfies the two-witness gate remains robust to deleting any single positive witness, since at least one verified witness per component survives.
  • The three-stage separation of nomination, ordering, and verification applies to any diagnostic gate with non-compensatory evidence requirements, not only LLM serving.
  • Stage-resolved analysis shows source proposals carry high component Macro-F1 yet still mislabel a substantial fraction of candidate incidences, so direct verification is not redundant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The source-ECDF calibration could be made self-checking by adding a small calibration set of target responses whose true bottleneck status is known from exhaustive profiling, allowing the gap threshold to be tuned per target and removing the strongest assumption.
  • The hyperedge-and-weakest-component machinery likely transfers to other multi-component testing settings, such as microservice latency diagnosis or multi-tenant resource contention, where evidence for one failure mode cannot substitute for another.
  • A testable extension would replace Extra-Trees nomination with a ranker trained directly to maximize verified-gate recall, which may reduce the number of executions needed even further.
  • The WBRC-AUC metric, which integrates worst-component recall over cost, could become a standard for evaluating suite-selection policies beyond LLM serving because it penalizes policies that chase aggregate accuracy at the expense of a single weak component.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Bottleneck-Preserving Witnessing (BPW), a three-stage framework for selecting compact replay suites from LLM serving traces while preserving diagnostic evidence for four bottleneck components (scheduler, prefill, decode, KV-cache). Stage 1 uses response-blind pressure features and source-side measurements to train classifiers that nominate candidate bottleneck workloads; Stage 2 orders candidates using a coverage-priority score that emphasizes the weakest and least-covered components; Stage 3 verifies bottleneck labels on the target system and returns the earliest prefix satisfying a non-compensatory two-witness quality gate. Experiments on BurstGPT, ServeGen, and Mooncake compare BPW against 16 baselines and report that BPW reaches the gate with fewer workloads and higher Mean prefix Macro-F1 and WBRC-AUC. Ablations and sensitivity analyses are also presented.

Significance. If the framework's claims hold, the idea of shifting trace reduction from distribution preservation to evidence preservation is valuable for LLM-serving testing, particularly the explicit rejection of compensatory evidence across bottleneck components. The paper includes useful ablations (stage-resolved and sensitivity), public code, and an unusual degree of transparency about candidate composition, tie-breaking, and stopping rules. However, the central claim of prediction-independent target truth is weakened by the definition of target labels, which depend on source-side ECDFs. The empirical comparison also lacks variance information, leaving small reported improvements hard to assess. The non-compensatory gate and the separation of source-side proposal from target-side verification remain conceptually sound and worth publishing after the label-definition issue is addressed.

major comments (3)
  1. [§4, Bottleneck Truth Verification, Eq. (4)] Equation (4) defines the target bottleneck indicator using z_tar,h_ikj = F_kj(y_tar,h_ikj), where F_kj is an ECDF constructed from source responses. This means target truth is not derived solely from direct target-system measurements, contradicting the abstract and the statement in §4 that source measurements 'do not define target truth.' If source and target response scales differ, all target labels—and therefore the feasibility of the two-witness gate, WBRC-AUC, and every Table 2 comparison—can change without any change in the target system. The authors should either construct F_kj from target-only responses (e.g., target ECDFs or a held-out target calibration) or explicitly treat source-ECDF transfer as an assumption and validate it with a scale-shift or distribution-shift analysis.
  2. [§5, Table 2] Table 2 reports only point estimates, with no variance, confidence intervals, or significance tests, despite the fact that each cell is based on only 24 deterministic requests and three restarts. The reported gains over the strongest baseline are small on ServeGen (0.0139 in Mean prefix Macro-F1) and Mooncake (0.0185), so without a measure of variability the claim that BPW 'outperforms 16 policies' is not empirically supported. The authors should report standard deviations, confidence intervals, or a paired test across the three restarts (or justify why deterministic execution makes variability negligible).
  3. [§4 and §5, source/target setup] The manuscript does not clearly establish that the source and target are sufficiently different to test the transfer of F_kj. The experiments use the same hardware, model, and vLLM version, and it is not explained how the 'source workload pool' and 'target workload pool' are disjoint or how distribution shift arises. Since the load-bearing assumption is that source ECDFs remain valid calibrations for target responses, the paper should include at least one scenario with a clear source/target shift (e.g., different GPU, different model, or a synthetic scaling of target responses) to demonstrate that the gate and the headline comparisons are robust to such shifts.
minor comments (4)
  1. [§5, Parameters and Environment] The phrase 'formal GPU is capped' appears to be a typo; it should be 'GPU is capped'.
  2. [§5, Table 2] The table would be easier to interpret if it separated the 'Length (control)' row from the other baselines, since its WBRC-AUC values are strikingly low (e.g., 0.0281 on BurstGPT) and the paper does not explain why this control behaves so differently.
  3. [§5, Ablation Study, Figure 4] The caption says 'Values are restart means,' but the figure does not show the underlying variance; adding error bars or per-restart points would strengthen the ablation claim.
  4. [General] The code repository URL (https://github.com/llmllmllm/BPW) should be checked for availability before publication; a placeholder URL would undermine the reproducibility claim.

Circularity Check

1 steps flagged · score 3.0 of 10

Target 'truth' is source-calibrated: Eq. (4) reuses source ECDFs to define target labels, so the claimed prediction-independent verification is coupled to source by construction.

  1. self definitional [Section 4.3 (Bottleneck Truth Verification), Eq. (4); cf. Section 4 intro: 'Source measurements ... do not define target truth.']
    "Using the source ECDFs constructed before target verification, BPW computes z^{tar,h}_{ikj}=F_{kj}(y^{tar,h}_{ikj}). The direct target bottleneck indicator for workload i, dimension j, and restarth is b^{tar,h}_{ij}=I[max_{k∈K}(z^{tar,h}_{ikj}-z^{tar,h}_{a^{tar}kj})≥δ^{tar}]. (4)"

    The target label is defined by passing direct target responses through F_kj, the ECDF built from source responses, and comparing anchor-relative ECDF gaps. The same F_kj and the same anchor-relative gap rule define the source training labels (d^e,h_ij=2max_k(z^e,h_ikj−z^e,h_asrckj)≥1) used by the proposal classifiers. Therefore target 'truth' is not a function of target measurements alone; it is a source-calibrated re-encoding. The paper's assertion that source measurements 'do not define target truth' is contradicted by Eq. (4), so the 'prediction-independent verification' claim reduces by construction to the transfer assumption that source ECDFs remain valid on target. A source/target scale shift changes the gate and all Table 2 comparisons without changing raw target severity.

full rationale

BPW's own rule disables the most obvious circularity: the proposal classifier's confidence s_ij and predicted hyperedges bL_i never enter Eq. (4), so the evaluation is not literally fitting to target labels. No load-bearing self-citations or imported uniqueness theorems appear. The remaining circularity-like step is definitional: the 'direct target truth' is not target-only because F_kj is a source-side fitted ECDF. Source labels and target labels share both the ECDF transform and the anchor-relative gap rule, so the verification gate is coupled to the source calibration by construction. This makes the paper's 'prediction-independent' claim overstated and exposes the headline numbers to source/target scale shift, but it does not reduce BPW's ordering to target-label fitting; conditional on the transfer assumption, the comparison against 16 policies is still informative. Score 3 reflects one partial, definitional circularity rather than a fully forced result.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim depends on several hand-chosen thresholds and on the assumption that source-side ECDF calibration transfers to the target system. No new physical or computational entities are postulated; the bottleneck witnesses are operational labels defined by the paper's own measurement rule.

free parameters (6)
  • delta_tar (anchor-relative ECDF-gap threshold) = not reported
    Eq. (4); pre-specified before target evaluation but no value is given; controls how conservative target bottleneck labels are.
  • eta_M, eta_R, eta_A (quality gate thresholds) = not reported
    Eqs. (6)-(7); required for Macro-F1, worst-dimension recall, and restart stability; no values reported, which affects which prefix is the earliest feasible one.
  • proposal threshold for predicted bottleneck hyperedge = not reported
    Workload Candidate Nomination section; determines which dimensions enter the predicted set bLi and hence affects ordering; described only as pre-specified.
  • source label threshold d >= 1 with factor 2 = 1 with scale factor 2
    Source label construction uses an arbitrary scaling to a unit decision threshold; this affects source training labels and therefore candidate nomination.
  • prefix-reuse coefficient 0.75 in KV pressure coordinate = 0.75
    v_i = (p_i + o_i)c_i(1 - 0.75 r_i); a hand-chosen weight for prefix reuse in the KV-cache pressure feature.
  • discount function omega(distance) = not specified
    Discounts proposal support for target workloads far from the source support; only monotonicity is specified, and the exact form affects ordering.
assumptions (6)
  • domain assumption Source ECDFs F_kj transfer to target responses as a valid calibration.
    Bottleneck Truth Verification says: using the source ECDFs constructed before target verification, BPW computes z_tar = F_kj(y_tar).
  • domain assumption An anchor-relative ECDF-gap threshold detects actual bottleneck presence.
    Eq. (4) defines a target bottleneck label solely by the maximum positive ECDF gap relative to the target anchor.
  • domain assumption Four pressure coordinates suffice to expose scheduler, prefill, decode, and KV-cache bottlenecks.
    Workload Candidate Nomination uses only concurrency, input tokens, output tokens, and prefix reuse as bottleneck-revealing features.
  • domain assumption Majority agreement across restarts yields stable bottleneck truth.
    Eq. (5) converts restart-specific labels into a single target label using a majority vote over three restarts.
  • domain assumption Neutral anchor workloads exist and are identifiable by lexicographic minimization.
    Eq. (1) assumes the lexicographic minimizer of pressure coordinates is a valid neutral reference for both source and target.
  • domain assumption Greedy weakest-component ordering with diminishing returns is an effective approximation for the quality gate.
    Eqs. (2)-(3) define the selection score as a lexicographic combination of weakest-component gain, uncovered-component gain, and reusable-hyperedge gain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces." pith.science (2026). https://pith.science/paper/5JO6LGHJ

@misc{pith2026260800423,
  author       = {Pith},
  title        = {Pith review of: Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JO6LGHJ}},
  note         = {Machine review of arXiv:2608.00423}
}
read the original abstract

Production LLM serving generates millions of diverse requests, making full-trace replay across serving configurations increasingly expensive. Existing trace reduction methods mainly preserve workload distributions or representative requests, but bottleneck-revealing workloads may be rare and non-representative. Moreover, evidence for one component cannot compensate for missing evidence in another, while using predicted bottlenecks as target truth creates circular evaluation. These limitations make it necessary to preserve evidence for every bottleneck component rather than rely on workload representativeness alone. We propose Bottleneck-Preserving Witnessing (BPW), a quality-constrained framework for compact and diagnostically reliable LLM serving replay suites. BPW first performs Workload Candidate Nomination using response-blind workload features and closed source-side measurements. This stage identifies workloads that may expose scheduler, prefill, decode, or KV-cache bottlenecks. Coverage-Priority Sequence Construction then organizes multi-component proposals as reusable hyperedges and prioritizes weak and uncovered dimensions. Finally, Bottleneck Truth Verification derives prediction-independent labels solely from direct target-system measurements. The verified results determine the earliest prefix satisfying the direct two-witness requirement for every component. Experiments on BurstGPT, ServeGen, and Mooncake show that BPW reaches the verified gate with a compact workload set and outperforms 16 policies, achieving relative improvements of 2.3% and 16.3% in Mean prefix Macro-F1 and WBRC-AUC, respectively. Stage-resolved and sensitivity analyses confirm the distinct contributions and local stability of its three stages. Our code is publicly available at https://github.com/llmllmllm/BPW

Figures

Figures reproduced from arXiv: 2608.00423 by the authors.

Figure 1
Figure 1. Practical costs of exhaustive replay and undetected bottlenecks in LLM serving. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Component-level evidence sufficiency under BPW’s equal budget. A green check indicates at least [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Stage-resolved analysis on the same completed matrices. The upper panels remove component pro [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Stability of gate workload count, selected prefix, and execution pairs under parameter changes, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 39 canonical work pages

  1. [1]

    18th USENIX symposium on operating systems design and implementation , pages=

    Llumnix: Dynamic scheduling for large language model serving , author=. 18th USENIX symposium on operating systems design and implementation , pages=

    work page

  2. [2]

    18th USENIX Symposium on Operating Systems Design and Implementation , year =

    Yinmin Zhong and Shengyu Liu and Junda Chen and Jianbo Hu and Yibo Zhu and Xuanzhe Liu and Xin Jin and Hao Zhang , title =. 18th USENIX Symposium on Operating Systems Design and Implementation , year =

    work page

  3. [3]

    23rd USENIX Conference on File and Storage Technologies , year =

    Ruoyu Qin and Zheming Li and Weiran He and Jialei Cui and Feng Ren and Mingxing Zhang and Yongwei Wu and Weimin Zheng and Xinran Xu , title =. 23rd USENIX Conference on File and Storage Technologies , year =

    work page

  4. [4]

    and Pinto, C

    Aydar, E. and Pinto, C. and Venugopal, S. and Chatzopoulos, D. , title =. Proceedings of the Sixth European Workshop on Machine Learning and Systems , year =

    work page

  5. [5]

    and Ghasemi, M

    Bhargav, J. and Ghasemi, M. and Sundaram, S. , title =. Proceedings of the 6th Annual Learning for Dynamics and Control Conference , pages =

    work page

  6. [6]

    and Li, Z

    Kwon, W. and Li, Z. and Zhuang, S. and Sheng, Y. and Zheng, L. and Yu, C. H. and Gonzalez, J. E. and Zhang, H. and Stoica, I. , title =. Proceedings of the 29th Symposium on Operating Systems Principles , pages =

    work page

  7. [7]

    and Ma, J

    Li, Y. and Ma, J. and Ballesteros, M. and Benajiba, Y. and Horwood, G. , title =. Proceedings of the 42nd International Conference on Machine Learning , year =

    work page

  8. [8]

    and Weber, L

    Maia Polo, F. and Weber, L. and Choshen, L. and Sun, Y. and Xu, G. and Yurochkin, M. , title =. Proceedings of the 41st International Conference on Machine Learning , year =

    work page

Show all 41 references
  1. [9]

    Azure Public Dataset:

  2. [10]

    Sloman, S. J. and Bharti, A. and Martinelli, J. and Kaski, S. , title =. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , pages =

  3. [11]

    and Chen, Y

    Wang, Y. and Chen, Y. and Li, Z. and Kang, X. and Fang, Y. and Zhou, Y. and Zheng, Y. and Tang, Z. and He, X. and Guo, R. and Wang, X. and Wang, Q. and Zhou, A. C. and Chu, X. , title =. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining , year =

  4. [12]

    and Li, X

    Xiang, Y. and Li, X. and Qian, K. and Yu, W. and Zhai, E. and Jin, X. , title =. arXiv preprint arXiv:2505.09999 , year =

    arXiv

  5. [13]

    and Cand

    Zrnic, T. and Cand. Active Statistical Inference , booktitle =

  6. [14]

    23rd USENIX Symposium on Networked Systems Design and Implementation , year =

    Xiang, Yuxing and Li, Xue and Qian, Kun and Zhang, Yan and Yu, Wenyuan and Zhai, Ennan and Jin, Xin and Zhou, Jingren , title =. 23rd USENIX Symposium on Networked Systems Design and Implementation , year =

  7. [15]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics , pages =

    Saranathan, Gayathri and Xu, Cong and Alam, Mahammad Parwez and Kumar, Tarun and Foltin, Martin and Wong, Soon Yee and Bhattacharya, Suparna , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics , pages =

  8. [16]

    18th USENIX Symposium on Operating Systems Design and Implementation , year =

    Amey Agrawal and Nitin Kedia and Ashish Panwar and Jayashree Mohan and Nipun Kwatra and Bhargav Gulavani and Alexey Tumanov and Ramachandran Ramjee , title =. 18th USENIX Symposium on Operating Systems Design and Implementation , year =

  9. [17]

    2025 USENIX Annual Technical Conference , year =

    Jiahao Wang and Jinbo Han and Xingda Wei and Sijie Shen and Dingyan Zhang and Chenguang Fang and Rong Chen and Wenyuan Yu and Haibo Chen , title =. 2025 USENIX Annual Technical Conference , year =

    2025

  10. [18]

    2024 USENIX Annual Technical Conference , year =

    Bin Gao and Zhuomin He and Puru Sharma and Qingxuan Kang and Djordje Jevdjic and Junbo Deng and Xingkun Yang and Zhou Yu and Pengfei Zuo , title =. 2024 USENIX Annual Technical Conference , year =

    2024

  11. [19]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    Perlitz, Yotam and Bandel, Elron and Gera, Ariel and Arviv, Ofir and Ein-Dor, Liat and Shnarch, Eyal and Slonim, Noam and Shmueli-Scheuer, Michal and Choshen, Leshem , title =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa...

    2024

  12. [20]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Kossen, Jannik and Farquhar, Sebastian and Gal, Yarin and Rainforth, Tom , title =. Proceedings of the 38th International Conference on Machine Learning , pages =

  13. [21]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    Li, Guohong and Xiong, Deyi , title =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    2025

  14. [22]

    23rd USENIX Symposium on Networked Systems Design and Implementation , year =

    Chaoyi Ruan and Yinhe Chen and Dongqi Tian and Yandong Shi and Yongji Wu and Jialin Li and Cheng Li , title =. 23rd USENIX Symposium on Networked Systems Design and Implementation , year =

  15. [23]

    20th USENIX Symposium on Operating Systems Design and Implementation , pages=

    \ OpenTela \ : Unifying Decentralized Computing Resources for Heterogeneous \ LLM \ Serving (Operational Systems) , author=. 20th USENIX Symposium on Operating Systems Design and Implementation , pages=

  16. [24]

    20th USENIX Symposium on Operating Systems Design and Implementation , year =

    Zhiqiang Xie and Ziyi Xu and Mark Zhao and Yuwei An and Vikram Sharma Mailthody and Scott Mahlke and Michael Garland and Christos Kozyrakis , title =. 20th USENIX Symposium on Operating Systems Design and Implementation , year =

  17. [25]

    2025 USENIX Annual Technical Conference , year =

    Shiwei Gao and Qing Wang and Shaoxun Zeng and Youyou Lu and Jiwu Shu , title =. 2025 USENIX Annual Technical Conference , year =

    2025

  18. [26]

    and Ung, Megan and Williams, Adina , title =

    Gupta, Vipul and Ross, Candace and Pantoja, David and Passonneau, Rebecca J. and Ung, Megan and Williams, Adina , title =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    2025

  19. [27]

    How to Select Datapoints for Efficient Human Evaluation of

    Zouhar, Vil. How to Select Datapoints for Efficient Human Evaluation of. Transactions of the Association for Computational Linguistics , pages =

  20. [28]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , year =

    Liusie, Adian and Raina, Vatsal and Fathullah, Yassir and Gales, Mark , title =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , year =

    2024

  21. [29]

    Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics , year =

    Vivek, Rajan and Ethayarajh, Kawin and Yang, Diyi and Kiela, Douwe , title =. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics , year =

  22. [30]

    and Tu, Yuheng and Liang, Percy and Li, Bo and Koyejo, Sanmi , title =

    Truong, Sang T. and Tu, Yuheng and Liang, Percy and Li, Bo and Koyejo, Sanmi , title =. Proceedings of the 42nd International Conference on Machine Learning , pages =

  23. [31]

    and Ramjee, Ramachandran and Tumanov, Alexey , title =

    Agrawal, Amey and Kedia, Nitin and Mohan, Jayashree and Panwar, Ashish and Kwatra, Nipun and Gulavani, Bhargav S. and Ramjee, Ramachandran and Tumanov, Alexey , title =. Proceedings of Machine Learning and Systems (MLSys) , year =

  24. [32]

    arXiv preprint arXiv:2407.07000 , year =

    Agrawal, Amey and Agarwal, Anmol and Kedia, Nitin and Mohan, Jayashree and Kundu, Souvik and Kwatra, Nipun and Ramjee, Ramachandran and Tumanov, Alexey , title =. arXiv preprint arXiv:2407.07000 , year =

    arXiv

  25. [33]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Zang, Chengbo and Turkcan, Mehmet Kerem and Zussman, Gil and Kostic, Zoran and Ghaderi, Javad , title =. Proceedings of the 42nd International Conference on Machine Learning , pages =

  26. [34]

    Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , pages =

    Nie, Xinkun and Tian, Xiaoying and Taylor, Jonathan and Zou, James , title =. Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , pages =

  27. [35]

    19th USENIX Symposium on Networked Systems Design and Implementation , pages =

    Weng, Qizhen and Xiao, Wencong and Yu, Yinghao and Wang, Wei and Wang, Cheng and He, Jian and Li, Yong and Zhang, Liping and Lin, Wei and Ding, Yu , title =. 19th USENIX Symposium on Networked Systems Design and Implementation , pages =

  28. [36]

    22nd USENIX Symposium on Networked Systems Design and Implementation , year =

    Khare, Alind and Garg, Dhruv and Kalra, Sukrit and Grandhi, Snigdha and Stoica, Ion and Tumanov, Alexey , title =. 22nd USENIX Symposium on Networked Systems Design and Implementation , year =

  29. [37]

    and Haghgoo, Behzad and Chen, Annie S

    Liu, Evan Z. and Haghgoo, Behzad and Chen, Annie S. and Raghunathan, Aditi and Koh, Pang Wei and Sagawa, Shiori and Liang, Percy and Finn, Chelsea , title =. Proceedings of the 38th International Conference on Machine Learning , pages =

  30. [38]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    Neel, Seth and Roth, Aaron , title =. Proceedings of the 35th International Conference on Machine Learning , pages =

  31. [39]

    2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) , pages =

    Ma, Yiming and Chen, Ting and Li, Ke , title =. 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) , pages =

    2025

  32. [40]

    IEEE Transactions on Software Engineering , year =

    Xia, Yutong and Liao, Liang and Chen, Junjie and Li, Haopeng and Shang, Weiyi , title =. IEEE Transactions on Software Engineering , year =

  33. [41]

    Maia Polo, Felipe and Weber, Lucas and Choshen, Leshem and Sun, Yuekai and Xu, Gongjun and Yurochkin, Mikhail , booktitle=. tiny

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.