Pith. sign in

REVIEW 3 major objections 3 minor 40 references

A validation-gated coding harness driven by an AI code engine can convert continuously arriving Starlink measurements into validated analysis artifacts, discovering 49 of 56 hidden system-risk cases and beating the strongest predefined base

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:59 UTC pith:WNVVYQIB

load-bearing objection A well-built LLM coding harness for Starlink analysis with a clean evaluation protocol, but the headline AP superiority over supervised baselines is a small, unquantified difference; the honest result is parity plus a useful benchmark construction. the 3 major comments →

arxiv 2607.15541 v1 pith:WNVVYQIB submitted 2026-07-17 eess.SY cs.SY

StarCodex: Dynamic Coding Harness for Starlink Measurement Analysis and Experiment Automation

Pengcheng Luo , Zhiming Shao , Bowen Zhang , Genke Yang , Jian Chu This is my paper
classification eess.SY cs.SY
keywords dynamic coding harnessStarlink measurementexperiment automationadaptive bitrate streamingsystem-risk discoverymeasurement-to-artifact workflowLLM code generationbenchmark construction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that an AI coding engine, wrapped in a validation-gated workflow, can take continuously arriving Starlink measurements and convert them into trustworthy analysis artifacts—risk rankings, benchmarks, prediction-risk windows, and replay evidence—without a human writing each analysis script. The authors claim that on real Starlink traces, this harness discovers 49 of the 56 system-risk segments that fall outside pre-defined profile rules, matches or slightly beats the strongest fixed supervised baseline in average precision, and builds a 512-segment benchmark with more system-risk evidence and broader source coverage than the baselines. If true, it would mean the bottleneck in network measurement analysis—hand-written code and expert inspection—can be partially automated while keeping a human-checkable acceptance layer. The paper also shows that the artifacts change downstream ABR replay outcomes, revealing different quality-of-experience risks among controllers.

Core claim

The discovery is that a 'measurement-to-artifact' loop can close the gap between raw LEO satellite data and reusable experiment evidence. StarCodex maintains an analysis state, detects missing analyses as 'gaps', converts each gap into a structured coding task, asks the Codex AI coding engine to generate or repair an executable artifact, and accepts it only after code-level, data-interface, measurement-semantics, and output validation. In experiments on 3,198 real Starlink segments, the harness found 49 of 56 uncovered system-risk cases at a 5% review budget (recall 0.875, average precision 0.703), outperformed the strongest predefined baseline in average precision, and constructed a 512-seg

What carries the argument

The central mechanism is the 'measurement-to-artifact' problem model plus a four-layer validation gate. StarCodex represents each incoming measurement batch, the current analysis state, and missing analyses ('gaps') as formal objects; it turns gaps into structured coding tasks for Codex, and accepts an artifact only when it passes code-level, data-level, measurement-semantics, and output-level checks, with repair feedback routed back to the coder. The artifact memory and state update let accepted artifacts accumulate across batches.

Load-bearing premise

The ground-truth 'system-risk' label set is defined by the authors' chosen thresholds (10 s cumulative rebuffering for BOLA or RobustMPC, and a history-mean prediction above 1.5× subsequent mean capacity with that capacity below 20 Mbps), and every discovery and benchmark result is measured against that constructed label set, so the reported figures would change if those thresholds or the underlying prediction and replay implementations changed.

What would settle it

Re-run the outer-fold discovery with the risk threshold moved to, say, 20 s of rebuffering or a 2× overestimation ratio; if StarCodex no longer matches or exceeds fixed Random Forest in average precision and recall, the reported 49/56 discovery is tied to the original threshold choice. Alternatively, freeze the coding engine to its first candidate output and check whether the validation gate alone preserves the benchmark's risk rate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • New measurement batches can be processed continuously: the harness detects a gap, generates or repairs an analysis script, and accepts it only after four validation layers, so risk discovery does not wait for a manual audit.
  • The 512-segment benchmark built from accepted artifacts carries more system-risk evidence than the baseline sets (147 vs 116 for fixed RF, 133 for Registry+IF) while retaining 55 of 56 uncovered-risk cases, and covers 35 source runs.
  • Downstream ABR replay on the benchmark separates controllers: Pensieve reaches the highest mean bitrate (117.29 Mbps) but shows 11.72% severe-session ratio and 68.51 s worst-5% rebuffering, while BOLA, RobustMPC, and SafeSABR stay under 0.8% severe-session ratio.
  • Prediction-risk artifacts detect overestimation windows with AP 0.291 at a 5% alert budget, comparable to a purpose-trained supervised random forest (AP 0.288), indicating generated scorers can substitute for fitted models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The measurement-to-artifact formulation is generic; porting the harness to other LEO networks or to terrestrial mobile traces would test whether the gap-detection and validation layers transfer.
  • The ablation suggests the coding LLM is not the main source of accuracy—a fixed Random Forest matches the discovery recall—while the validation-and-repair loop, which decides which artifact gets committed, adds the measurable value. The reader should attribute the gain to the harness gate rather than to the coding engine's skill.
  • A cheap test of the mechanism: replace the strongest code model with a weaker one and see if the four-layer validation plus repair feedback keeps artifact quality within the reported range; if it does, the harness, not the model, is the load-bearing component.
  • The 49-of-56 number is conditional on one risk definition; adopting a different rebuffering or overestimation threshold would shift the target set and could change the relative ranking of StarCodex and the fixed baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes StarCodex, a Codex-based dynamic coding harness that converts evolving Starlink measurements into validated analysis artifacts. The workflow detects analysis gaps, builds structured coding tasks, generates or repairs executable artifacts via Codex, and accepts them only after code-, data-, measurement-, and output-level validation. Experiments on 3,198 valid segments from 233 source runs compare StarCodex with supervised, unsupervised, and random baselines for hard-case discovery, benchmark construction, prediction-risk analysis, downstream ABR replay, and a capability ablation. The paper claims StarCodex discovers 49 of 56 uncovered system-risk cases, attains higher average precision than the strongest predefined baseline (AP 0.703 vs. 0.690 for fixed RF), constructs a benchmark with denser and broader system-risk evidence (147 vs. 116 system-risk segments), and reveals prediction risks and QoE-risk differences among ABR controllers. The evaluation uses source-run-grouped cross-fitting and makes the source code publicly available.

Significance. If the headline claims are fully supported, the paper would provide a useful demonstration that an LLM coding agent, embedded in a validation-gated workflow, can automate parts of network measurement analysis and experiment preparation. The strengths are the use of real Starlink measurements, source-run-disjoint data partitioning, a validation gate that goes beyond simple execution, and the availability of source code. The paper also reports bootstrap intervals for some comparisons, which is a positive methodological gesture. However, the central 'higher average precision' claim currently rests on a small, unquantified AP gap, while recall at the operating point is statistically indistinguishable; the robust finding at this stage is parity with strong supervised baselines, with a more defensible advantage in benchmark breadth and density.

major comments (3)
  1. [§V-B, Table VIII] The headline 'higher average precision than the strongest predefined analysis baseline' rests on AP 0.703 vs. 0.690 for fixed RF and 0.689 for the Registry incumbent, a 0.013–0.014 absolute difference. No confidence interval or significance test is reported for this AP difference, although the paper computes source-run bootstrap intervals for Recall@5% ([-0.055,0.054]) and uses a paired bootstrap for prediction-risk AP in Appendix A-D. At the 5% budget, both StarCodex and fixed RF find 49/56 targets and the text states their recall is comparable. Given that Table XII attributes the entire AP improvement to dynamic coding, the main contribution claim requires a quantified interval for the AP difference (e.g., a source-run bootstrap) or a reframing of the claim in terms of benchmark density rather than AP superiority.
  2. [§V-F, Table XII] The capability ablation has the same statistical weakness as the headline result. Full StarCodex improves AP from 0.689 (w/o dynamic coding) to 0.703 and from 0.698 (w/o validation-guided repair) to 0.703; the differences are 0.014 and 0.005, respectively, with no interval or per-fold breakdown. The 'w/o validation-guided repair' row commits dynamic artifacts in only 2 of 4 folds, so the comparison is noisy. Please report bootstrap intervals for these AP differences or per-fold AP values, so the reader can judge whether 'repair improves AP' and 'dynamic coding improves AP' are supported beyond sampling noise.
  3. [§V-A, Table VI and §V-B] The system-risk and overestimation labels are defined by fixed thresholds: more than 10 s of cumulative rebuffering for BOLA or RobustMPC, and a history-mean prediction exceeding 1.5× the subsequent mean capacity while that capacity is below 20 Mbps. All discovery, benchmark, and downstream claims are measured against this label set. No sensitivity analysis is provided for these thresholds, which are free parameters of the evaluation. Since the 'uncovered system-risk' construct is partly author-constructed, the paper should show that the main results (49/56 hits, AP ordering, 147 vs. 116 benchmark density) are stable over a reasonable range of thresholds, e.g., 5–20 s rebuffering, 1.2×–2× overestimation, and 10–30 Mbps capacity bounds. This would also mitigate the concern that the discovery target is defined using the same family of replay and prediction metrics used downstream.
minor comments (3)
  1. [§V-E, Table XI] The downstream ABR replay differences are reported without uncertainty quantification. Since replay is deterministic for a given controller–segment pair, a matched-pair bootstrap over segments could show whether controller-specific shifts (e.g., Pensieve severe-session ratio from 14.26% to 11.72%) are robust; this is not essential but would strengthen the interpretation.
  2. [§IV-F] There are formatting issues such as 'F . Accepted Artifact' with an extra space before the period; please also check the section numbering and the appendix cross-references. The bootstrap interval for prediction-risk AP appears in Appendix A-D rather than a separate Appendix D; please align the in-text references.
  3. [§V-A] The comparison-method description is dense. It would help to state explicitly, for each method, whether it uses the frozen profile labels as inputs or only as a candidate-pool restriction. The current wording is clear for 'Profile-complement random' but less so for the supervised scorers.

Circularity Check

0 steps flagged

No significant circularity: the evaluation uses source-run-disjoint partitions and held-out labels, so the headline claims are not equivalent to their inputs by construction.

full rationale

The paper's derivation chain (measurement batches -> analysis gaps -> structured tasks -> Codex artifacts -> validation -> accepted artifacts -> discovery/benchmark/prediction/replay) is evaluated with source-run-grouped cross-fitting. The 56 'uncovered system-risk' targets are defined by frozen profile rules plus independent failure evidence (BOLA/RobustMPC replay >10 s or prediction overestimation), and the paper states that 'acceptance and outer-test labels remain hidden from Codex' (Appendix A). The discovery AP, recall, and benchmark metrics are computed on outer-test folds, not on the acceptance partition used by the Harness gate, so the reported 0.703 AP is not forced by the gate that requires improvement over the Registry incumbent on a disjoint partition. The profile thresholds are fitted on discovery runs, but they are frozen before evaluation and are not themselves the failure outcome. The downstream ABR replay differences are a selection consequence of building a risk-enriched benchmark, but the paper does not present them as a derivation from first principles; moreover, the Pensieve result moves opposite to the selection signal, indicating independent empirical content. The only self-citations ([12], [17], [18]) are background references and downstream controllers, not load-bearing evidence for the central claim. The lack of a confidence interval on the AP gap and the author-chosen label thresholds are statistical-validity concerns, not circularity. No step in the paper reduces a reported prediction to a fitted input or to a self-citation chain.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The central claim rests on several hand-set thresholds that define the risk labels, on the faithfulness of external ABR controllers and measurement data, and on the assumption that the Codex-based generation and validation process is representative. No new physical entities are introduced. The free parameters are mostly evaluation-construction choices, not physical constants.

free parameters (8)
  • severe rebuffering threshold = 10 s cumulative rebuffering
    Defines the system-risk label for ABR replay; chosen without sensitivity analysis.
  • overestimation ratio = 1.5x
    Positive prediction-overestimation rule: history-mean prediction > 1.5x subsequent mean capacity; directly constructs the prediction-risk label.
  • low-capacity threshold = 20 Mbps
    Pairwise condition for overestimation positive windows; hand-set.
  • segment length = 600 s non-overlapping at 1 Hz
    Segmentation parameter chosen for the valid replay segment corpus.
  • coverage and gap thresholds = throughput coverage ≥95%, latency ≥80%, active throughput ≥95%, max gaps 5 s/15 s
    Data inclusion criteria for valid segments; affect the corpus and all subsequent results.
  • profile thresholds = not reported numerically
    Thresholds for low-tail, high-volatility, high-latency, and handover-heavy profiles are fitted on discovery source runs and frozen; these define the 'uncovered' risk set.
  • benchmark fusion weight = selected on acceptance partition
    Fusion weight for combining discovery score and anomaly evidence in benchmark construction.
  • maximum tasks per batch M and repair rounds R = unspecified
    Algorithm hyperparameters that bound the harness effort but are not reported.
axioms (5)
  • domain assumption The Starlink measurement data from [9] are accurate, representative, and sufficient for the claims.
    All evaluations rest on this external dataset; the paper does not ship the data or validate its provenance beyond the summary table.
  • domain assumption The ABR replay implementations (BOLA, RobustMPC, Pensieve, SafeSABR) faithfully represent real playback behavior.
    The system-risk labels and downstream replay outcomes depend on these controllers being reasonable proxies for actual ABR performance.
  • standard math Source-run-grouped cross-fitting prevents label leakage and the acceptance gate generalizes.
    The evaluation assumes that training on one set of source runs and selecting artifacts on a disjoint acceptance partition yields unbiased performance estimates.
  • domain assumption Codex-generated artifacts are a representative sample of the dynamic coding harness's capability.
    The single proprietary model and version (GPT-5.5 with xhigh reasoning) are treated as the harness; no variance across model versions or seeds is reported.
  • ad hoc to paper The four validation layers (code, data, measurement, output) are sufficient to guarantee that accepted artifacts are meaningful.
    The paper defines validation as these four checks, but does not show that this is sufficient to avoid subtle semantic errors in generated analysis code.

pith-pipeline@v1.3.0-alltime-deepseek · 19278 in / 9306 out tokens · 108132 ms · 2026-08-01T22:59:35.970265+00:00 · methodology

0 comments
read the original abstract

Starlink and other low Earth orbit (LEO) satellite broadband systems are producing increasingly diverse measurement data across regions, time periods, and access conditions. These measurements are valuable for throughput prediction, adaptive bitrate (ABR) evaluation, and network experimentation, but converting continuously arriving data into reusable experimental evidence still relies heavily on manually developed analysis code and expert-guided data inspection and failure-case organization. This paper proposes StarCodex, a dynamic coding harness for Starlink measurement analysis and experiment automation. StarCodex detects analysis gaps from the current measurement state, converts them into structured coding tasks, uses Codex to generate or repair executable analysis artifacts, and accepts artifacts through code, data-interface, measurement-semantics, and output validation. Experiments on real Starlink measurements show that StarCodex discovers 49 of 56 uncovered system-risk cases, attains higher average precision than the strongest predefined analysis baseline, and constructs a benchmark with denser and broader system-risk evidence. The generated prediction and replay artifacts further reveal prediction risks and quality-of-experience (QoE)--risk differences among ABR controllers. These results demonstrate the feasibility of using a dynamic coding harness to convert evolving Starlink measurements into validated analysis artifacts for automated experiment workflows.

Figures

Figures reproduced from arXiv: 2607.15541 by Bowen Zhang, Genke Yang, Jian Chu, Pengcheng Luo, Zhiming Shao.

Figure 1
Figure 1. Figure 1: Starlink measurement-analysis problem. Evolving measurements [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Conceptual overview of StarCodex. features, hard-case benchmarks, figures, and case studies still requires substantial manual scripting, data inspection, debug￾ging, and engineering judgment. As data volume increases and failure modes evolve, a central question emerges: how can changing Starlink measurements be converted into reliable analysis artifacts that support system experiments and conclu￾sions? [P… view at source ↗
Figure 3
Figure 3. Figure 3: Overall workflow of StarCodex. steps. This makes artifact quality the basis for evaluating whether dynamic coding contributes to experiment automa￾tion. IV. STARCODEX FRAMEWORK StarCodex can be described as a workflow from new data to accepted artifacts. The workflow detects analysis gaps, builds structured coding tasks, invokes Codex Dynamic Coding, validates candidate artifacts, returns repair feedback w… view at source ↗
Figure 4
Figure 4. Figure 4: Artifact validation and repair feedback in StarCodex. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Uncovered system-risk recall across review budgets and Harness [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prediction-overestimation recall and exposure coverage across matched [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Same-pipeline capability ablation for discovery and benchmark [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: End-to-end case study of validation-guided artifact repair, benchmark [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 5 linked inside Pith

  1. [1]

    Eu- nomia: A multicontroller domain partitioning framework in hierarchical satellite networks,

    Q. Zhang, K. Qiu, Z. Chen, W. Zhu, X. Xu, P. Du, and Y . Gao, “Eu- nomia: A multicontroller domain partitioning framework in hierarchical satellite networks,”IEEE Journal on Selected Areas in Communications, 2025

    2025

  2. [2]

    A multifaceted look at starlink performance,

    N. Mohan, A. E. Ferguson, H. Cech, R. Bose, P. R. Renatin, M. K. Marina, and J. Ott, “A multifaceted look at starlink performance,” in Proceedings of the ACM Web Conference 2024, 2024, pp. 2723–2734

    2024

  3. [3]

    Llm-guided drl for multi-tier leo satellite networks with hybrid fso/rf links,

    J. Li, G. Sun, Z. Sun, J. Wang, Y . Liu, R. Zhang, D. Niyato, and S. Mao, “Llm-guided drl for multi-tier leo satellite networks with hybrid fso/rf links,”IEEE Journal on Selected Areas in Communications, 2025

    2025

  4. [4]

    A first look at starlink performance,

    F. Michel, M. Trevisan, D. Giordano, and O. Bonaventure, “A first look at starlink performance,” inProceedings of the 22nd ACM Internet Measurement Conference, 2022, pp. 130–136

    2022

  5. [5]

    A browser-side view of starlink connectivity,

    M. M. Kassem, A. Raman, D. Perino, and N. Sastry, “A browser-side view of starlink connectivity,” inProceedings of the 22nd ACM Internet Measurement Conference, 2022, pp. 151–158

    2022

  6. [6]

    Federated learning driven covert communication in satellite terrestrial integrated networks: A privacy-preserving framework,

    M. Wu, K. Guo, Z. Wang, C. Dong, Y . Liu, Q. Wu, and Z. Zheng, “Federated learning driven covert communication in satellite terrestrial integrated networks: A privacy-preserving framework,”IEEE Journal on Selected Areas in Communications, 2025

    2025

  7. [7]

    On an intelligent hierarchical routing strategy for ultra-dense free space optical low earth orbit satellite networks,

    B. Mao, X. Zhou, J. Liu, and N. Kato, “On an intelligent hierarchical routing strategy for ultra-dense free space optical low earth orbit satellite networks,”IEEE Journal on Selected Areas in Communications, vol. 42, no. 5, pp. 1219–1230, 2024

    2024

  8. [8]

    Cooperative federated learning over ground-to-satellite integrated networks: Joint local computation and data offloading,

    D.-J. Han, S. Hosseinalipour, D. J. Love, M. Chiang, and C. G. Brin- ton, “Cooperative federated learning over ground-to-satellite integrated networks: Joint local computation and data offloading,”IEEE Journal on Selected Areas in Communications, vol. 42, no. 5, pp. 1080–1096, 2024

    2024

  9. [9]

    Vivisecting starlink throughput: Measurement and prediction,

    Z. Liu, F.-X. G. Reidys, S. Tanveer, and D. Vasisht, “Vivisecting starlink throughput: Measurement and prediction,”Proceedings of the ACM on Networking, vol. 3, no. CoNEXT4, pp. 1–23, 2025

    2025

  10. [10]

    Lumos: Towards better video streaming QoE through accurate throughput prediction,

    G. Lv, Q. Wu, W. Wang, Z. Li, and G. Xie, “Lumos: Towards better video streaming QoE through accurate throughput prediction,” inIEEE INFOCOM 2022 - IEEE Conference on Computer Communications. IEEE, 2022, pp. 650–659

    2022

  11. [11]

    Accurate throughput prediction for improving QoE in mobile adaptive streaming,

    G. Lv, Q. Wu, Q. Tan, W. Wang, Z. Li, and G. Xie, “Accurate throughput prediction for improving QoE in mobile adaptive streaming,”IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5799–5817, 2024

    2024

  12. [12]

    Risk-aware safe throughput forecasting for starlink networks,

    H. Xie, C. Zhang, P. Luo, Z. Zhang, G. Yang, X. Zhang, and B.-H. Soong, “Risk-aware safe throughput forecasting for starlink networks,” arXiv preprint arXiv:2605.09508, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    A control-theoretic approach for dynamic adaptive video streaming over HTTP,

    X. Yin, A. Jindal, V . Sekar, and B. Sinopoli, “A control-theoretic approach for dynamic adaptive video streaming over HTTP,” inPro- ceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, 2015, pp. 325–338

    2015

  14. [14]

    BOLA: Near-optimal bitrate adaptation for online videos,

    K. Spiteri, R. Urgaonkar, and R. K. Sitaraman, “BOLA: Near-optimal bitrate adaptation for online videos,”IEEE/ACM Transactions on Net- working, vol. 28, no. 4, pp. 1698–1711, 2020

    2020

  15. [15]

    Neural adaptive video stream- ing with pensieve,

    H. Mao, R. Netravali, and M. Alizadeh, “Neural adaptive video stream- ing with pensieve,” inProceedings of the Conference of the ACM Special Interest Group on Data Communication, 2017, pp. 197–210

    2017

  16. [16]

    Comyco: Quality-aware adaptive video streaming via imitation learning,

    T. Huang, C. Zhou, R.-X. Zhang, C. Wu, X. Yao, and L. Sun, “Comyco: Quality-aware adaptive video streaming via imitation learning,” in Proceedings of the 27th ACM International Conference on Multimedia, 2019, pp. 429–437

    2019

  17. [17]

    Sabr: A stable adaptive bitrate framework using behavior cloning pretraining and reinforcement learning fine-tuning,

    P. Luo, Y . Zhao, B. Zhang, G. Yang, B.-H. Soong, and C. Yuen, “Sabr: A stable adaptive bitrate framework using behavior cloning pretraining and reinforcement learning fine-tuning,” in2026 IEEE Wireless Com- munications and Networking Conference (WCNC). IEEE, 2026, pp. 1–6

    2026

  18. [18]

    SafeSABR: Risk-calibrated adaptive bitrate streaming over Starlink networks,

    H. Xie, J. Zhu, Z. Shao, C. Fan, Z. Zhang, G. Yang, and P. Luo, “SafeSABR: Risk-calibrated adaptive bitrate streaming over Starlink networks,”arXiv preprint arXiv:2605.23560, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    OpenAI, “Codex,” https://openai.com/codex/, 2026, accessed: 2026-06- 08

    2026

  20. [20]

    Claude Code Documentation,

    Anthropic, “Claude Code Documentation,” https://docs.anthropic.com/ en/docs/claude-code/getting-started, 2026, accessed: 2026-06-08

    2026

  21. [21]

    Network characteristics of LEO satellite constellations: A starlink-based measure- ment from end users,

    S. Ma, Y . C. Chou, H. Zhao, L. Chen, X. Ma, and J. Liu, “Network characteristics of LEO satellite constellations: A starlink-based measure- ment from end users,” inIEEE INFOCOM 2023 - IEEE Conference on Computer Communications, 2023, pp. 1–10

    2023

  22. [22]

    Multi-timescale evaluation of starlink throughput,

    J. Garcia, S. Sundberg, G. Caso, and A. Brunstrom, “Multi-timescale evaluation of starlink throughput,” inProceedings of the 1st ACM Workshop on LEO Networking and Communication, 2023, pp. 31–36

    2023

  23. [23]

    A global perspective on the past, present, and future of video streaming over starlink,

    L. Izhikevich, R. Enghardt, T.-Y . Huang, and R. Teixeira, “A global perspective on the past, present, and future of video streaming over starlink,”Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 8, no. 3, pp. 1–22, 2024

    2024

  24. [24]

    Genet: Automatic curriculum generation for learning adaptation in networking,

    Z. Xia, Y . Zhou, F. Y . Yan, and J. Jiang, “Genet: Automatic curriculum generation for learning adaptation in networking,” inProceedings of the ACM SIGCOMM 2022 Conference, 2022, pp. 397–413

    2022

  25. [25]

    Realtime multimedia services over starlink: A reality check,

    H. Zhao, H. Fang, F. Wang, and J. Liu, “Realtime multimedia services over starlink: A reality check,” inProceedings of the 33rd Workshop on Network and Operating System Support for Digital Audio and Video, 2023, pp. 43–49

    2023

  26. [26]

    Low-latency live video streaming over a low-earth- orbit satellite network with DASH,

    J. Zhao and J. Pan, “Low-latency live video streaming over a low-earth- orbit satellite network with DASH,” inProceedings of the 15th ACM Multimedia Systems Conference, 2024, pp. 109–120

    2024

  27. [27]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations, 2023

    2023

  28. [28]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” inAdvances in Neural Information Processing Systems, vol. 36, 2023

    2023

  29. [29]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023

    2023

  30. [30]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inInternational Conference on Learning Representations, 2024

    2024

  31. [31]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    2024

  32. [32]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems, vol. 37, 2024

    2024

  33. [33]

    AutoCodeRover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “AutoCodeRover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    2024

  34. [34]

    Agentless: Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1–24, 2025

    2025

  35. [35]

    Data interpreter: An LLM agent for data science,

    S. Hong, Y . Lin, B. Liu, B. Liu, B. Wu, C. Zhang, C. Wei, D. Li, J. Chen, J. Zhanget al., “Data interpreter: An LLM agent for data science,”arXiv preprint arXiv:2402.18679, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    MLAgentBench: Eval- uating language agents on machine learning experimentation,

    Q. Huang, J. V ora, P. Liang, and J. Leskovec, “MLAgentBench: Eval- uating language agents on machine learning experimentation,”arXiv preprint arXiv:2310.03302, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    MLE-bench: Evaluating machine learning agents on machine learning engineering,

    J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Madry, “MLE-bench: Evaluating machine learning agents on machine learning engineering,”arXiv preprint arXiv:2410.07095, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Netllm: Adapting large language models for networking,

    D. Wu, X. Wang, Y . Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “Netllm: Adapting large language models for networking,” inProceed- ings of the ACM SIGCOMM 2024 Conference, 2024, pp. 661–678

    2024

  39. [39]

    LLexus: An AI agent system for incident management,

    P. Las-Casas, A. G. Kumbhare, R. Fonseca, and S. Agarwal, “LLexus: An AI agent system for incident management,”ACM SIGOPS Operating Systems Review, vol. 58, no. 1, pp. 23–36, 2024

    2024

  40. [40]

    LogGPT: Log anomaly detection via GPT,

    X. Han, S. Yuan, and M. Trabelsi, “LogGPT: Log anomaly detection via GPT,” in2023 IEEE International Conference on Big Data (BigData), 2023, pp. 1117–1122. 14 APPENDIXA STARCODEXINTERFACEEXAMPLES This appendix instantiates the StarCodex interfaces with the hard-case discovery task from outer fold 3. The running example follows one profile gap from its str...

    2023