The machine learning platform for developers of large systems

Alexey Naikov; Alexey Shvetsov; Anatoly Oreshkin; Andrey Shevel

REVIEW 4 major objections 4 minor 11 references

The machine learning platform for developers of large systems

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Iterative documentation refinement turns a naive local RAG into a reliable development partner for large computing systems.

desk verdict A useful workflow note that overclaims a 'breakthrough'; the self-referential evaluation cannot support the generalization. read the letter →

arxiv 2501.13881 v4 pith:55TLGXX7 submitted 2025-01-23 physics.gen-ph

Alexey Naikov , Anatoly Oreshkin , Alexey Shvetsov , Andrey Shevel This is my paper · ORCID

classification physics.gen-ph

keywords retrievalaugmentedgenerationlargelanguagemodelcomputingsystemdevelopmentdocumentationrefinementhuman-in-the-looptestinglocalmachinelearningdeploymentartificialneuralnetworksmaintenance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a simple 'naive' retrieval-augmented generation (RAG) setup—local documents, an embedding search step, and a large language model—can give reliable answers to the developers and administrators of a large computing system, provided the human team treats wrong answers as bugs in the documentation. The authors report a working loop: enter test questions, judge RAG's answers, refine the system description and prompt when answers fail, and repeat until all questions score at least 7 out of 10. In their case, roughly ten loops improved a network-segment description of about 100 pages, and the description grew by about 30% as gaps were filled. The paper's central proposal is to develop the RAG architecture and the system description symbiotically, using RAG inference as a feedback tool during development and maintenance. If this works, technical teams can keep sensitive data in-house while reducing maintenance burden and improving system reliability.

What carries the argument

The central object is the testing loop, a human-in-the-loop cycle the paper names in its Figure 1: ask a batch of questions, let the RAG pipeline retrieve passages and generate answers, have the development team rate each answer from 0 to 10, and for any answer below 7 investigate what in the description or prompt misled the model, then edit and repeat. RAG itself is defined as retrieval-augmented generation: an embedding model converts both the query and local PDF documents into vectors, a retriever finds the most relevant passages, and a language model generates an answer conditioned on those passages plus a prompt template. The loop turns documentation gaps into concrete failures that the team can fix, which is why the authors call the approach 'symbiotic development' of RAG and the system.

What would settle it

Lock the refined description and prompt after the loop, then have a different set of developers write a fresh batch of questions from the same documentation and score the answers; if many fall below the 7-of-10 threshold despite the original test set passing, the generalization claim is false.

Watch

Extended reading notes

Core claim

The central claim is that RAG quality depends less on the choice of language model or embedding tool and more on the completeness and clarity of the local system description, and that this dependency can be exploited deliberately. The authors show how a naive RAG architecture with a low generation temperature (0.1), an edited prompt, and a refined document set passed a developer-authored test set after an iterative loop of questioning and rewriting. They assert that if the architecture answers 50 to 100 such test questions correctly, it will accurately answer many other questions not included in the set. The proposed breakthrough is to build the system description and the RAG installation in parallel, so documentation and inference continuously improve each other and the resulting description functions as a living digital twin of the system.

Load-bearing premise

The load-bearing premise is that the developers' own ratings of answers to self-authored test questions capture true RAG reliability, so passing the test set guarantees good answers to unseen questions.

Editorial extensions

If this is right

A development team can deploy this loop on a local server with commodity GPUs, keeping proprietary system details off the internet.
Documentation quality becomes an explicit acceptance criterion: a system description is done when the RAG test set passes at a defined score.
New team members and administrators can query the continuously maintained description instead of reading hundreds of pages, lowering the barrier to participation.
RAG-maintained documentation naturally tracks system changes, simplifying patching, upgrades, and post-deployment troubleshooting.
With log-aware agents added later, the same architecture could answer questions about the live state of the system, not just its static description.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: a natural formalization is to treat the test questions as a regression suite, rerunning them after every documentation or system change just as software tests are rerun after code changes.
Inference: the reported metric is self-referential, so the method would be stronger with a disjoint held-out question set written by people who did not participate in the refinement loop.
Inference: the approach probably transfers best to teams that already understand the system deeply, because recognizing a misleading answer as misleading requires domain knowledge the model does not have.
Inference: one testable extension is to measure how the minimal number of pages or loops scales with system complexity, an open question the paper itself flags.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this is an experience report about using a local RAG system to answer questions about a 150-page computing network description, and the authors' practical suggestion—refine the documentation until the RAG answers your test questions—is plausible and probably useful. But the paper's 'breakthrough' framing is not supported by the evidence, which is a single informal case with self-authored questions and self-assessed answers.

What's actually new? Not much scientifically. RAG's dependency on source document quality is known, and the cited surveys and networking papers already apply RAG. The co-development loop is a natural extension of that. The paper does give a concrete recipe: start with a short description, ask ~10 test questions, rate answers 0-10, edit the docs and prompt, repeat until the minimum score is 7/10, and scale up to 50-100 questions for confidence. There's a useful list of parameters (temperature, top_p, num_ctx) and an honest caveat that users must check every answer. The local-data privacy motivation is also legitimate.

The soft spots are structural. The evaluation is a closed loop: the same developers who write the questions rate the answers, with no independent ground truth, no held-out questions, no baseline, and no quantitative measurement. The authors explicitly assume that passing their own questions generalizes to unseen queries—that assumption is load-bearing and untested. The reported improvements could measure alignment with the developers' mental model rather than real retrieval quality. Also, the design conflates documentation editing with team discussion, so the specific contribution of RAG is not isolated. The conclusion's 'breakthrough' claim (symbiotic development streamlining both processes) is an overclaim at this level of evidence.

Who is this for? Developers who want a practical starting point for local RAG on internal documentation. They'll get a reasonable workflow and some gotchas. A researcher looking for evidence on RAG evaluation or documentation quality will find the anecdote too thin.

I would not send this to a serious peer-reviewed venue in its current form. It's a practice note, not a research result. If the authors add an external benchmark, a baseline comparison, and held-out questions, there might be a credible empirical claim. Until then, treat it as a useful blog post and an example of how not to evaluate a RAG system.

Referee Report

4 major / 4 minor

Summary. The paper is a short experience report on using a naive Retrieval-Augmented Generation (RAG) system as an assistant for developers of a computing network segment. The authors describe building a local RAG stack with open-source LLMs and embeddings, preparing approximately ten test questions, iteratively refining the system description and the prompt until the developers themselves rated the answers as acceptable, and then generalizing this into a proposed workflow for 'symbiotic development' of system documentation and RAG. The central conclusion is that this feedback loop reduces maintenance burden and improves reliability. The paper contains no quantitative accuracy measurement, no baseline comparison, and no independent evaluation; its main evidence is a single case study and several screenshots of RAG outputs.

Significance. The proposed workflow is plausible and could be practically useful for development teams that maintain local, sensitive documentation. The authors explicitly state their key generalization assumption, which is an honest disclosure and makes the paper easy to assess; however, honoring that disclosure also requires acknowledging that the evidence is not sufficient to support the 'breakthrough' claim in the Conclusion. The paper would be acceptable as a short, clearly framed case study or workflow proposal, but it does not currently demonstrate reliability or reduced maintenance burden in a measurable way.

major comments (4)

[RAG testing and lessons] The load-bearing assumption appears in the sentence 'It is assumed that if correct answers are provided for these questions, the architecture will be able to accurately answer many other questions not included in the set of test questions.' No held-out question set, no baseline comparison, and no independent ground truth are provided to support this assumption. Because the Conclusion's claims about reliable answers and reduced maintenance burden depend on this premise, please either supply evidence for the generalization or explicitly restrict the claims to the developers' own question set and describe generalization as an open question.
[RAG testing and lessons] The evaluation is self-referential: the same developers who wrote and refined the descriptions and prompts also decide whether each answer is acceptable, and they set the threshold at 7/10. This risks measuring convergence to the developers' own expectations rather than objective retrieval quality. Please acknowledge this circularity and add at least a modest external check, such as having an independent domain expert rate a sample of answers, or blinding raters to whether an answer was generated before or after a documentation refinement.
[RAG testing and lessons] The reported improvement confounds several simultaneous interventions: the wording and completeness of the documentation, the prompt template, and the generation temperature (set to 0.1). The text attributes the gain to 'refining the system description,' but the individual contribution of documentation changes is not isolated. Please either hold the prompt and sampling parameters fixed while varying only the documentation, or rephrase the result as the effect of the entire workflow rather than of documentation refinement alone.
[Conclusion] The first sentence calls the approach a 'breakthrough,' and the conclusion asserts that it 'reduces the long-term burden of system maintenance, enhances system reliability, and lowers the risk of incorrect or incomplete answers during operation.' These claims extrapolate beyond the evidence: the paper reports only subjective scores for roughly ten questions in one network segment with about 150 pages of documentation, and no operational deployment. Please temper the language to match the actual scope of the observations and specify what measurements would be needed to support the broader claims about large technical or scientific systems.

minor comments (4)

[Screenshot examples] There are several typographical errors: 'RAG_embed_model – the na e of model' should read 'name of model,' and the caption of Figure 8 reads 'Screenshot of of the output.' Please correct these and any similar typos (e.g., 'must not be desired with any system' in the Introduction).
[References] Several references are incomplete: entry [9] is a bare URL without author or title, and entries [10] and [11] are web pages without clear publication or access dates. Please provide full bibliographic information for all web resources.
[Screenshot examples] Figures 2-8 are not discussed in the main text. Please add a short paragraph explaining what each screenshot demonstrates, such as how the displayed parameters relate to the described configuration and what the reader should notice about the answer quality.
[Author contributions] The footnote on page 1, which uses the same asterisk as the author affiliation, mixes contribution statements with affiliation markers. It would be clearer to move the contribution statement to an Acknowledgements section and keep only the affiliation in the author footnote.

Circularity Check

2 steps flagged · score 5.0 of 10

The quality improvement and the generalized reliability claim are defined by the developers' own ratings on the same self-authored test questions used to refine the system, so the central evaluation is partly circular.

self definitional [Section 'RAG testing and lessons' (pages 2-4), loop description and 7/10 stopping rule; echoed in Conclusion (page 6)]
"Each time, the developers assessed the quality of the answers based on their own opinions regarding whether they were acceptable. If the answer was deemed acceptable, the next question was sent. If the answer was found unacceptable, the developers would begin discussing what could be improved in the description and the prompt. Refining the system description and addressing other gaps significantly improved inference quality. ... The correction of the project or system description should continue until the minimum score for the answers to all applicable questions is no less than 7 out of 10."

The outcome variable ('quality', 'acceptable', 'correct answer') is the same developers' subjective judgment that drives the refinement loop. The description and prompt are edited until those same raters give scores of at least 7 to those same questions. The reported 'significantly improved inference quality' and the Conclusion's claim that 'the model yields high-quality, relevant information' therefore restate, by construction, convergence to the team's own expectations rather than an independently verified property. No external ground truth, held-out question set, or independent rater is introduced, so the measured improvement is self-referential.
fitted input called prediction [Section 'RAG testing and lessons', paragraph beginning 'To obtain sufficient confidence...' (page 4)]
"It is assumed that if correct answers are provided for these questions, the architecture will be able to accurately answer many other questions not included in the set of test questions. The developers themselves can compose the questions."

This sentence states the paper's predictive claim: reliability on unseen operational queries is inferred from passing the developers' own test questions. Because those same questions were used to drive the description/prompt refinements and the acceptance threshold was chosen by the same team, the 'prediction' is not an independent evaluation but an extrapolation from the fitting set. No held-out questions or external benchmark are reported anywhere in the paper, so the generalization claim reduces to the team's assumption rather than a demonstrated result.

full rationale

The paper contains no equations, no formal derivation, and no load-bearing self-citation; its core is an experience report of a local RAG loop. The circularity that exists is in the evaluation design rather than in a mathematical chain. The acceptance criterion (answers deemed acceptable, minimum score 7/10) is the developers' own subjective judgment, and the same self-authored test questions both drive the description/prompt refinements and serve as evidence that quality significantly improved. Consequently, the conclusion that the model yields high-quality, relevant information is close to a restatement of the stopping rule. The paper is transparent that generalization to unseen questions is an assumption rather than a measured result, but that assumption is load-bearing for the claimed reduction of maintenance burden and system reliability. No external benchmark, held-out set, or independent rater is provided. This is a genuine partial circularity: the reported improvement is measured on the fitting set, and the prediction for other questions is assumed, not derived. However, the paper also notes 'Please check any answer from RAG!', which mitigates but does not cure the self-referential evaluation, and the main contribution is a workflow suggestion rather than a quantitative scientific claim, so the score is moderate rather than maximal.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unmeasured evaluation loop: the same developers wrote the documentation, wrote the test questions, judged the answers, and set the stopping threshold. No external ground truth or held-out questions are reported. The free parameters listed are hand-chosen settings from the case study, not systematically optimized or validated.

free parameters (4)

Generation temperature = 0.1
Set by hand to reduce hallucinations; not systematically varied across the reported tests.
Minimum acceptable answer score = 7/10
Chosen by the team as the stopping rule for the refinement loop, with no external basis.
Test question count = 50 to 100
Recommended as sufficient for confidence in RAG reliability, but not derived from statistical power analysis.
Number of refinement loops = approximately 10
Observed in one test and presented as typical, without justification for other systems.

assumptions (4)

domain assumption Developers' subjective ratings of RAG answers are a valid measure of answer correctness.
Used throughout 'RAG testing and lessons' as the acceptance criterion: 'the developers assessed the quality of the answers based on their own opinions'.
ad hoc to paper Correct answers to 50-100 self-prepared test questions generalize to all questions users will ask.
Explicitly assumed in 'RAG testing and lessons': 'It is assumed that if correct answers are provided for these questions, the architecture will be able to accurately answer many other questions not included in the set of test questions.'
ad hoc to paper Answer failures are caused primarily by incomplete or unclear descriptions, not by the embedding model, LLM, prompt, or retrieval settings.
The authors attribute initial poor answers to description gaps and respond by editing the description; temperature and prompt are only minimally adjusted.
ad hoc to paper A single case study on one computing network segment with roughly 150 pages of documentation supports conclusions about large systems generally.
The conclusion generalizes from one deployment to 'any large system development' without comparative evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The machine learning platform for developers of large systems." pith.science (2026). https://pith.science/paper/55TLGXX7

@misc{pith2026250113881,
  author       = {Pith},
  title        = {Pith review of: The machine learning platform for developers of large systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55TLGXX7}},
  note         = {Machine review of arXiv:2501.13881}
}

read the original abstract

The machine learning system in the form of Retrieval Augmented Generation (RAG) has developed steadily since about 2021. RAG could be observed as a version of the knowledge transfer. In the studied case, the large computing systems are observed as the application point of RAG, which includes large language model (LLM), as a partner for the developing team. Such an approach has advantages during the development process and further in exploitation time.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 6 canonical work pages

[1]

Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu and Zhaofeng Liu // Evaluation of Retrieval-Augmented Generation: A Survey // arXiv:2405.07437 [cs.CL] (or arXiv:2405.07437v2 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2405.07437

arXiv doi:10.48550/arxiv.2405.07437
[2]

Wenqi Fan et al // A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models // arXiv:2405.06211 [cs.CL] (or arXiv:2405.06211v3 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2405.06211

arXiv doi:10.48550/arxiv.2405.06211
[3]

Amar Abane, Abdella Battou, Mheni Merzouki // An Adaptable AI Assistant for Network Management // NOMS 2024-2024 IEEE Network Operations and Management Symposium | 979-8-3503-2793-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/NOMS59830.2024.10574957

arXiv 2024
[4]

Ruichen Zhang et al // Interactive AI with Retrieval-Augmented Generation for Next Generation Networking // arXiv:2401.11391 [cs.NI] (or arXiv:2401.11391v1 [cs.NI] for this version) https://doi.org/10.48550/arXiv.2401.11391

work page Pith review arXiv doi:10.48550/arxiv.2401.11391
[5]

Hugo Touvron et al // LLaMA: Open and Efficient Foundation Language Models // https://doi.org/10.48550/arXiv.2302.13971

arXiv doi:10.48550/arxiv.2302.13971
[6]

Jiang et al // Mistral 7B // https://doi.org/10.48550/arXiv.2310.06825

Albert Q. Jiang et al // Mistral 7B // https://doi.org/10.48550/arXiv.2310.06825

arXiv doi:10.48550/arxiv.2310.06825
[7]

Ollama https://ollama.com [last visited 2024-10-28]

work page 2024
[8]

// https://github.com/streamlit // [last visited 2024-10-28]

A faster way to build and share data apps. // https://github.com/streamlit // [last visited 2024-10-28]. 7

work page 2024

Show all 11 references

[9]

https://gitlab.com/rahasak-labs/iollama [visited 2025-01-23]

2025
[10]

GPT that Reviews Your Research Proposal https://peerrecognized.com/gpt-reviews-your-research-proposal/ [visited 2024-12-07]

2024
[11]

NotebookLM https://notebooklm.google.com [visited 2024-12-07] The screenshot examples from RAG output Figure

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu and Zhaofeng Liu // Evaluation of Retrieval-Augmented Generation: A Survey // arXiv:2405.07437 [cs.CL] (or arXiv:2405.07437v2 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2405.07437

arXiv doi:10.48550/arxiv.2405.07437

[2] [2]

Wenqi Fan et al // A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models // arXiv:2405.06211 [cs.CL] (or arXiv:2405.06211v3 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2405.06211

arXiv doi:10.48550/arxiv.2405.06211

[3] [3]

Amar Abane, Abdella Battou, Mheni Merzouki // An Adaptable AI Assistant for Network Management // NOMS 2024-2024 IEEE Network Operations and Management Symposium | 979-8-3503-2793-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/NOMS59830.2024.10574957

arXiv 2024

[4] [4]

Ruichen Zhang et al // Interactive AI with Retrieval-Augmented Generation for Next Generation Networking // arXiv:2401.11391 [cs.NI] (or arXiv:2401.11391v1 [cs.NI] for this version) https://doi.org/10.48550/arXiv.2401.11391

work page Pith review arXiv doi:10.48550/arxiv.2401.11391

[5] [5]

Hugo Touvron et al // LLaMA: Open and Efficient Foundation Language Models // https://doi.org/10.48550/arXiv.2302.13971

arXiv doi:10.48550/arxiv.2302.13971

[6] [6]

Jiang et al // Mistral 7B // https://doi.org/10.48550/arXiv.2310.06825

Albert Q. Jiang et al // Mistral 7B // https://doi.org/10.48550/arXiv.2310.06825

arXiv doi:10.48550/arxiv.2310.06825

[7] [7]

Ollama https://ollama.com [last visited 2024-10-28]

work page 2024

[8] [8]

// https://github.com/streamlit // [last visited 2024-10-28]

A faster way to build and share data apps. // https://github.com/streamlit // [last visited 2024-10-28]. 7

work page 2024