Pith. sign in

REVIEW 3 major objections 4 minor 34 references

AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An AI agent that combines a Devito knowledge graph with retrieval-augmented generation and iterative quality checking can automatically translate legacy Fortran finite-difference code into Devito, with a claimed 76.9% Grade-A success rate a

desk verdict A plausible engineering demo with self-referential evaluation; the retrieval numbers don't internally square, but the failure-mode catalog and three-level query idea are worth engaging. read the letter →

arxiv 2601.18381 v2 pith:KZT5EP4B submitted 2026-01-26 cs.AI cs.SE

Yinghan Hou , Zongyou Yang This is my paper
classification cs.AIcs.SE
keywords AIagentRetrieval-AugmentedGenerationGraphRAGknowledgeFortranDevitofinitedifferencecodetranslation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a purpose-built AI agent can reliably modernize legacy Fortran finite-difference codes by translating them into Devito, a Python domain-specific language for stencil and finite-difference computation. The agent is not a plain LLM call: it builds a knowledge graph from Devito's documentation and codebase, retrieves relevant patterns through a multi-mode GraphRAG pipeline driven by static analysis of the Fortran source, and iterates through a graph-based orchestration loop until its own validation criteria are satisfied. The authors report a 76.9% Grade-A conversion rate across 13 test cases with perfect execution, structural, and API-compliance scores, and retrieval metrics of Precision@5=0.964, Recall@5=0.930, and MRR=1.000. If the claims hold, retired Fortran scientific codes—common in weather, climate, and geophysics—could be migrated to a modern high-performance framework with far less manual effort.

What carries the argument

The load-bearing mechanism is a graph-guided retrieval-and-refinement loop: a graph database containing Devito's documentation and code (organized into 70 semantic communities via community detection) supplies retrieval context; a four-mode parallel RAG pipeline (full-text search, exact community match, concept expansion via entity links, and semantic similarity traversal) fuses and re-ranks results; typed output schemas force the LLM to emit structured conversions; and a graph-based orchestration state machine scores each output by blending static checks with a separate LLM judge's grade, then routes it to accept, refine, or reconvert based on quality thresholds (8.5 / 5.5 / 3).

What would settle it

Take the same pipeline and run it on a held-out suite of, say, 50 legacy Fortran finite-difference programs from other sources; have human experts (not an LLM) judge the generated Devito code for execution, structural correctness, and mathematical equivalence, and then check whether the Grade-A rate and retrieval scores approach the claimed 76.9%, 0.964, and 0.930.

Watch

Extended reading notes

Core claim

The central discovery claim is that an agentic pipeline can convert legacy Fortran finite-difference code into Devito with a Grade-A rate of 76.9% across 13 test cases, and perfect 1.00 scores on execution, structural integrity, and API compliance in every case. The pipeline builds a graph of Devito's documentation and code (12,793 nodes, 62,362 relationships, 70 semantic communities), retrieves relevant knowledge through four parallel modes (full-text, community, concept-expansion, and semantic-similarity), then iterates through a graph-based orchestration state machine that grades each candidate via static checks and a separate LLM judge, routing low-scoring outputs back for refinement. Th

Load-bearing premise

The load-bearing premise is that the author-constructed ground-truth relevance sets and the LLM-as-judge scores actually measure real-world retrieval and translation quality; if those sets are incomplete or biased, the reported metrics (Precision@5 0.964, Recall@5 0.930, MRR 1.000, and a 76.9% Grade-A rate) would overstate how well the system performs on Fortran code it has not seen.

Editorial extensions

If this is right

  • A version-locked knowledge base plus denylist/allowlist rules eliminates the API hallucinations (e.g., u.dx.backward) that plague vanilla LLM code generation, so the generated code is far more likely to run as-is.
  • With retrieval responses in ~12 ms and throughput of 131 files/hour at four concurrent agents, the pipeline supports batch modernization of large legacy codebases, not just one-off conversions.
  • The quality loop's threshold routing (pass above 8.5, refine between 5.5 and 8.5, reconvert below 5.5) gives a practical way to decide when a conversion is ready without human review, but the 23.1% B-grade cases indicate that human oversight is still needed for a meaningful minority.
  • The combination of static five-dimensional checks and a separate LLM judge reduces self-enhancement bias, and the reported correlation between the two scoring methods supports the claim that the evaluation is reliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The MRR of 1.000 is an artifact of the evaluation design, as the authors themselves note: with ground-truth answer sets defined from the same knowledge base, the first result always lands in the set. Real-world retrieval on unlabeled code would almost certainly show lower precision and recall.
  • If the architecture generalizes beyond Devito, the same pattern—curate a target-DSL knowledge graph, reverse-engineer the legacy language, retrieve with GraphRAG, validate with an LLM judge—could apply to other migration pairs such as Fortran to Firedrake or CUDA. But each new target requires a fresh curated knowledge base, so the cost of adoption is nontrivial.
  • The 'conversion fidelity' scores in Table III (0.60–0.96) are the weakest and most subjective dimension; independent human inspection of the B-grade cases would be the quickest way to see whether the remaining failures are mathematical or merely stylistic.
  • Because the parallel efficiency of 143.7% exceeds 100%, the 5.75× speedup is likely the product of I/O overlap rather than genuine compute scaling; a controlled benchmark with identical task batches and no external API latency would separate these effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes a LangGraph-based AI agent that reverse-engineers legacy Fortran finite-difference code and translates it to the Devito DSL. The system constructs a knowledge graph from Devito documentation and code, applies GraphRAG with multi-stage retrieval (community, full-text, concept expansion, semantic similarity), and uses an LLM (via a Pydantic-constrained schema) to generate Devito code. A validation framework combines static checks with G-Eval LLM scoring. The paper reports retrieval metrics (Precision@5=0.964, Recall@5=0.930, MRR=1.000) and a 76.9% Grade-A success rate across 13 translation test cases.

Significance. If the reported performance were reliable, the system would be a meaningful step toward automating legacy scientific code modernization. The architectural integration of GraphRAG, static analysis, and structured LLM output is plausible and the paper gives a reasonably detailed design. However, the evaluation is not yet convincing: the retrieval metrics are internally inconsistent, the Grade-A percentage disagrees with the table, and the ground truth is partly self-constructed and LLM-judged. The lack of released code, data, or a human-validated benchmark limits reproducibility. The contribution is potentially useful but the current evidence does not support the headline claims.

major comments (3)
  1. [III.A.2 vs III.A.3, Tables I and II] The retrieval evaluation contains a direct numerical contradiction. Table I reports Recall@5=0.930 for the GraphRAG system over 11 queries. Section III.A.3 states that each strategy processed 'an identical set of ten benchmark queries' and Table II reports the Comprehensive strategy—the full pipeline—with Recall@5=0.668. Since the Comprehensive strategy is the GraphRAG system described in Table I, a discrepancy of 0.262 (about 28%) is unexplained. The paper needs to reconcile these numbers and clarify the relationship between the 11-query and 10-query evaluation sets.
  2. [Section III.B, Table III] The abstract and Section III.B claim a Grade-A success rate of 76.9%. However, Table III lists 13 cases: 11 are graded A and 2 are B. 11/13 = 84.6%, not 76.9%. The 76.9% corresponds to 10/13. This is a load-bearing numerical inconsistency in the principal translation-quality claim. The authors must correct the text or the table and explain which 10 cases are considered Grade A.
  3. [Section III.A.1 and Eq. (2)] The retrieval and translation evaluations are partially circular. The ground-truth relevance sets in III.A.1 are authored from the same Devito knowledge base under test, so Precision/Recall primarily measure internal consistency rather than external utility. Similarly, the final score in Eq. (2) blends a static score with GPT OSS 120B as the LLM judge, a model from the same open-source ecosystem as the generation model. The paper acknowledges the MRR limitation but not the broader circularity. An external human-validated benchmark is needed to support the claimed 'high accuracy and real-time responsiveness.'
minor comments (4)
  1. [Section II.B.5] The claim that graph sparsification 'enhances query performance by a factor of 540' is undefined. Please specify the baseline, the measurement, and how the factor was computed.
  2. [Section II.D.4] The 143.7% parallel efficiency needs clarification. I/O overlap can plausibly give superlinear speedup over a sequential baseline, but the text should define the theoretical bound and the exact measurement methodology.
  3. [Section III.A.3, Table II] The 'Deep' strategy yields identical Precision@5 and Recall@5 as the 'Comprehensive' strategy. This is suspicious and may be a copy-paste error; please verify and report distinct values if they exist.
  4. [Section III.A.2] The MRR=1.000 is admitted to be an artifact of set-based ground truth. Since MRR is not discriminative here, consider reporting a set-based metric such as nDCG@5 or Bpref.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain found: the claimed retrieval and translation metrics are empirical evaluations, not results forced by construction or by self-citation.

full rationale

Walking the paper's claimed derivation chain, no step reduces a headline result to its own inputs. The pipeline is empirical: knowledge-base construction (Section II.B), retrieval evaluation (Section III.A), and translation validation (Section III.B.2). The retrieval ground-truth references are author-defined (III.A.1: 'Each query is paired with predefined ground truth references...'), and the paper itself concedes that MRR is inflated ('The reported MRR values “look too good”... When the top-ranked retrieval result belongs to the predefined ground truth set, MRR=1'). This is an acknowledged limitation of the metric under multi-relevant ground-truth sets, not an equation-by-construction: Precision@5, Recall@5, and MRR are computed against those reference sets, not derived from them. The translation grades (Eqs. 1-2) combine static dimensions with G-Eval; the paper explicitly states 'a model distinct from the one used for code generation was applied,' and execution success, structural integrity, and API compliance are objectively checkable. The 'RL-motivated' feedback is an overclaim: Section IV.A admits 'fixed, predefined thresholds, such as excellent_threshold=0.85 and acceptable_threshold=0.55.' That is a naming/overclaim issue, not circularity. The Table I (Recall@5=0.930) versus Table II (Comprehensive Recall@5=0.668) discrepancy is a serious internal-validity/correctness problem, but it is not a circular reduction: it does not show that any metric is equivalent to its inputs by construction. No self-citations are load-bearing; the references are external. Therefore the circularity score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The system is an engineering composition; the headlined 'RL-motivated' adaptive behavior is deterministic threshold routing, so the main burden is the unverified evaluation scaffolding plus hand-set hyperparameters.

free parameters (8)
  • excellent_threshold = 0.85
    Hand-set routing threshold in Section III.E; outputs >=0.85 bypass refinement. Used in the 'adaptive' claim but not learned.
  • acceptable_threshold = 0.55
    Hand-set threshold; outputs 0.55-0.85 trigger refinement.
  • minimum_quality_threshold = 3.0
    Lowest permitted quality boundary before reconversion.
  • retrieval_fusion_weights = full-text 1.0, community 0.9, concept 0.8, similarity 0.7; query weights 1.0/0.7/0.5
    Hand-chosen weights in Section II.C.3; directly shape retrieval rankings and thus the reported Precision/Recall.
  • cosine_similarity_threshold = 0.6
    Semantic-similarity graph traversal threshold in Section II.C.2.
  • top_k_neighbors = 8
    Graph sparsification keeps only 8 most similar neighbors per node; central to the 540x speedup claim.
  • leiden_resolutions = 0.3, 0.5, 0.8, 1.0, 1.2
    Community detection resolutions evaluated in Section II.B.5; final choice not specified.
  • lambda_blend = 0.5
    Eq. 2 blends traditional and LLM scores equally; hand-set.
assumptions (4)
  • domain assumption Devito repository examples/tutorials are correct and sufficient for the translation task
    Section II.B.1 builds the entire knowledge base from the Devito repo; the paper itself notes early-version examples caused hallucinated APIs, so correctness of the corpus is load-bearing.
  • domain assumption Static Fortran analysis reliably extracts query features
    Section II.C.1 assumes FortranCodeAnalyzer can identify PDE type, dimensions, boundary conditions, and time-stepping from source alone; no accuracy for this analyzer is reported.
  • domain assumption G-Eval/LLM-as-judge scores correlate with true code quality
    Section III.B uses GPT OSS 120B as judge with lambda=0.5 blending; no human validation of the judge is provided.
  • ad hoc to paper Ground-truth relevance sets used for retrieval metrics are complete and unbiased
    Section III.A.1: authors define expected topics/keywords/output types for 11 queries; the paper admits MRR=1 follows from set-based ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito." pith.science (2026). https://pith.science/paper/KZT5EP4B

@misc{pith2026260118381,
  author       = {Pith},
  title        = {Pith review of: AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZT5EP4B}},
  note         = {Machine review of arXiv:2601.18381}
}
read the original abstract

To facilitate the transformation of legacy finite difference implementations into the Devito environment, this study develops an integrated AI agent framework. Retrieval-Augmented Generation (RAG) and open-source Large Language Models are combined through multi-stage iterative workflows in the system's hybrid LangGraph architecture. The agent constructs an extensive Devito knowledge graph through document parsing, structure-aware segmentation, extraction of entity relationships, and Leiden-based community detection. GraphRAG optimisation enhances query performance across semantic communities that include seismic wave simulation, computational fluid dynamics, and performance tuning libraries. A reverse engineering component derives three-level query strategies for RAG retrieval through static analysis of Fortran source code. To deliver precise contextual information for language model guidance, the multi-stage retrieval pipeline performs parallel searching, concept expansion, community-scale retrieval, and semantic similarity analysis. Code synthesis is governed by Pydantic-based constraints to guarantee structured outputs and reliability. A comprehensive validation framework integrates conventional static analysis with the G-Eval approach, covering execution correctness, structural soundness, mathematical consistency, and API compliance. The overall agent workflow is implemented on the LangGraph framework and adopts concurrent processing to support quality-based iterative refinement and state-aware dynamic routing. The principal contribution lies in the incorporation of feedback mechanisms motivated by reinforcement learning, enabling a transition from static code translation toward dynamic and adaptive analytical behavior.

Figures

Figures reproduced from arXiv: 2601.18381 by the authors.

Figure 1
Figure 1. LangChain RAG from scratch pipeline overview. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. System architecture for Fortran-to-Devito pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Devito knowledge base tected, and complex structures including tables, lists, and embedded code blocks are systematically parsed. Each resulting knowledge block is organized around a section heading and aggregates all associated text, tables, images, and code elements contained within that section. Through this pro￾cess, 18,083 initially unstructured data items are transformed into structured components. Every compo… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Community hierarchy by theme categories. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Retrieval-Augmented Generation (RAG) pipeline [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: LangGraph-Based optimization loop. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Radar chart of evaluation results across 13 test cases [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: shows the top communities and their sizes returned by this query [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 7 linked inside Pith

  1. [1]

    Langraph conditional workflow

    Adarishanmukh. Langraph conditional workflow. Online, 2025. Medium. Available from: https://medium.com/@adarishanmukh15501/ langraph-c61c8fcaac8f [Accessed 18 August 2025]

    2025

  2. [2]

    Prompt design and engineering: Introduction and advanced methods.arXiv preprint arXiv:2401.14423, 2024

    Xavier Amatriain. Prompt design and engineering: Introduction and advanced methods.arXiv preprint arXiv:2401.14423, 2024. Submitted on 24 January 2024; revised 5 May 2024

    arXiv 2024

  3. [3]

    Software engineering for machine learning: a case study

    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. Software engineering for machine learning: a case study. InProceedings of the 41st IEEE/ACM International Conference on Software Engineering – Software Engineering in Practice (ICSE-SEIP ’19), pages 291–300. IEEE...

    2019

  4. [4]

    Blackwell, Jon Barry, and Anthony G

    Robert E. Blackwell, Jon Barry, and Anthony G. Cohn. Towards reproducible llm evaluation: Quantifying uncertainty in llm benchmark scores.arXiv preprint arXiv:2410.03492, 2024. [Accessed 28 August 2025]

    arXiv 2024

  5. [5]

    Williams, Edwin Brown, Liam Burke-Moore, Ed Chapman, Kate Onslow, Tvesha Sippy, Jonathan Bright, and Evelina Gabasova

    Ryan Sze-Yin Chan, Federico Nanni, Angus R. Williams, Edwin Brown, Liam Burke-Moore, Ed Chapman, Kate Onslow, Tvesha Sippy, Jonathan Bright, and Evelina Gabasova. Prompto: An open source library for asynchronous querying of llm endpoints.arXiv preprint arXiv:2408.11847, 2024. Submitted on 12 August 2024; revised 16 December 2024

    arXiv 2024

  6. [6]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024

    arXiv 2024

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  8. [8]

    Bruce Croft, Donald Metzler, and Trevor Strohman.Search Engines: Information Retrieval in Practice

    W. Bruce Croft, Donald Metzler, and Trevor Strohman.Search Engines: Information Retrieval in Practice. Addison-Wesley, 2010

    2010

Show all 34 references
  1. [9]

    Bruce Croft, Donald Metzler, and Trevor Strohman.Search engines: information retrieval in practice

    W. Bruce Croft, Donald Metzler, and Trevor Strohman.Search engines: information retrieval in practice. Addison-Wesley, Boston, MA, 2010

    2010

  2. [10]

    Devito tutorials

    Devito Project. Devito tutorials. [Online], 2025. [Accessed: 23 July 2025]

    2025

  3. [11]

    J. Exson. Langgraph visualization with get graph. Online, 2025. Medium. Available from: https://medium.com/@josephamyexson/ langgraph-visualization-with-get-graph-ffa45366d6cb [Accessed 18 August 2025]

    2025

  4. [12]

    Asynchronous llm function calling.arXiv preprint arXiv:2412.07017, 2024

    In Gim, Seung seob Lee, and Lin Zhong. Asynchronous llm function calling.arXiv preprint arXiv:2412.07017, 2024. Published on 9 December 2024

    2024 arXiv

  5. [13]

    Large language models for code: security hardening and adversarial testing

    Jingxuan He and Martin Vechev. Large language models for code: security hardening and adversarial testing. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23), November 2023, Copenhagen, Denmark, pages 1865–1879, New York, NY , 2023. ACM

    2023

  6. [14]

    The prompt canvas: A literature- based practitioner guide for creating effective prompts in large language models.arXiv preprint arXiv:2412.05127, 2024

    Michael Hewing and Vincent Leinhos. The prompt canvas: A literature- based practitioner guide for creating effective prompts in large language models.arXiv preprint arXiv:2412.05127, 2024. Submitted on 6 December 2024

    2024

  7. [15]

    Rag-anything: All-in-one multi- modal rag system

    HKUDS/RAG-Anything Developers. Rag-anything: All-in-one multi- modal rag system. Online, 2025. Available from: https://github.com/ HKUDS/RAG-Anything [Accessed 18 August 2025]

    2025

  8. [16]

    Eval- uating large language models trained on code.arXiv preprint arXiv:2301.12507, 2023

    Marie-Anne Lachaux, Baptiste Roziere, Erwan Chan, et al. Eval- uating large language models trained on code.arXiv preprint arXiv:2301.12507, 2023. [Accessed 28 August 2025]

    2023 arXiv

  9. [17]

    Rag from scratch: Part 1 (overview)

    LangChain. Rag from scratch: Part 1 (overview). YouTube video, 2023. Accessed: 28 August 2025

    2023

  10. [18]

    Structured outputs

    LangChain Documentation. Structured outputs. Online, 2025. LangChain Documentation. Available from: https://python.langchain. com/docs/concepts/structured outputs/ [Accessed 18 August 2025]

    2025

  11. [19]

    Langgraph graphs reference

    LangGraph Project. Langgraph graphs reference. Online,

  12. [20]

    we need structured output

    Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. “we need structured output”: Towards user-centered constraints on large language model output. InExtended Abstracts of the CHI Conference on Human Factors in Co...

    2024

  13. [21]

    G-eval: Nlg evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, 2023. Association...

    2023

  14. [22]

    Witte, Felix J

    Mathias Louboutin, Michael Lange, Fabio Luporini, Navjot Kukreja, Philipp A. Witte, Felix J. Herrmann, Paulius Velesko, and Gerard J. Gorman. Devito (v3.1.0): an embedded domain-specific language for finite differences and geophysical exploration.Geoscientific Model Developmen...

    2019

  15. [23]

    Witte, Jan H ¨uckelheim, Charles R

    Fabio Luporini, Mathias Louboutin, Michael Lange, Navjot Kukreja, Philipp A. Witte, Jan H ¨uckelheim, Charles R. Yount, Paul H. J. Kelly, Felix J. Herrmann, and Gerard J. Gorman. Architecture and perfor- mance of devito, a system for automated stencil computation.ACM Transacti...

    2020

  16. [24]

    Manning, Prabhakar Raghavan, and Hinrich Sch ¨utze

    Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch ¨utze. Introduction to information retrieval. Cambridge University Press, Cambridge, UK, 2008

    2008

  17. [25]

    Graphrag: Unlocking llm discovery on narrative private data

    Microsoft Research. Graphrag: Unlocking llm discovery on narrative private data. Microsoft Research Blog [Online], 2024. [Accessed: 28 August 2025]

    2024

  18. [26]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Lo ¨ıc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein, editors,Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 201...

    2014

  19. [27]

    Doing rag on pdfs using file search in the responses api

    OpenAI. Doing rag on pdfs using file search in the responses api. OpenAI Cookbook, 2025. [Accessed 28 August 2025]

    2025

  20. [28]

    Offline evaluation for ranking models: precision and recall at k

    Pinecone. Offline evaluation for ranking models: precision and recall at k. Online, 2023. Available from: https://www.pinecone.io/learn/ offline-evaluation/ [Accessed 18 August 2025]

    2023

  21. [29]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national...

    2019

  22. [30]

    O’Reilly Media, Sebastopol, CA, 2nd edition, 2015

    Ian Robinson, Jim Webber, and Emil Eifrem.Graph databases. O’Reilly Media, Sebastopol, CA, 2nd edition, 2015

    2015

  23. [31]

    The ruff formatter

    Ruff Documentation. The ruff formatter. Online, 2025. Astral Documen- tation. Available from: https://docs.astral.sh/ruff/formatter/ [Accessed 18 August 2025]

    2025

  24. [32]

    V oorhees

    Ellen M. V oorhees. The trec-8 question answering track report. In Proceedings of the Eighth Text REtrieval Conference (TREC-8), Novem- ber 1999, Gaithersburg, MD, USA, volume 500-246 ofNIST Special Publication, pages 77–82, Gaithersburg, MD, 1999. National Institute of Standa...

    1999

  25. [33]

    Enhancing llm-based code translation in repository context via triple knowledge-augmented.arXiv preprint arXiv:2503.18305, 2025

    Shitao Wang, Zhicheng Wu, Kai Yang, Chen Qian, Xin Wang, Weiqiang Zheng, and Junfeng Yang. Enhancing llm-based code translation in repository context via triple knowledge-augmented.arXiv preprint arXiv:2503.18305, 2025

    2025

  26. [2025]

    Available from: https://langchain-ai.github.io/langgraph/reference/ graphs/ [Accessed 18 August 2025]

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.