Pith. sign in

REVIEW 4 major objections 5 minor 83 references

Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MPDetector claims to detect inconsistencies between API documentation and code for multi-parameter constraints in Python data science libraries, achieving 92.8% precision and 11 confirmed real issues.

desk verdict A plausible and useful tool for a real gap, but the headline precision number is miscalculated and the preprocessing equivalence assumption needs testing before I'd trust the quantitative claims. read the letter →

arxiv 2411.11410 v2 pith:ILFTJG73 submitted 2024-11-18 cs.SE

Xiufeng Xu , Fuman Xie , Chenguang Zhu , Guangdong Bai , Sarfraz Khurshid , Yi Li This is my paper · ORCID
classification cs.SE
keywords multi-parameterconstraintscode-documentationinconsistencysymbolicexecutionLLMconstraintextractionfuzzylogicPythondatasciencelibrariesAPIdocumentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MPDetector claims that documentation errors involving multiple interdependent API parameters can be found automatically by checking documentation-derived constraints against code-derived path constraints. The paper combines symbolic execution to extract constraints from Python source with GPT-4 to extract constraints from free-form docstrings, then reconciles the two using a custom fuzzy constraint logic that tolerates minor LLM mistakes. On a mutation-based dataset of 216 constraints, it reports 92.8% precision for detecting inconsistencies, and 11 of 14 issues reported to library maintainers have been confirmed. If accurate, this gives maintainers a way to keep documentation in sync with code for the many-parameter APIs common in data science libraries.

What carries the argument

The load-bearing component is the fuzzy constraint satisfaction framework, defined on an EBNF grammar of constraints that are viewed as binary trees of atomic expressions (parameter-operator-value triples) joined by negation, conjunction, and disjunction. Expression similarity combines normalized Levenshtein distance between parameter names and values with cosine similarity between operator vectors; constraint similarity aggregates atomic similarities by min for AND, max for OR, and complement for NOT; and a membership function averages, over all code path constraints, the product of that similarity with a satisfiability predicate computed by the Z3 SMT solver. The fuzzy layer is what lets the tool tolerate near-miss LLM extractions (typos, reversed operators) while still flagging genuine mismatches; symbolic execution supplies the ground truth — the set of path constraints the parameters actually obey in code.

What would settle it

For a sample of functions in the dataset, manually enumerate the true path constraints and compare them to the constraint set produced after the paper's code transformations; if any transformation demonstrably omits or changes a branch condition, the extracted code-constraints are unreliable and the reported precision would not hold.

Watch

Extended reading notes

Core claim

The central claim is that multi-parameter code-documentation inconsistencies can be detected by pairing symbolic execution with LLM-based natural language constraint extraction. The paper contributes MPDetector, which first rewrites Python functions (e.g., splitting classes into functions, replacing strings and external calls with symbolic inputs) and runs concolic execution to collect path constraints; separately, it prompts GPT-4 with chain-of-thought and few-shot examples to emit logical expressions for documented constraints, including special fuzzy predicates such as ignore(x) for implicit constraints. A fuzzy constraint logic then computes, via expression similarity and a membership function, how well each doc-constraint is satisfied across all path constraints, flagging a violation when satisfaction is low. Reported results: constraint extraction accuracy of 91.7% (66/72), inconsistency detection precision of 92.8% (117 true positives, 2 false positives), compared to an LLM-only checker at 41.3% precision; 14 issues were reported to developers, with 11 confirmed.

Load-bearing premise

The manual source-code transformations (splitting classes into functions, replacing strings and calls with symbolic inputs, rewriting syntax) preserve the original path constraints exactly, even though the paper does not formally prove or empirically validate this equivalence.

Editorial extensions

If this is right

  • Documentation updates could be checked automatically against code before each release, catching drift early.
  • The technique extends beyond the four training libraries: the paper reports confirmed issues in keras, dask, and statsmodels, suggesting generalization to other Python libraries with NumPy- or Google-style docstrings.
  • Implicit constraints expressed with vague words like 'ignore' and 'override' become detectable, which pattern-based checkers miss.
  • The reported 92.8% precision is substantially higher than an LLM-only checker (41.3%), implying that fusing symbolic reasoning with LLM extraction is the key to usable automated doc checking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fuzzy similarity membership function could be reused as a confidence score, letting maintainers triage the highest-scoring inconsistencies first; the paper does not explore this operational use.
  • The manual code transformations are the scalability bottleneck; automating them with verified source-to-source rewriting would be the natural next step, whereas today the tool requires human effort per function.
  • The evaluation's precision is dominated by the synthetic mutation dataset; on real-world drift (e.g., constraints changed across versions) the true precision could differ, so an independent blind study on a fresh library would be a natural test of the 92.8% figure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MPDetector, a tool that detects inconsistencies between multi-parameter constraints stated in Python data science library documentation and constraints actually enforced by the library code. The pipeline extracts doc-constraints with GPT-4 using chain-of-thought and few-shot prompting, extracts code-constraints through dynamic symbolic execution of manually rewritten source functions, and reconciles the two with a custom fuzzy constraint logic evaluated by an SMT solver. The authors construct a constraint dataset of 72 real constraints and a mutation-based inconsistency dataset of 126 inconsistent and 90 consistent constraints, report 91.7% accuracy for doc-constraint extraction, report 92.8% precision for inconsistency detection, and describe 14 reported issues with 11 developer confirmations.

Significance. If the approach is sound, the paper addresses a genuine and underexplored problem: detecting multi-parameter code-documentation inconsistencies in a dynamic language. The combination of LLM-based natural language constraint extraction with symbolic execution and fuzzy matching is a reasonable design, and the developer-confirmed issues provide real-world grounding that is rare in this area. The artifact and dataset are valuable assets. However, two load-bearing points need scrutiny before the reported results can be accepted: the unvalidated semantic-preservation claim for the code rewrites in §3.2.1, and the construction of the inconsistency benchmark in §4.1.1, which appears to label mutations relative to the original documentation rather than to the actual code behavior. The metric labeling error in Table 3 also obscures the true performance.

major comments (4)
  1. [§3.2.1, Figure 5] The paper asserts that the manual rewrites applied before symbolic execution are 'equivalent code transformations that do not affect path exploration,' but this claim is neither formally justified nor empirically validated, and the example in Figure 5 shows a concrete semantic change. Rewriting `if sample_weight is not None` as `if sample_weight != 'None'` changes the branch outcome when `sample_weight` is `None` (the first is False, the second is True) and also when it is the string `'None'` (the reverse). Since the extracted code-constraint set is built entirely from the rewritten functions, every downstream doc-code comparison, and therefore the RQ2 and RQ3 results, is computed against an artifact whose relationship to the real library code is unverified. The paper should provide a formal equivalence argument for each rewrite class, or, more practically, a differential test or path-coverage comparison between original and rewritten functions on representative inputs, and should either fix the `is not None` / `!= 'None'` rewrite or justify why it is safe in the specific contexts where it is applied.
  2. [Table 3, §4.2.2] The reported precision of 92.8% for 'MPDetector w/ fuzzy words&fuzzy constraints' is arithmetically inconsistent with the table's own TP and FP values. The row shows FP=2 and TP=117, so precision is TP/(TP+FP)=117/119=98.3%; the value 92.8% equals 117/126, which is recall (or true positive rate) over the 126 inconsistent constraints. The accompanying text 'successfully identifying 119 inconsistencies with a precision of 92.8%' further conflates the number of predicted positives (TP+FP=119) with precision. This mislabeling matters because the paper's headline claim is precision. The authors should correct the metric names, report precision and recall separately (and ideally F1), and update the abstract and conclusions accordingly.
  3. [§4.1.1, Inconsistency Dataset] The construction of the inconsistency dataset is partially circular with respect to the tool's own doc-constraint extraction. The text states that mutations are 'fed ... into an SMT solver to verify if the mutations violate the original constraints,' i.e., the labels are generated by checking whether a mutated doc-constraint contradicts the original doc-constraint, not by independently checking whether the mutated constraint contradicts the actual code behavior. Since the target problem is code-documentation inconsistency, ground truth should be established against the code. The sentence 'We also manually inspected each them to ensure the constraint is inconsistent' is not enough detail: the authors should describe how many of the 216 mutations were judged inconsistent by manual code inspection, how disagreements were resolved, and whether the final 126/90 split reflects code-verified inconsistencies. Without this, the RQ2 evaluation may reward the tool for matching its own doc-derived oracle rather than for detecting real code-doc mismatches.
  4. [§3.3, Definition 3.3] The membership function in Definition 3.3 is not fully well-defined. The notation $P(c_i \wedge \varphi_i)$ appears to use a predicate whose arguments are a constraint and a path constraint, but the surrounding text says '$P(c,\varphi)$ represents whether a given constraint satisfies one of path constraint,' and the sentence '0.7False = 0.3True' is not standard and is unexplained. The paper should give a precise inductive definition of $P$, clarify whether $c_i$ is obtained by replacing expressions in $c$ with the closest matching expressions in $\varphi_i$ (and how ties are broken), and state what properties of the membership function are relied upon for the fuzzy consistency verdict. These details matter because the fuzzy checker is a central contribution and is used to claim a 23.8% precision improvement.
minor comments (5)
  1. [§1, Abstract] The abstract and contributions describe 'precision of 92.8%' without distinguishing it from recall; this should be corrected once the metric in Table 3 is fixed.
  2. [§3.2.1, Figure 5] The modified source code in Figure 5 contains an invalid Python f-string with misplaced quotes: `return (f '(sample_weight = {sample_weight}) ...')`; this typo makes the example harder to follow and should be fixed.
  3. [§4.2.2, RQ2 answer] The answer to RQ2 says MPDetector 'successfully detected 117 out of 126 inconsistent constraints, achieving a 92.8% precision;' this conflates recall with precision. The sentence should be reworded after the metric correction.
  4. [§6, Related Work] References [77] and [78] are duplicates of the same paper (Zhang 2024), and the citation [75] is used for the 'conjunctive combination principle' but points to Zadeh's 1965 fuzzy sets paper; a citation to the specific fuzzy aggregation principle would be more appropriate.
  5. [Throughout] There are numerous typos and grammatical errors, including 'Intuitivelly', 'unpredicatability', 'experssions', 'theorem of strings' (should be 'theory of strings'), 'to use tegother correctly', and 'alternate function calls' (should be 'replace function calls'). A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

The 92.8% precision figure rests on a mutation benchmark whose 'inconsistent' labels are generated by an SMT check of mutated doc-constraints against original doc-constraints, not against code; the 11 developer-confirmed issues supply the independent evidence that keeps the central claim from being wholly circular.

  1. other [Section 4.1.1 (Inconsistency Dataset)]
    "We feed the mutated constraints and the original constraints into an SMT solver to verify if the mutations violate the original constraints. We also manually inspected each them to ensure the constraint is inconsistent. It is important to note that modifying a correct constraint does not necessarily turn it into an incorrect one. As a result, we obtained an Inconsistency Dataset containing 126 inconsistent constraints and 90 consistent constraints."

    The benchmark's ground-truth label 'inconsistent' is defined by a constraint-logic check between the mutated doc-constraint and the original doc-constraint, not between the mutated doc-constraint and the library code. MPDetector is then evaluated on detecting code-documentation inconsistencies, and since the original doc-constraint is never independently validated against code, a mutation that violates the original doc-constraint need not be a code-doc inconsistency; conversely, a mutation logically equivalent to the original can still disagree with the code. The reported precision is therefore measured against labels constructed from the same doc-constraint comparison activity the tool performs, with the original documentation standing in for the code oracle.

full rationale

The central evaluation claim is the 92.8% precision on the 216-constraint inconsistency dataset, and that dataset's positives are produced by checking mutated doc-constraints against original doc-constraints, not against code behavior. This makes the headline precision partially self-referential. The paper's developer-confirmed issues (11 of 14 reported) provide independent grounding that MPDetector does find real code-documentation inconsistencies, which is why the score is not higher. Separately, Section 3.2.1 asserts that manual rewrites are 'equivalent code transformations that do not affect path exploration' without formal or empirical validation; Figure 5's rewrite of 'sample_weight is not None' as 'sample_weight != "None"' changes branch semantics for None and for the string 'None'. This is a serious validity threat to the code-constraint oracle, but it is an unverified equivalence assumption rather than a circular reduction, so it is not counted in the circularity score. The only same-author citation in the load-bearing related-work taxonomy is [83], used for the standard incorrectness/incompleteness categorization; it is not used to justify MPDetector's design or to forbid alternatives. The fuzzy constraint equations (5) and (6) are definitions of the method's own membership function, not predictions derived from independent premises.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on documented constraints being checkable against code path constraints, which requires equivalence-preserving code transformations and a fuzzy similarity threshold. These are assumed or under-specified rather than demonstrated. No new physical entities are introduced.

free parameters (4)
  • beta (operator weight in expression similarity) = not reported
    In Equation (1), beta weights operator similarity and alpha=(1-beta)/2. The paper never gives beta or a sensitivity analysis, yet it controls how much an operator mismatch contributes to the fuzzy satisfaction score and therefore to inconsistency decisions.
  • membership threshold for inconsistency verdict = not reported
    Definition 3.3 defines mu(c) in [0,1], but the paper never states the cutoff or rule that converts mu(c) into a binary "inconsistent" finding. Without this, the detection decision is under-specified and any threshold would be a free parameter.
  • documentation chunk size = 1500 words
    Chosen by a preliminary heuristic study on ten documents; it affects how much context GPT-4 sees and thus extraction accuracy.
  • number of few-shot examples = 4
    Four in-context examples are used in the constraint extraction prompt. The choice is not justified beyond being part of prompt design, and it can materially affect LLM extraction quality.
assumptions (6)
  • standard math Fuzzy min and max combination principles (Zadeh) apply to conjunction and disjunction of constraints.
    Equation (5) uses min for conjunction and max for disjunction, citing the conjunctive and disjunctive combination principles from fuzzy set theory.
  • standard math Levenshtein distance and cosine similarity are appropriate measures for comparing parameter names, values, and operators.
    Equation (1) uses normalized Levenshtein distance for strings and cosine similarity for operator vectors; these are standard but the choice is not empirically validated here.
  • domain assumption A documented multi-parameter constraint describes intended behavior that the code should satisfy, so a mismatch is a defect.
    The entire CDI detection task assumes documentation and code should agree, stated in Sections 1 and 3. This is reasonable but is a domain premise rather than a proven fact.
  • domain assumption Manual source code transformations preserve the exact path constraints of the original Python code.
    Section 3.2.1 states the transformations must not alter path constraints, but no equivalence proof or empirical validation is provided.
  • ad hoc to paper The five-feature operator encoding (C,E,G,L,N) in Equation (3) is a sufficient representation of operator similarity.
    The encoding is custom-designed for this tool and its adequacy is assumed. Different encodings would change similarity values and potentially detection outcomes.
  • ad hoc to paper The eight mutation patterns are representative of real code-documentation inconsistencies.
    Section 4.1.1 derives the patterns from around 20 resolved GitHub issues. This is a small inductive basis for claiming the mutations model realistic defects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries." pith.science (2026). https://pith.science/paper/ILFTJG73

@misc{pith2026241111410,
  author       = {Pith},
  title        = {Pith review of: Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILFTJG73}},
  note         = {Machine review of arXiv:2411.11410}
}
read the original abstract

Modern AI- and Data-intensive software systems rely heavily on data science and machine learning libraries that provide essential algorithmic implementations and computational frameworks. These libraries expose complex APIs whose correct usage has to follow constraints among multiple interdependent parameters. Developers using these APIs are expected to learn about the constraints through the provided documentations and any discrepancy may lead to unexpected behaviors. However, maintaining correct and consistent multi-parameter constraints in API documentations remains a significant challenge for API compatibility and reliability. To address this challenge, we propose MPDetector, for detecting inconsistencies between code and documentation, specifically focusing on multi-parameter constraints. MPDetector identifies these constraints at the code level by exploring execution paths through symbolic execution and further extracts corresponding constraints from documentation using large language models (LLMs). We propose a customized fuzzy constraint logic to reconcile the unpredictability of LLM outputs and detects logical inconsistencies between the code and documentation constraints. We collected and constructed two datasets from four popular data science libraries and evaluated MPDetector on them. The results demonstrate that MPDetector can effectively detect inconsistency issues with the precision of 92.8%. We further reported 14 detected inconsistency issues to the library developers, who have confirmed 11 issues at the time of writing.

Figures

Figures reproduced from arXiv: 2411.11410 by the authors.

Figure 1
Figure 1. Examples of an explicit constraint from Statsmodels. (4) We evaluated our tool on four real-world popular data science libraries. We reported 14 inconsistency issues discovered by MPDetector to the developers, who have confirmed 11 inconsistencies at the time of writing. 2 Background In this section, we review the essential terminology and background necessary for understanding the remainder of the paper. 2.1 Multi-… view at source ↗
Figure 2
Figure 2. Examples of implicit constraint from Scikit-learn. part of the documentation only explicitly mentions one parameter affinity, omitting the subject “gamma”. More importantly, “ignore” is not a specific identifier or value but rather a description of the program logic—if the parameter affinity is set to nearest_neighbors, then the parameter gamma will not be used. Whereas, above constraint does not faithfully reflect … view at source ↗
Figure 3
Figure 3. The architectural overview of MPDetector. In this section, we define the issue of code-documentation inconsistency caused by multi￾parameter constraint and provide a detailed description of our approach. An API documentation error is an inconsistency between the library source code and its API documentation. Multi￾parameter constraints refer to conditional dependency relationships that exist among multiple parameter… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Example of two docstring styles in the form of key-value pairs, based on the two aforementioned docstring styles. We then further applied a rule-based heuristic approach to keep documentations that potentially contain constraints and discard the rest. For instance, if …
Figure 5
Figure 5. Figure 5: Extracting constraint from code (1) Current python symbol execution tool can not solve class directly. Therefore, it is necessary to split the class into functions (i.e. member functions). The corresponding member variables also need to be changed and used as symbolic …
Figure 6
Figure 6. Figure 6: Prompt structure for constraints extraction [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Extended Backus-Naur form for multi-parameter constraint. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Examples of implicit constraint from Scikit-learn. unable to verify them accurately at that time. The third inconsistency stemmed from ambiguity in the natural language, which resulted in a different interpretation diverged from the developers’ original intent. Further…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 67 canonical work pages

  1. [1]

    Dask issue#11336

    2024. Dask issue#11336. https://github.com/dask/dask/issues/11336

    2024

  2. [2]

    Keras issue#20141

    2024. Keras issue#20141. https://github.com/keras-team/keras/issues/20141

    2024

  3. [3]

    Scikit-learn issue#28469

    2024. Scikit-learn issue#28469. https://github.com/scikit-learn/scikit-learn/issues/28469

    2024

  4. [4]

    Scikit-learn issue#28470

    2024. Scikit-learn issue#28470. https://github.com/scikit-learn/scikit-learn/issues/28470

    work page 2024

  5. [5]

    Scikit-learn issue#28473

    2024. Scikit-learn issue#28473. https://github.com/scikit-learn/scikit-learn/issues/28473

    work page 2024

  6. [6]

    Scikit-learn issue#29440

    2024. Scikit-learn issue#29440. https://github.com/scikit-learn/scikit-learn/issues/29440

    work page 2024

  7. [7]

    Scikit-learn issue#29463

    2024. Scikit-learn issue#29463. https://github.com/scikit-learn/scikit-learn/issues/29463

    work page 2024

  8. [8]

    Scikit-learn issue#29464

    2024. Scikit-learn issue#29464. https://github.com/scikit-learn/scikit-learn/issues/29464

    work page 2024

Show all 83 references
  1. [9]

    Scikit-learn issue#29509

    2024. Scikit-learn issue#29509. https://github.com/scikit-learn/scikit-learn/issues/29509. MPDetector 19

    2024

  2. [10]

    Scikit-learn issue#30099

    2024. Scikit-learn issue#30099. https://github.com/scikit-learn/scikit-learn/issues/30099

    2024

  3. [11]

    Statsmodels issue#9304

    2024. Statsmodels issue#9304. https://github.com/statsmodels/statsmodels/issues/9304

    2024

  4. [12]

    Emad Aghajani, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2018. A large-scale empirical study on linguistic antipatterns affecting apis. In 2018 IEEE International conference on software maintenance and evolution (ICSME) . IEEE, 25–35

    2018

  5. [13]

    Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shep- herd. 2020. Software documentation: the practitioners’ perspective. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 590–601

    2020

  6. [14]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

    2019

  7. [15]

    Hiralal Agrawal and Joseph R Horgan. 1990. Dynamic program slicing. ACM SIGPlan Notices 25, 6 (1990), 246–256

    1990

  8. [16]

    Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2016. Linguistic antipatterns: What they are and how developers perceive them. Empirical Software Engineering 21 (2016), 104–158

    2016

  9. [17]

    Thomas Ball and Jakub Daniel. 2015. Deconstructing dynamic symbolic execution. In Dependable Software Systems Engineering. IOS Press, 26–41

    2015

  10. [18]

    Arianna Blasi and Alessandra Gorla. 2018. Replicomment: identifying clones in code comments. In Proceedings of the 26th Conference on Program Comprehension . 320–323

    2018

  11. [19]

    Sally C Brailsford, Chris N Potts, and Barbara M Smith. 1999. Constraint satisfaction problems: Algorithms and applications. European journal of operational research 119, 3 (1999), 557–581

    1999

  12. [20]

    Alessandro Disney Bruni, Tim Disney, and Cormac Flanagan. 2011. A peer architecture for lightweight symbolic execution. Universidad de California, Santa Cruz (2011)

    2011

  13. [21]

    Gerardo Canfora, Aniello Cimitile, and Andrea De Lucia. 1998. Conditioned program slicing. Information and Software Technology 40, 11-12 (1998), 595–607

    1998

  14. [22]

    Barthélémy Dagenais and Martin P Robillard. 2010. Creating and evolving developer documentation: understanding the decisions of open source contributors. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering . 127–136

    2010

  15. [23]

    Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A comparative analysis of large language models for code documentation generation. In Proceedings of the 1st ACM International Conference on AI-Powered Software

    2024

  16. [24]

    Google. 2024. AI for every developer . https://ai.google.dev

    2024

  17. [25]

    Andrew Habib and Michael Pradel. 2018. Is this class thread-safe? inferring documentation using graph-based learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . 41–52

    2018

  18. [26]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis...

    2024

  19. [27]

    Mary Jean Harrold and Mary Lou Soffa. 1994. Efficient computation of interprocedural definition-use chains. ACM Transactions on Programming Languages and Systems (TOPLAS) 16, 2 (1994), 175–204

    1994

  20. [28]

    Andrew Head, Caitlin Sadowski, Emerson Murphy-Hill, and Andrea Knight. 2018. When not to comment: Questions and tradeoffs with API documentation for C++ projects. In Proceedings of the 40th International Conference on Software Engineering. 643–653

    2018

  21. [29]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu

  22. [30]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs (ESEC/FSE 2023). Association for Computing Machinery

    2023

  23. [31]

    Hong Jin Kang and David Lo. 2021. Active learning of discriminative subgraph patterns for api misuse detection. IEEE Transactions on Software Engineering 48, 8 (2021), 2761–2783

    2021

  24. [32]

    Ken Kennedy. 1978. Use-definition chains with applications. Computer Languages 3, 3 (1978), 163–179

    1978

  25. [33]

    Bart Kosko and Satoru Isaka. 1993. Fuzzy logic. Scientific American 269, 1 (1993), 76–81

    1993

  26. [34]

    Seonah Lee, Rongxin Wu, Shing-Chi Cheung, and Sungwon Kang. 2019. Automatic detection and update suggestion for outdated API names in documentation. IEEE Transactions on Software Engineering 47, 4 (2019), 653–675

    2019

  27. [35]

    Bo Lin, Shangwen Wang, Kui Liu, Xiaoguang Mao, and Tegawendé F Bissyandé. 2021. Automated comment update: How far are we?. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) . IEEE, 36–46

    2021

  28. [36]

    Shuang Liu, Jun Sun, Yang Liu, Yue Zhang, Bimlesh Wadhwa, Jin Song Dong, and Xinyu Wang. 2014. Automatic early defects detection in use case documents. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 785–790. 20 Xiufeng Xu, Fuman...

    2014

  29. [37]

    Yang Liu, Mingwei Liu, Xin Peng, Christoph Treude, Zhenchang Xing, and Xiaoxin Zhang. 2020. Generating concept based API element comparison using a knowledge graph. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 834–845

    2020

  30. [38]

    Zhongxin Liu, Xin Xia, David Lo, Meng Yan, and Shanping Li. 2021. Just-in-time obsolete comment detection and update. IEEE Transactions on Software Engineering 49, 1 (2021), 1–23

    2021

  31. [39]

    Zhongxin Liu, Xin Xia, Meng Yan, and Shanping Li. 2020. Automating just-in-time comment updating. In Proceedings of the 35th IEEE/ACM International conference on automated software engineering . 585–597

    2020

  32. [40]

    Pedro Meseguer, Francesca Rossi, and Thomas Schiex. 2006. Soft constraints. In Foundations of Artificial Intelligence. Vol. 2. Elsevier, 281–328

    2006

  33. [41]

    Meta. 2024. Introducing Llama 3. https://www.llama.com

    2024

  34. [42]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 (2022)

    2022 arXiv

  35. [43]

    Martin Monperrus, Michael Eichberg, Elif Tekes, and Mira Mezini. 2012. What should developers be aware of? An empirical study on the directives of API documentation. Empirical Software Engineering 17 (2012), 703–737

    2012

  36. [44]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help With Code Understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, 13 pages

    2024

  37. [45]

    Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J Mooney, and Milos Gligoric. 2019. A framework for writing trigger-action todo comments in executable format. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and S...

    2019

  38. [46]

    OpenAI. 2024. OpenAI Models. https://platform.openai.com/docs/models/o1

    2024

  39. [47]

    OpenAI. 2024. What are tokens and how to count them? https://help.openai.com/en/articles/4936856-what-are-tokens- and-how-to-count-them

    2024

  40. [48]

    Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney. 2021. Deep just-in-time inconsistency detection between comments and source code. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 427–435

    2021

  41. [49]

    PyExSMT. 2024. Python Symbolic Execution. https://github.com/FedericoAureliano/PyExSMT

    2024

  42. [50]

    PyExZ3. 2024. Python Exploration with Z3. https://github.com/thomasjball/PyExZ3

    2024

  43. [51]

    pySMT. 2024. A library for SMT formulae manipulation and solving. https://github.com/pysmt/pysmt

    2024

  44. [52]

    Sawan Rai, Ramesh Chandra Belwal, and Atul Gupta. 2022. A review on source code documentation.ACM Transactions on Intelligent Systems and Technology (TIST) 13, 5 (2022), 1–44

    2022

  45. [53]

    Pooja Rani, Suada Abukar, Nataliia Stulova, Alexandre Bergel, and Oscar Nierstrasz. [n. d.]. Do Comments follow Commenting Conventions? A Case Study in Java and Python. In 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM)

    2021

  46. [54]

    Inderjot Kaur Ratol and Martin P Robillard. 2017. Detecting fragile comments. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 112–122

    2017

  47. [55]

    Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. 2024. Code Comment Inconsistency Detection and Rectification Using a Large Language Model. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Socie...

    2024

  48. [56]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633 (2021)

    2021 arXiv

  49. [57]

    Zsofi Ruttkay. 1994. Fuzzy constraint satisfaction. InProceedings of 1994 IEEE 3rd International Fuzzy Systems Conference. IEEE, 1263–1268

    1994

  50. [58]

    Mohamed Aymen Saied, Houari Sahraoui, and Bruno Dufour. 2015. An observational study on API usage constraints and their documentation. In 2015 IEEE 22nd International conference on software analysis, evolution, and reengineering (SANER). IEEE, 33–42

    2015

  51. [59]

    How to handle soft constraints?

    Thomas Schiex. 1992. Possibilistic constraint satisfaction problems or “How to handle soft constraints?”. InUncertainty in Artificial Intelligence. Elsevier, 268–275

    1992

  52. [60]

    scikit learn. 2024. Machine learning in Python. https://github.com/scikit-learn/scikit-learn

    2024

  53. [61]

    Lin Shi, Hao Zhong, Tao Xie, and Mingshu Li. 2011. An empirical study on evolution of API documentation. In Fundamental Approaches to Software Engineering: 14th International Conference, FASE 2011, Held as Part of the Joint European Conferences on Theory and Practice of Softwa...

    2011

  54. [62]

    Sphnix. 2024. Example Google Style Python Docstrings. https://www.sphinx-doc.org/en/master/usage/extensions/ example_google.html

    2024

  55. [63]

    Sphnix. 2024. Example Numpy Style Python Docstrings. https://www.sphinx-doc.org/en/master/usage/extensions/ example_numpy.html MPDetector 21

    2024

  56. [64]

    statsmodels. 2024. Statistical modeling and econometrics in Python. https://github.com/statsmodels/statsmodels

    2024

  57. [65]

    Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analysis of source code comments. In 2013 21st international conference on program comprehension (icpc) . Ieee, 83–92

    2013

  58. [66]

    Gias Uddin and Martin P Robillard. 2015. How API documentation fails. Ieee software 32, 4 (2015), 68–75

    2015

  59. [67]

    Nalin Wadhwa, Jui Pradhan, Atharv Sonwane, Surya Prakash Sahu, Nagarajan Natarajan, Aditya Kanade, Suresh Parthasarathy, and Sriram Rajamani. 2024. CORE: Resolving Code Quality Issues using LLMs. Proc. ACM Softw. Eng. FSE, Article 36 (2024), 23 pages

    2024

  60. [68]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models (NIPS ’22). 14 pages

    2024

  61. [69]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  62. [70]

    Mark Weiser. 1984. Program slicing. IEEE Transactions on software engineering 4 (1984), 352–357

    1984

  63. [71]

    Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2019. A large-scale empirical study on code-comment inconsistencies. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 53–64

    2019

  64. [72]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024). Association for Comput...

    2024

  65. [73]

    Baowen Xu, Ju Qian, Xiaofang Zhang, Zhongqiang Wu, and Lin Chen. 2005. A brief survey of program slicing. ACM SIGSOFT Software Engineering Notes 30, 2 (2005), 1–36

    2005

  66. [74]

    Bissyandé, and Shunfu Jin

    Boyang Yang, Haoye Tian, Weiguo Pian, Haoran Yu, Haitao Wang, Jacques Klein, Tegawendé F. Bissyandé, and Shunfu Jin. 2024. CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Soft...

    2024

  67. [75]

    Lotfi A Zadeh. 1965. Fuzzy sets. Information and Control (1965)

    1965

  68. [76]

    Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020. CPC: Automatically classifying and propagating natural language comments via program analysis. In Proceedings of the ACM/IEEE 42nd International confe...

    2020

  69. [78]

    Yichi Zhang. 2024. Detecting Code Comment Inconsistencies using LLM and Program Analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024) . Association for Computing Machinery, 683–685

    2024

  70. [79]

    Yichi Zhang, Zixi Liu, Yang Feng, and Baowen Xu. 2024. Leveraging Large Language Model to Assist Detecting Rust Code Comment Inconsistency. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24) . Association for Computing Mach...

    2024

  71. [80]

    Hao Zhong, Na Meng, Zexuan Li, and Li Jia. 2020. An empirical study on API parameter rules. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 899–911

    2020

  72. [81]

    Hao Zhong and Zhendong Su. 2013. Detecting API documentation errors. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications . 803–816

    2013

  73. [82]

    Yu Zhou, Ruihang Gu, Taolue Chen, Zhiqiu Huang, Sebastiano Panichella, and Harald Gall. 2017. Analyzing APIs documentation and code to detect directive defects. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 27–37

    2017

  74. [83]

    Chenguang Zhu, Ye Liu, Xiuheng Wu, and Yi Li. 2022. Identifying solidity smart contract api documentation errors. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1–13

    2022

  75. [2024]

    arXiv preprint arXiv:2401.01325 (2024)

    Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325 (2024)

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.