Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Improving LLMs with a knowledge from databases

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that mining enhanced association rules, rewriting them as plain sentences, and retrieving them by RAG significantly improves LLM answers to database questions, beating a leading model with agents.

desk verdict A plausible rule-to-text RAG pipeline for structured-data QA, but the headline improvement claim lacks the no-method baseline needed to support it. read the letter →

arxiv 2506.05560 v1 pith:T7Y522AF submitted 2025-06-05 cs.CL

Petr Máša This is my paper
classification cs.CL
keywords enhancedassociationrulesLLMretrieval-augmentedgenerationknowledgediscoveryfromdatabasesAIagentsrule-to-textconversion4ft-Minerquestionansweringovertabulardata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a database's hidden regularities, expressed as enhanced association rules, can be mined offline, rewritten as ordinary sentences, and injected into an LLM's context through retrieval-augmented generation (RAG). Doing so, the author argues, materially improves zero-shot answers to questions about the data, so much that a small local model using the rule-sentence document outperforms a leading commercial LLM using agents. The payoff is that the improvement comes without the LLM executing any SQL or Python at answer time, removing the safety and reliability risks of agent-generated code. The paper supports this with a working pipeline on the UK accidents dataset and with three progressively larger rule sets.

What carries the argument

The load-bearing object is the enhanced association rule in the 4ft-Miner pattern, together with its quantifier aad ("above average difference"), defined as lift minus one, i.e. the relative increase in probability of the succedent given the antecedent compared with the dataset-wide baseline. CleverMiner computes the rules; the rule-to-text converter (Algorithms 1-2) maps each aad value into a verbal band ("more likely," "much more likely") and joins enumerated attribute values with "and." The resulting sentences are the smallest retrieval units, so the whole argument rides on RAG selecting the sentence that contains the rule relevant to the user's question and on the LLM parsing its category lists and quantifier phrases.

What would settle it

Measure retrieval quality directly: construct a set of query-rule pairs on a held-out dataset, ask the pipeline to retrieve sentences for each query, and check whether the sentence containing the true rule is returned in the top few positions. If correct rule sentences are not retrieved, or if answer accuracy does not improve when they are, the central claim fails; a simpler variant is to ask the same question with the rule document present versus replaced by an equally long random sampling of sentences, and show that the answers do not change.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that interpretable database knowledge can be served to an LLM as text. The author defines a 4ft-Miner task with Severity as the target, mines a ruleset with the CleverMiner package, converts each rule to a sentence such as "If drivers age is 16 - 35 and driver is Male and Area is 2,Rural then Fatal accidents occur more likely," and uses that sentence file as the RAG document. On the question "On which circumstances occur fatal accidents more than usual?", the augmented ChatGPT answer includes a strong "Highest Risk Combinations" section that the unaugmented answer lacks, and the deepseek r1:1.5b model with 21 rules produces a sharp, correct answer without agents. The author's summary table rates the method above ChatGPT with agents, and the paper reports the same qualitative failure mode at higher rule counts: answers become more general and the model struggles to separate fatal from fatal-or-serious categories.

Load-bearing premise

The weakest load-bearing premise is that the RAG retrieval step will select the rule sentences relevant to the user's question, and that the LLM will then parse those sentences correctly, including the enumerated categories and the verbal quantifier phrases; the paper does not measure retrieval quality and notes boundary cases such as fatal versus fatal-or-serious that the model can confuse.

Editorial extensions

If this is right

  • A database can be summarized once, offline, into a text file of rule sentences, and any RAG-capable LLM can then answer data questions without access to the database or the ability to run code.
  • A small local model without agents can answer the accidents question as well as or better than a leading commercial model with agents, provided the rule document is present.
  • Because no code is executed, the approach is safer for company databases than agent-based SQL/Python generation, which the author explicitly presents as the safety motivation.
  • Rule-set size matters: small rulesets give specific, actionable answers, larger rulesets yield more general answers, and the rule-to-text conversion's aad bands need tuning for the best trade-off.
  • The same mechanism can be extended to other enhanced-association-rule patterns and to agentic use, which the paper lists as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The reported gain is not isolated from retrieval quality; without retrieval measurements, the result is also consistent with any RAG system that happens to fetch sentences containing the right attribute values. A retrieval-only control would separate the rule-mining contribution from the sentence-writing contribution.
  • Inference: Because the safety argument depends on running no code at answer time, the method's practical niche is environments where agent-generated queries are vetoed; in those settings the rule document must be re-mined whenever the data changes, since the knowledge is static.
  • Inference: The degradation observed at 7,224 rules suggests a tunable ceiling: the verbal banding of aad and the inclusion of base/confidence in the sentence are parameters that could be optimized for larger rulesets, and the paper does not explore that optimization.
  • Inference: The pipeline is pattern-agnostic in principle, so the same rule-to-text-and-RAG recipe should transfer to other interpretable patterns, such as action rules, subgroup discovery, or couples of rules, and to other domains; the paper names only the generic extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a method to improve LLM answers to questions about structured data by mining enhanced association rules (4ft-Miner via CleverMiner) from the dataset, converting the mined rules into natural-language sentences, and injecting these sentences into the LLM's context through RAG. The proposed pipeline is demonstrated on a UK road-accidents dataset with a single question, comparing a rule-augmented DeepSeek 1.5B model (no agents) with an unreported no-rules control, and a ChatGPT model with and without rules (the original ChatGPT condition uses agents). The paper claims a significant improvement in answer quality, including for the no-agent model.

Significance. The rule-mining component is technically sound and reproducible: the quantifiers, parameters, and CleverMiner call are specified, and the example rules are consistent with the stated base and aad thresholds. If the claimed improvement were substantiated, the approach would be a valuable, safe, and interpretable alternative to agentic SQL generation for database question answering, fitting well within the journal's scope. The credibility of the result, however, depends entirely on the evaluation, and the evaluation as reported is not sufficient to support the central claim.

major comments (4)
  1. [Section 4.3, Table 4] The central claim that the method 'improves answer to questions based on a dataset significantly, even without using agents' (Section 4.4) is not supported by the reported experiment. Section 4.3 only shows outputs from DeepSeek 1.5B when the rule document is uploaded; no control run of the same model on the same question without the rule document is reported, and Table 4 leaves the 'Original solution' cell blank for the DeepSeek row. The ChatGPT comparison in Section 4.2 is not a substitute because the original condition there uses agents, which does not test the 'without agents' claim. A controlled A/B comparison with the same model, prompt, and dataset, differing only in the presence of the rule text, is required.
  2. [Table 4] The star ratings in Table 4 appear to contradict the narrative: the 'Original solution' for 'ChatGPT with agents' is marked with nine stars, while both cells for 'OUR METHOD (text rules via RAG)' are marked with seven stars. As printed, the table actually indicates the proposed method is rated lower than the original ChatGPT-with-agents answer, which is the opposite of the claimed 'significant improvement.' If this is a typographical or formatting error, it must be corrected; otherwise, the conclusion is undermined by the paper's own quantitative summary.
  3. [Sections 4.1 and 4.2] The entire evaluation rests on a single manually formulated question and a single manual star-rating, with no rubric, no blinding, no repeated trials, and no statistical analysis. The word 'significant' in Section 4.4 has no statistical backing, and for a stochastic system, one or even a few generations cannot establish a reliable improvement. The paper should evaluate on a set of questions (potentially drawn from the benchmarks surveyed in Section 2.4) and report aggregate results across multiple runs, ideally with independent human or automated scoring.
  4. [Section 3 and Section 4.3] The method's effectiveness assumes that the RAG retriever selects the rule sentences relevant to the question and that the LLM correctly parses the enumerated categories and quantifier phrases ('more likely', 'fatal or serious'). No retrieval-quality measurement is reported, and Section 4.3 itself notes that the model sometimes fails to distinguish 'fatal' from 'fatal or serious.' This is a load-bearing assumption that is acknowledged but never tested. Without a control condition or retrieval evaluation, the contribution of the rule text to the observed outputs is not isolated from the model's priors, the dataset context, or the prompt wording.
minor comments (5)
  1. [Throughout] The manuscript contains numerous typos and formatting issues, including 'aproiri' (Abstract/Introduction), 'Deespeek' in Table 4, 'S i m p l e I m p u t e r' in the code block, 'K eywords' in the keyword line, and several grammar errors (e.g., 'they was trained' in Section 2.1). These should be corrected.
  2. [Algorithm 1] Algorithm 1 relies on arrays `aad_boundaries` and `aad_statements` that are not specified for the experiments. The example in Section 3 ('(2,1,0.25,-0.2,...)') is informal; the exact band boundaries and phrase mappings used in the reported runs should be given for reproducibility.
  3. [Section 3 code] The code listing contains unused imports (`clm_vars`, `clm_seq`, `clm_lcut`, `clm_rcut`, `clm_subset`) and a machine-specific Windows path ('w:\\accidents.zip'). A cleaned, runnable version of the mining call would strengthen the reproducibility of the paper.
  4. [References] The CleverMiner reference is cited as 'Petr Máša, 2025' with no full bibliographic entry, only a repository link. The paper should provide a complete citation for the software package.
  5. [Table 3] Table 3 reports the number of rules for each parameter setting, but it would be useful to also report the resulting text-document length and the retrieval time or context-window usage, since the method's scalability is relevant to the RAG discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the pipeline is an empirical RAG evaluation with fixed thresholds, not a fitted predictor; self-citations are present but not load-bearing.

full rationale

The paper's chain is dataset -> enhanced association rule mining (CleverMiner, Section 3) -> rule-to-text (Algorithms 1-2) -> RAG injection -> LLM answer. The central claim is an empirical evaluation of a knowledge-injection pipeline, not a derivation of a predicted quantity from a fitted input. The thresholds Base and aad are explicitly set by the author in the Section 3 code block and are not fitted to the evaluation outcome; the rule-to-text aad bands are also fixed before the experiments. The comparison in Section 4.2 contrasts the same ChatGPT with and without the rule document, so the difference is attributable to the injected rules rather than to a hidden fit. Section 4.3's Deepseek experiments lack a no-rule baseline (Table 4 leaves 'Original solution' blank for the Deepseek row), so the 'even without using agents' improvement claim in Section 4.4 is not fully supported; however, a missing control is an experimental-validity problem, not circularity. The paper cites the author's CleverMiner package and Máša & Rauch works, but these are implementation/tool references and rule-handling techniques; they do not substitute for the evaluation or define the target result. The acknowledged limitation that the model sometimes fails to distinguish 'fatal' from 'fatal or serious' (Section 4.3, quoting the think section: 'But the exact phrase here says "one of fatal and serious occurs more likely," which doesn't specify which.') is a parsing-quality issue, again not a circular step. No equation or definition makes the output equivalent to the input by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method has no invented entities. It relies on user-set thresholds for rule interestingness (Base and aad), hand-crafted linguistic bands for the aad quantifier, and the untested assumption that RAG retrieval surfaces the right rules for a question.

free parameters (3)
  • Base threshold (support) = 300 to 4000 across experiments
    User-set threshold controlling the number of mined rules; lowered to generate 7224 rules in test 3.
  • aad threshold (relative confidence) = 0.5 to 1.3 across experiments
    User-set filter for interestingness, directly influences which rules appear in the text document.
  • aad band boundaries = (2, 1, 0.25, -0.2, ...) as stated in Algorithm 1
    Hand-chosen thresholds converting numeric aad to phrases like 'more likely'; affects the text the LLM reads.
assumptions (3)
  • domain assumption 4ft-Miner pattern and quantifiers (base, aad) capture knowledge relevant to the target question
    The paper assumes that rules with aad above threshold are good evidence for circumstances, but this is not proven for LLM QA.
  • domain assumption RAG retrieval will surface the relevant rule sentences when the question is asked
    Section 3 describes embedding but gives no evaluation of retrieval quality; the method depends on this step working.
  • domain assumption The LLM correctly interprets the textual rules, including the enumerated categories and the quantifier phrases
    The authors note reasoning about intervals is difficult and avoid joined categories, but they still rely on the model to parse the rule text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLMs with a knowledge from databases." pith.science (2026). https://pith.science/paper/T7Y522AF

@misc{pith2026250605560,
  author       = {Pith},
  title        = {Pith review of: Improving LLMs with a knowledge from databases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7Y522AF}},
  note         = {Machine review of arXiv:2506.05560}
}
read the original abstract

Large language models (LLMs) are achieving significant progress almost every moment now. Many advanced techniques have been introduced and widely accepted, like retrieval-augmentation generation (RAG), agents, and tools. Tools can query the database to answer questions from structured data files or perform groupings or other statistics. This unlocks huge opportunities, such as it can answer any question, but also poses threats, such as safety, because there is no control over the commands that are created. We would like to discuss whether we can create a new method that improves answers based on dataset/database via some interpretable ML methods, namely enhanced association rules. The advantage would be if the method can be also used in some safe technique like RAG. Association rules have a sound history. Since the introduction of CN2 and aproiri, many enhancements have been made. In parallel, enhanced association rules have been introduced and evolved over the last 40 years. The general problem is typically that there are too many rules. There are some techniques for handling it, but when LLM emerged, it turned out to be the best use case for the RAG technique for LLMs. We proposed a method that generates a ruleset based on defined knowledge patterns, then converts rules into text form via a rule-to-text converter, and includes the result as an RAG into LLM. We compared this method with ChatGPT (even with using agents) and we have discovered a significant improvement in answering questions based on the dataset. We have also tried several strategies how much rules to generate. We found this improvement interesting. Moreover, it can also be improved in many ways as future work, like incorporating other patterns, the use of rule mining as an agent, and many others.

Figures

Figures reproduced from arXiv: 2506.05560 by the authors.

Figure 1
Figure 1. Severity of fatal accident is almost doubled for male drivers on speed limit 60 compared to the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Proposed schema of augmenting knowledge from database into LLM [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Male drivers in the age 16-35 have in rural areas a 3.0%/1.9%=55% bigger risk of fatal accidents [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The result from ChatGPT. Results are based on 1D profiles. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The result from ChatGPT based on generated rules. We can see strong [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 27 canonical work pages

  1. [1]

    Agrawal, R., Imielinski, T., and Swami, A. N. (1993). Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, Washington, DC, USA, May 26-28, 1993 , pages 207--216

    work page 1993

  2. [2]

    Angelov, P., Soares, E., Jiang, R., Arnold, N., and Atkinson, P. (2021). Explainable artificial intelligence: an analytical review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 11

    work page 2021

  3. [3]

    Atzmueller, M. (2015). Subgroup discovery. WIREs Data Mining and Knowledge Discovery , 5(1):35--49

    work page 2015

  4. [4]

    Barredo Arrieta , A., Díaz-Rodríguez, N., Del Ser , J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., and Herrera, F. (2020). Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion , 58:82--115

    work page 2020

  5. [5]

    and Niblett, T

    Clark, P. and Niblett, T. (1989). The cn2 induction algorithm. Machine Learning , 3(4):261--283

    work page 1989

  6. [6]

    Dardzinska, A. (2013). Action Rules , pages 47--89. Springer Berlin Heidelberg, Berlin, Heidelberg

    work page 2013

  7. [7]

    Hahsler, M. (2023). Arulespy: Exploring association rules and frequent itemsets in python

    work page 2023

  8. [8]

    Hahsler, M., Gruen, B., and Hornik, K. (2005). arules -- A computational environment for mining association rules and frequent item sets. Journal of Statistical Software , 14(15):1--25

    work page 2005

Show all 29 references
  1. [9]

    and Havr\' a nek, T

    H\' a jek, P. and Havr\' a nek, T. (1978). Mechanising Hypothesis Formation - Mathematical Foundations for a General Theory . Springer

    1978

  2. [10]

    H \' a jek, P., Hole n a, M., and Rauch, J. (2010). The GUHA method and its meaning for data mining. J. Comput. Syst. Sci. , 76(1):34--48

    2010

  3. [11]

    Hu, E., Shen, Y., Wallis, P., et al. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  4. [12]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In NIPS'20: Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 9459--9474

    2020

  5. [13]

    Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. B. (2020). Explainable ai: A review of machine learning interpretability methods. Entropy , 23

    2020

  6. [14]

    Liu, X., Wu, Z., Wu, X., Lu, P., Chang, K.-W., and Feng, Y. (2024). Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data

    2024

  7. [15]

    Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30 , pages 4765--4774. ...

    2017

  8. [16]

    and Rauch, J

    M \' a s a, P. and Rauch, J. (2022). Enhanced association rules and python. In Nicosia, G., Ojha, V., Malfa, E. L., Malfa, G. L., Pardalos, P. M., Fatta, G. D., Giuffrida, G., and Umeton, R., editors, Machine Learning, Optimization, and Data Science - 8th International Worksho...

    2022

  9. [17]

    and Rauch, J

    M \' a s a, P. and Rauch, J. (2023). A novel algorithm for mining couples of enhanced association rules based on the number of output couples and its application. Journal of Intelligent Information Systems , pages 1--28

    2023

  10. [18]

    and Rauch, J

    M \' a s a, P. and Rauch, J. (2024). A novel algorithm weighting different importance of classes in enhanced association rules. Knowledge-Based Systems , 294:111741

    2024

  11. [19]

    A., Mart \'i nez C \'a mara, E., and Camacho-Collados, J

    Os \'e s Grijalba, J., Ure \ n a-L \'o pez, L. A., Mart \'i nez C \'a mara, E., and Camacho-Collados, J. (2024). Question answering over tabular data with D ata B ench: A large-scale empirical evaluation of LLM s. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., ...

    2024

  12. [20]

    S., Jain, S

    Pahilajani, A., Trivedi, D., Shuai, J., Yone, K. S., Jain, S. R., Park, N., Rossi, R. A., Ahmed, N. K., Dernoncourt, F., and Wang, Y. (2024). Grs-qa -- graph reasoning-structured question answering dataset

    2024

  13. [21]

    Cleverminer: Beyond apriori

    Petr M\' a s a (2020--2025). Cleverminer: Beyond apriori

    2020

  14. [22]

    and Liu, Z

    Quan, Y. and Liu, Z. (2024). E con L ogic QA : A question-answering benchmark for evaluating large language models in economic sequential reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Findings of the Association for Computational Linguistics: EMNLP 2024 ,...

    2024

  15. [23]

    and Wieczorkowska, A

    Ra \'s , Z. and Wieczorkowska, A. (2000). Action-rules: How to increase profit of a company. sl

    2000

  16. [24]

    Rauch, J., S im 23unek, M., Chud\' a n, D., and M\' a s a, P. (2022). Mechanising Hypothesis Formation - Principles and Case Studies . CRC Press

    2022

  17. [25]

    why should I trust you?

    Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , page...

    2016

  18. [26]

    Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. , 1(5):206--215

    2019

  19. [27]

    S im u nek, M. (2003). Academic kdd project lisp-miner. In Abraham, A., Franke, K., and K \"o ppen, M., editors, Intelligent Systems Design and Applications , pages 263--272, Berlin, Heidelberg. Springer Berlin Heidelberg

    2003

  20. [28]

    S im 23unek, M. (2003). Academic KDD project LISp-Miner . In Abraham, A., Franke, K., and K \"o ppen, M., editors, Intelligent Systems Design and Applications , pages 263--272, Berlin, Heidelberg. Springer Berlin Heidelberg

    2003

  21. [29]

    Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., and Chua, T.-S. (2021). TAT - QA : A question answering benchmark on a hybrid of tabular and textual content in finance. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual M...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.