Pith. sign in

REVIEW 4 major objections 4 minor 26 references

Using ChatGPT to refine draft conceptual schemata in supply-driven design of multidimensional cubes

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read With a carefully engineered prompt, ChatGPT's average refinement errors per draft data-cube schema drop from 9 to 4, and residual errors are fixed by one extra prompt, though a human designer is still needed for validation.

desk verdict A clear, honest pilot study, but the headline 9-to-4 improvement is measured on the five cases used to tune the prompt, so the generalization claim needs an out-of-sample check. read the letter →

arxiv 2502.02238 v1 pith:TNW6NKJ4 submitted 2025-02-04 cs.DB cs.SE

Stefano Rizzi This is my paper · ORCID
classification cs.DBcs.SE
keywords ConceptualdesignMultidimensionalmodelLargelanguagemodelsChatGPTRefinementSupply-drivenDimensionalFactPromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Refinement of draft multidimensional-cube schemas is normally a manual, collaborative task for designers and end-users. This paper asks whether a large language model like ChatGPT can take over most of it, reporting a controlled experiment on five test cases using the Dimensional Fact Model. With basic prompts the model averaged 9 refinement errors per test case; with an engineered prompt that adds role, format, step-by-step procedures, and few-shot chain-of-thought examples, the average dropped to 4. The paper concludes that residual errors can usually be fixed with one additional prompt, but that some designer involvement is still necessary to ensure the refined schemas are valid. If true, this makes LLM-assisted refinement a practical way to speed up data-warehouse design while keeping designers in the loop.

What carries the argument

The machinery is the experimental comparison between two prompt conditions on the same five test cases. The target formalism is the Dimensional Fact Model (DFM), a graph of facts, measures, dimensions, attributes, and many-to-one roll-up dependencies, with annotations for descriptive, optional, and non-additive constructs; schemas are exchanged in a tagged YAML format. The basic condition uses a simple instruction prompt with role, format, and task; the improved condition adds a procedure component for each refinement step and an example component using two worked cases, one following the chain-of-thought style. Refinement quality is measured by manually counting, for each step, deviations from a set of feasible refinements that the author defined for each draft schema. The engine of the result is the error-count reduction produced by the added prompt components, particularly the procedural rules for removing attributes and the few-shot examples.

What would settle it

Give a second group of designers the same five draft schemas and ask them, independently, to define the feasible refinements and count errors for the basic and improved prompts; if their counts place the improvement outside the reported 9-to-4 range, the central claim is not robust. Alternatively, rerun the RQ.3 protocol with temperature above 0 or with a different LLM and check whether one corrective prompt still fixes all residual errors.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central finding is that prompt engineering materially changes whether ChatGPT can refine a draft DFM schema: the average number of refinement errors per test case falls from 9 to 4 when the instruction prompt includes detailed procedures and examples, and the average errors per multidimensional concept fall from 0.5 to 0.2. The model is good at renaming attributes and labeling measure additivity even with basic prompts, and weak at identifying descriptive or discretizable attributes, recognizing shared hierarchies, and correctly rewiring dependencies after removing an attribute. With the improved prompt, additivity and optional-attribute errors disappear, while shared hierarchies and descriptive attributes remain the main trouble spots; all residual errors in the tests could be corrected by one additional, often generic, prompt. The author still judges the residual error rate too high to let end-users refine schemas without designers, but high enough to support designers significantly.

Load-bearing premise

The quantitative conclusion stands or falls on the assumption that the author's manual count of deviations from a personally defined set of feasible refinements is a reliable measure of refinement quality, even though refinement is acknowledged to be subjective and multiple ground truths are defensible.

Editorial extensions

If this is right

  • Organizations using supply-driven cube design can hand a large share of schema refinement to an LLM-assisted end-user, with a designer only checking the result.
  • Prompt quality becomes a first-class design artifact: the same five test cases go from 9 to 4 errors per schema purely by changing the instruction prompt.
  • Residual errors concentrate in shared hierarchies and descriptive or discretized attributes, so those steps are the ones to keep human review focused on.
  • One corrective prompt after the first answer is often enough to repair remaining errors, making an interactive refinement loop practical.
  • Basic LLM output cannot be trusted as final: GPT-4o mixes conceptual and logical design levels, so a human verification step remains non-negotiable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension: the error counts should be read as ranges rather than point estimates, because the feasible-refinement sets were defined by one author; independent sets could shift the 9-to-4 gap.
  • A natural generalization to test is whether the same prompt template transfers to other LLMs or to other conceptual formalisms; if the improvement is prompt-specific rather than model-specific, the 9-to-4 effect may shrink.
  • Because shared hierarchies cause a disproportionate share of residual errors, adding an explicit procedural rule for merging duplicate time-hierarchy nodes, analogous to the removal rule, would be a concrete next experiment that could plausibly push error counts below 4.
  • Computing precision and recall on node and arc matching alongside the manual count would make the conclusion much stronger; the author flags this as future work, and it is the most direct way to test the reliability of the measure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports an empirical study of GPT-4o (ChatGPT) as a facilitator for the refinement of draft Dimensional Fact Model (DFM) schemata produced by supply-driven design. It poses three research questions: whether ChatGPT understands multidimensional modeling and the DFM (RQ.1), whether it can perform six refinement steps with simple prompts (RQ.2), and whether prompt engineering improves that performance (RQ.3). Five test cases of increasing size and complexity are used; inputs and outputs are YAML-coded DFM schemata, and refinement quality is measured by manually counting errors against a per-case set of 'feasible refinements' defined by the author. The paper reports that the improved prompt reduces the average number of errors per test case from 9 to 4, that residual errors can be fixed with one additional prompt, and that some designer involvement is still necessary.

Significance. If its quantitative result held out-of-sample, this would be a useful and clearly described case study on LLM-assisted multidimensional conceptual design. The paper has real strengths: explicit research questions; a reasonably reproducible protocol (temperature 0, a fresh chat per case, a fixed YAML serialization, and published prompt templates); per-step error breakdowns; and a candid Section 4 that acknowledges subjectivity, low statistical power, and run-to-run variability. The qualitative findings, such as GPT-4o mixing conceptual and logical design levels and struggling with shared hierarchies, are informative for the community. However, the central quantitative claim is weakened by the fact that the improved prompt was incrementally tuned on the same five test cases used for the evaluation, so the reported 9-to-4 reduction is an in-sample result. The contribution is therefore better characterized as a preliminary demonstration on five development cases than as evidence of a general prompt-engineering capability.

major comments (4)
  1. [Section 3.5, Figures 8 and 10] The improved prompt was 'incrementally crafted' by 'progressively adding specific sentences to try to fix the residual (or new) errors' observed in RQ.2, and the RQ.3 error counts in Figures 8 and 10 are measured on the same five test cases listed in Table 1. The 9-to-4 average reduction and the 0.5-to-0.2 errors-per-concept reduction are therefore fitted, in-sample results. They show only that a prompt built by inspecting these five cases outperforms a generic prompt on those same cases; they do not establish that the prompt-engineering recipe improves refinement accuracy for unseen draft schemata. The paper should either add an out-of-sample evaluation (e.g., held-out draft schemata, or a pre-specified prompt fixed before seeing the test cases) or explicitly reframe the RQ.3 conclusion as a demonstration on the five development cases rather than a general capability claim.
  2. [Section 3.2.5] The only outcome measure is the author's manual error count against a set of 'feasible refinements' defined by the author for each draft schema. No inter-rater check is reported, and the set of feasible refinements is not fully specified in the paper; Section 4 itself acknowledges that multiple ground truths can be defined for each step. Because every quantitative conclusion in the paper is expressed in these counts, the reliability of the measure is load-bearing. At minimum, a second annotator should independently apply the same rubric, or the evaluation should be supplemented with objective precision/recall on nodes and arcs, as the authors themselves suggest in Section 4.
  3. [Abstract and Section 3.5] The claim that prompt engineering 'significantly improves' the accuracy of refinement is not backed by a statistical test. With five test cases and repeated measurements on the same cases, a paired non-parametric test or an effect-size estimate would be needed to support the word 'significantly.' Section 4 acknowledges low statistical power, but the Results section still asserts significance without qualification. The authors should either provide an appropriate test and effect size or soften the language to, for example, 'improved in all five tested cases.'
  4. [Section 3.5 and Abstract] The claim that 'all residual errors could be fixed via one additional prompt' is not supported by systematic data. The section gives two illustrative follow-up prompts and states that errors were fixed, but there is no per-case record of how many follow-up prompts were needed, how many errors remained after each iteration, or what counts as 'fixed.' Since the abstract presents this as one of the main results, the authors should add a per-case iteration log or at least a summary table of the number and type of residual errors before and after the additional prompt.
minor comments (4)
  1. [Section 3.2.3, TASK component] In the additivity instruction, the second occurrence of 'semi-additive' should be 'additive': as printed, the instruction says that a semi-additive measure should be labeled '(SUM-AVG)' and then immediately says that if it is semi-additive it should be left unchanged.
  2. [Section 3.3, RQ.1 discussion] The sentence 'the diagram suggested as an example ... as no attinence to a real DFM schema' contains a typo ('as no attinence' should likely be 'has no adherence' or 'has no relation'), and the discussion would benefit from a clearer explanation of which specific graphical conventions are violated.
  3. [Section 3.4, Discussion] The statement that 'the errors clearly tend to increase with the complexity of the draft schema' is supported only by visual inspection of Figure 6; a simple correlation coefficient or even an ordered table of per-case totals would make the claim precise and testable.
  4. [Section 3.2.1, Reproducibility] The paper reports only 'GPT-4o' without the exact model snapshot or the dates of the runs; given OpenAI's versioning practices, specifying the model version and the experiment dates would improve reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ.3's improved prompt was tuned on the same five test cases used to measure the 9-to-4 error reduction, so the headline improvement is a fitted result rather than an out-of-sample prediction.

  1. fitted input called prediction [Section 3.5 (RQ.3), Figures 8 and 10, with test cases from Table 1]
    "To answer RQ.3 we incrementally crafted an instruction prompt by first trying to address the main issues emerged in RQ.2, then progressively adding specific sentences to try to fix the residual (or new) errors. ... The results, in terms of number of errors made at each step, are summarized in Figure 8. It appears that prompt engineering can significantly improve the accuracy of refinement, with the average number of total refinement errors per test case decreasing from 9 to 4."

    The improved prompt is not an independent intervention. It was constructed by observing and repairing errors on the five test cases in Table 1, and the same five test cases were then used to compute the reported improvement (from 9 to 4 errors per test case, and from 0.5 to 0.2 errors per concept in Figure 10). The 9-to-4 reduction therefore measures how well a prompt hand-tuned on these inputs performs on those same inputs, not how well the prompt-engineering recipe generalizes to unseen draft schemata. The separate claim that residual errors can be fixed with one additional prompt is also verified only on the same tuning cases. Section 4 acknowledges subjectivity and low statistical power but does not acknowledge this train/evaluation overlap.

full rationale

This paper contains no formal derivation, so the classic circularity patterns based on equations or imported uniqueness theorems do not apply. RQ.1 and RQ.2 are independent probes of ChatGPT's baseline competences and are not circular: they report observed behavior on test cases without fitting any parameter. The load-bearing issue is confined to RQ.3: the improved prompt was incrementally crafted by fixing errors seen on the five test cases, and the same five test cases were then used to report the headline error reduction. That is a fitted-input-called-prediction step: the prompt is the fitted artifact, and the reported 9-to-4 improvement is evaluated on the fitting data. No out-of-sample validation is provided, so the quantitative improvement is not independent evidence of generalization. The author's own validity-threat discussion mentions subjectivity and low statistical power but not this overfitting risk. Self-citations to the author's prior work ([1], [2], [19]) are background references and are not load-bearing for the empirical claims. Overall score 6 reflects partial circularity: the qualitative findings about ChatGPT's refinement behavior remain informative, but the central quantitative improvement claim reduces, to a meaningful extent, to a fit on the evaluation set.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the DFM refinement task definition, the manual error-counting metric, the representativeness of GPT-4o at temperature 0, and the representativeness of five course-derived test cases. No numeric free parameters or invented entities are introduced. The strongest non-empirical assumption is that the author's own error count is a valid measure of refinement quality.

assumptions (4)
  • domain assumption The refinement activities listed in the introduction (renaming, additivity labeling, descriptive/discretization, optional attributes, time hierarchy completion, removal of uninteresting attributes) are the correct and complete set of tasks for DFM refinement.
    Section 1 and RQ.2; the task definition comes from the author's prior work on DFM, so the experiment only tests this specific formulation, not refinement in general.
  • domain assumption Manual error counting against a manually built set of feasible refinements is a valid and sufficiently objective measure of refinement quality.
    Section 3.2.5 explicitly says refinement is subjective and multiple ground truths are possible, and Section 4 lists the reliability of this measure as a validity threat.
  • domain assumption GPT-4o with temperature set to 0 is sufficiently representative and reproducible for the claims made.
    Section 3.2.1 sets temperature to 0, but Section 4 concedes that some variability still occurs and tests were not repeated.
  • domain assumption The five test cases, derived from exercises in a master course, are representative of supply-driven design scenarios.
    Section 3.2.4; the author acknowledges low statistical power and the need for additional cases in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using ChatGPT to refine draft conceptual schemata in supply-driven design of multidimensional cubes." pith.science (2026). https://pith.science/paper/TNW6NKJ4

@misc{pith2026250202238,
  author       = {Pith},
  title        = {Pith review of: Using ChatGPT to refine draft conceptual schemata in supply-driven design of multidimensional cubes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNW6NKJ4}},
  note         = {Machine review of arXiv:2502.02238}
}
read the original abstract

Refinement is a critical step in supply-driven conceptual design of multidimensional cubes because it can hardly be automated. In fact, it includes steps such as the labeling of attributes as descriptive and the removal of uninteresting attributes, thus relying on the end-users' requirements on the one hand, and on the semantics of measures, dimensions, and attributes on the other. As a consequence, it is normally carried out manually by designers in close collaboration with end-users. The goal of this work is to check whether LLMs can act as facilitators for the refinement task, so as to let it be carried out entirely -- or mostly -- by end-users. The Dimensional Fact Model is the target formalism for our study; as a representative LLM, we use ChatGPT's model GPT-4o. To achieve our goal, we formulate three research questions aimed at (i) understanding the basic competences of ChatGPT in multidimensional modeling; (ii) understanding the basic competences of ChatGPT in refinement; and (iii) investigating if the latter can be improved via prompt engineering. The results of our experiments show that, indeed, a careful prompt engineering can significantly improve the accuracy of refinement, and that the residual errors can quickly be fixed via one additional prompt. However, we conclude that, at present, some involvement of designers in refinement is still necessary to ensure the validity of the refined schemata.

Figures

Figures reproduced from arXiv: 2502.02238 by the authors.

Figure 1
Figure 1. YAML code to represent a draft DFM schema (left) and a refined one (right) • dependencies, to introduce the list of FDs; • from, to, and role to denote the start and end nodes of each FD and, optionally, its role in shared hierar￾chies; • descriptive, to introduce the list of descriptive at￾tributes; • optional, to introduce the list of optional attributes. Examples of the YAML code used to represent draft and refin… view at source ↗
Figure 2
Figure 2. A DFM schema according to ChatGPT (left) and a real DFM schema (right) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Draft schema for test case C2 (only the first four letters of relation names are shown) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Partially refined schema for test case C2 with basic prompt (renaming and additivity) ChatGPT is quite good at dealing with additivity. This is surprising, considering that this task is often not easy even for end-users. The main errors we found (in C3, C4, and C5) wer…
Figure 5
Figure 5. Figure 5: Refined schema for test case C2 with basic prompt (all steps) 0 1 2 3 4 5 6 C1 C2 C3 C4 C5 RQ.2 Renaming Additivity Descriptive Optional Time hier. Removal [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Number of errors in the refinement of draft DFM schemata (basic prompts) 3.5. Answer to RQ.3: Improved refinement To answer RQ.3 we incrementally crafted an instruction prompt by first trying to address the main issues emerged in RQ.2, then progressively adding specifi…
Figure 10
Figure 10. Figure 10: Average number of refinement errors per multidimen￾sional concept In this work we have investigated the capabilities of ChatGPT to cope with a specific task in conceptual design, namely, the refinement of draft DFM schemata obtained by supply-driven conceptual design …
Figure 9
Figure 9. Figure 9: Refined schema for test case C5 with basic (top) and improved prompts (bottom) hierarchies and of descriptive/discretized attributes, as well as to the removal of uninteresting attributes. In our tests, all these errors could be fixed via an additional prompt that eith…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    Golfarelli, S

    M. Golfarelli, S. Rizzi, Data warehouse design: Modern principles and methodologies, McGraw-Hill, 2009

    work page 2009

  2. [2]

    Antonelli, S

    L. Antonelli, S. Bimonte, S. Rizzi, Multidimensional modeling driven from a domain language, Autom. Softw. Eng. 30 (2023) 6

    work page 2023

  3. [3]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. C. Grundy, H. Wang, Large language mod- els for software engineering: A systematic literature review, CoRR (2023)

    work page 2023

  4. [4]

    W. Ma, S. Liu, W. Wang, Q. Hu, Y. Liu, C. Zhang, L. Nie, Y. Liu, LLMs: Understanding code syntax and seman- tics for code analysis, CoRR abs/2305.12138 (2023)

    arXiv 2023

  5. [5]

    White, S

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, D. C. Schmidt, ChatGPT prompt patterns for improving code quality, refactoring, requirements elicitation, and software design, CoRR abs/2303.07839 (2023)

    arXiv 2023

  6. [6]

    H. Fill, J. Cabot, W. Maass, M. van Sinderen, AI-driven software engineering - the role of conceptual modeling, Enterp. Model. Inf. Syst. Archit. Int. J. Concept. Model. 19 (2024)

    work page 2024

  7. [7]

    H. Fill, P. Fettke, J. Köpke, Conceptual modeling and large language models: Impressions from first experi- ments with ChatGPT, Enterp. Model. Inf. Syst. Archit. Int. J. Concept. Model. 18 (2023) 3

    work page 2023

  8. [8]

    Lutze, K

    R. Lutze, K. Waldhör, Generating specifications from requirements documents for smart devices using large language models (LLMs), in: Proc. HCI, Washington, DC, USA, 2024, pp. 94–108

    work page 2024

Show all 26 references
  1. [9]

    W. Hariri, Unlocking the potential of ChatGPT: A com- prehensive exploration of its applications, advantages, limitations, and future directions in natural language processing, CoRR abs/2304.02017 (2023)

    2023 arXiv

  2. [10]

    Z. Zhou, J. Li, Z. Zhang, J. Yu, H. Duh, Examining how the large language models impact the conceptual design with human designers: A comparative case study, Int. J. Hum. Comput. Interact. (2024) 1–17

    2024

  3. [11]

    K. Ma, D. Grandi, C. McComb, K. Goucher-Lambert, Conceptual design generation using large language models, CoRR abs/2306.01779 (2023)

    2023 arXiv

  4. [12]

    L. Chen, Y. Tsang, Q. Jing, L. Sun, A LLM-augmented morphological analysis approach for conceptual de- sign, in: Proc. DRS, Boston, USA, 2024, pp. 1–19

    2024

  5. [13]

    S. J. Ali, I. Reinhartz-Berger, D. Bork, How are LLMs used for conceptual modeling? An exploratory study on interaction behavior and user perception, in: Proc. ER, Pittsburgh, USA, 2024, pp. 257–275

    2024

  6. [14]

    Romero, A

    O. Romero, A. Abelló, Data-driven multidimensional design for OLAP, in: Proc. SSDBM, Portland, OR, USA, 2011, pp. 594–595

    2011

  7. [15]

    Jovanovic, O

    P. Jovanovic, O. Romero, A. Simitsis, A. Abelló, D. May- orova, A requirement-driven approach to the design and evolution of data warehouses, Inf. Syst. 44 (2014) 94–119

    2014

  8. [16]

    F. D. Tria, E. Lefons, F. Tangorra, Hybrid methodol- ogy for data warehouse conceptual design by UML schemas, Inf. Softw. Technol. 54 (2012) 360–379

    2012

  9. [17]

    Romero, A

    O. Romero, A. Abelló, Automatic validation of re- quirements to support multidimensional design, Data Knowl. Eng. 69 (2010) 917–942

    2010

  10. [18]

    Niemi, J

    T. Niemi, J. Nummenmaa, P. Thanisch, Construct- ing OLAP cubes based on queries, in: Proc. DOLAP, Atlanta, Georgia, USA, 2001, pp. 9–15

    2001

  11. [19]

    Bimonte, L

    S. Bimonte, L. Antonelli, S. Rizzi, Requirements-driven data warehouse design based on enhanced pivot tables, Req. Eng. 26 (2021) 43–65

    2021

  12. [20]

    R. Nair, C. Wilson, B. Srinivasan, A conceptual query- driven design framework for data warehouse, Int. Jour. of Computer and Information Engineering 1 (2007) 62–67

    2007

  13. [21]

    Romero, A

    O. Romero, A. Abelló, A survey of multidimensional modeling methodologies, Int. J. Data Warehous. Min. 5 (2009) 1–23

    2009

  14. [22]

    F. D. Tria, E. Lefons, F. Tangorra, Cost-benefit analysis of data warehouse design methodologies, Inf. Syst. 63 (2017) 47–62

    2017

  15. [23]

    K. Chen, Y. Yang, B. Chen, J. A. H. López, G. Muss- bacher, D. Varró, Automated domain modeling with large language models: A comparative study, in: Proc. MODELS, Västerås, Sweden, 2023, pp. 162–172

    2023

  16. [24]

    T. B. Brown, et al., Language models are few-shot learners, in: Proc. NeurIPS, 2020

    2020

  17. [25]

    Wei, et al., Chain-of-thought prompting elicits rea- soning in large language models, in: S

    J. Wei, et al., Chain-of-thought prompting elicits rea- soning in large language models, in: S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Proc. NeurIPS, New Orleans, LA, USA, 2022

    2022

  18. [26]

    Cámara, J

    J. Cámara, J. Troya, L. Burgueño, A. Vallecillo, On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML, Softw. Syst. Model. 22 (2023) 781–793

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.