{"id":"13929f03-e94a-4de0-87a1-b081dbaac61f","arxiv_id":"2505.09116","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A tool that moves a learner's classes to the instructor's positions is associated with higher diagram similarity, but the study does not separate this effect from a concurrent color-hint feature.","lead":"This paper describes a classroom tool that rearranges a novice's UML class diagram to mirror the instructor's layout, aiming to give hints without revealing the answer. An experiment with 20 students found higher similarity scores for the group using the tool, but that group also received color-coded name feedback, so the layout effect is not isolated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported comparison cannot attribute the gains to automatic layout conversion: the experimental condition added color-based name feedback (III-D/III-E) that the control condition apparently lacked, and the outcome metric measures exactly what the color hints target.","rationale":"The paper's intended claim is causal: automatic layout conversion improved novices' class diagrams. For that claim to hold, the experimental group's advantage must be attributable to layout conversion rather than to any other feature added in the treatment. That condition is not met. Section III-D describes adding a color-based name-feedback function in the same implementation, and Section III-E explicitly lists it as a separate feedback channel (E) delivered to learners. Section IV-B differentiates the control group solely by the absence of the layout-conversion function; it never states whether the control tool included the color feedback. Because CDS/CSall/RSall are name-similarity metrics, color hints that mark names as red/black/blue directly provide the information the metrics reward, so the observed CDS p=0.0235 and RSall p=0.0050 differences could plausibly come from the color channel alone. This is the same load-bearing concern the reader identified. I do not find a stronger independent objection: the small sample size and uncorrected multiple comparisons are real but secondary, and the absence of code or data weakens reproducibility but does not by itself refute the central claim. The decisive remedy is a factorial design separating layout conversion from color feedback. The reader's REJECT verdict stands.","tokens_in":13685,"tokens_out":4639,"duration_ms":47605,"concrete_test":"Run a factorial experiment with four arms under the same one-hour exercise: (1) baseline KIfU with similarity score only, (2) automatic layout conversion with color feedback disabled, (3) color feedback only without layout conversion, and (4) both. If arm (2) does not significantly improve CDS/RSall over baseline while arm (3) replicates the original gains, the layout-conversion claim is refuted; if arm (2) shows the gain and arm (3) does not, the concern is settled in the paper's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing inference is that the experimental group's higher CDS (p=0.0235) and RSall (p=0.0050) are caused by automatic layout conversion. Section III-D states that the authors 'added a color-based feedback function for class names and attribute names' as part of the same implementation, and Section III-E lists this color feedback (E) alongside the layout effects (A-D) as feedback obtained by tool users. The control condition is defined in Section IV-B only as 'using KIfU without the automatic layout conversion function'; it is not stated that the control group received the same color feedback. If the color feedback was absent for the control group, the two groups differ in two independent feedback channels, not one. The color hints are especially threatening because the outcome metrics (Tanaka et al.'s CDS, CSall, RSall) are computed from name similarities of classes, attributes, and relationships; red/black/blue coloring directly tells the learner which names already match the correct solution. This alone could raise CDS and RSall, since correctly identified classes also enable correct relationship matches. Section VI's threats to validity mention only the similarity metric and problem complexity and omit this treatment confound. Thus the central claim—that layout conversion, rather than the bundled name-color feedback, was effective—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method for automatically converting the layout of a learner's class diagram to match that of an instructor's correct example, implemented as an extension of the KIfU tool. The method also includes a color-based feedback function that colors class and attribute names according to their name similarity to the correct model. The authors report an experiment with 20 students randomly assigned to a group with the automatic layout conversion function and a control group without it, measuring outcomes with Tanaka et al.'s class diagram similarity (CDS), class similarity (CSall), and relationship similarity (RSall). They find significantly higher CDS (p=0.0235) and RSall (p=0.0050) in the experimental group and conclude that automatic layout conversion is effective feedback for novice modelers.","tokens_in":13885,"tokens_out":5558,"duration_ms":46957,"significance":"If the causal claim were valid, the paper would make a useful contribution to automated feedback in modeling education, with a concrete tool implementation and an empirical evaluation. The strengths are the tool implementation, the use of an established similarity metric, and the attempt to measure learning processes via similarity transitions. However, the experiment does not isolate the layout conversion from the separately added color feedback, and this confound directly threatens all of the paper's positive conclusions. The statistical analysis also relies on multiple uncorrected t-tests. Within the current design, the central claim is not established.","major_comments":[{"comment":"The experimental manipulation is confounded. The experimental group received both automatic layout conversion and color-based name feedback, while the control group received neither. The control group is defined only as using KIfU without the automatic layout conversion function, with no statement that the color feedback was also present in the control condition. Because the outcome metrics CDS, CSall, and RSall are computed from name similarities (Section II-C), the color feedback alone could plausibly produce the observed improvements in CDS and RSall. Consequently, the significant differences reported in Section V-A do not establish that automatic layout conversion, rather than the color hints, was effective. This is the central claim of the paper, so the experiment as reported cannot support it.","section":"§III-D, §III-E, §IV-B"},{"comment":"The paper reports the results of multiple t-tests (CDS, CSall, RSall, per-class CS for six classes, and multiple time-point comparisons in Figure 7) without any adjustment for multiple comparisons. At α=0.05 with this many tests, the probability of at least one false positive is substantially higher than 0.05. For instance, the CDS p-value of 0.0235 is not strong evidence once the number of tests is taken into account. The authors should either apply a correction (e.g., Bonferroni or FDR) or clearly state the exploratory nature of these tests.","section":"§V-A, Figures 3–5, Table II"},{"comment":"The automatic layout conversion identifies corresponding classes using the same Tanaka et al. similarity measure (with a threshold of CS≥0.4) that is later used to compute the evaluation metrics CDS, CSall, and RSall. This means the experimental condition is directly optimized against the evaluation metric, while the control condition is not. This design choice inflates the likelihood of finding a difference and limits the conclusions to this specific metric. The threat-to-validity section should address this alignment.","section":"§III-A and §II-C"},{"comment":"The threats-to-validity discussion lists only the choice of similarity metric and the complexity of the correct example. It omits the treatment confound described above and the multiple-comparison issue. Both are more serious threats to the validity of the reported conclusions than the two issues named.","section":"§VI"}],"minor_comments":[{"comment":"The axis labels appear garbled (e.g., '/aj17/aj15/aj17' and '/aj36/aj40'), likely due to a PDF rendering artifact; please provide legible figures.","section":"Figures 3, 4, 5, and 7"},{"comment":"In RQ1, 'effect' should be 'affect' in 'How does the layout of the correct example effect the creation'.","section":"§IV-A"},{"comment":"Please clarify explicitly whether the control group received the color-based name feedback; as written, the group definitions do not specify this, which is essential for interpreting the results.","section":"§IV-B"},{"comment":"The statement 'This indicates that learners naturally avoid overlapping elements without being explicitly told about layout rules' is based on the observation that no submitted diagrams had overlaps, but overlapping elements may be avoided by default tool behavior; please clarify.","section":"§V-A"}],"recommendation":"reject","confidential_remarks":"The core issue is not a presentation problem but a fundamental experimental design flaw: the treatment is confounded, and the control condition is underspecified. A reanalysis of the existing data cannot separate the two feedback channels; a new experiment with a 2x2 design (layout conversion on/off × color feedback on/off) would be needed. Given that the paper's central claim concerns layout conversion, the current evidence does not support it. I therefore recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: interesting idea, weak test. The method—automatically rearranging a learner's class diagram to match the instructor's layout—is genuinely new in this literature, and the implementation sounds straightforward. The authors also report a couple of suggestive observations: the 'Order Item' class difference and the correlation between how often students used the conversion and their final similarity. I'd believe that layout feedback helps some students.\n\nThe problem is the experiment cannot isolate the effect. Section III-D describes adding a color-based feedback function (red/black/blue on class and attribute names) as part of the same implementation. Section III-E lists this color feedback (E) alongside the layout effects as what users get when they press the check button. The control group is defined only as 'KIfU without the automatic layout conversion function' (Section IV-B). It is never stated that the control had the color feedback. If it didn't, the two groups differ in two ways, and the color feedback is a very likely confound: the outcome measures (CDS, CSall, RSall) are all computed from name similarities, and the coloring directly tells learners which names already match the correct answer. That alone could drive the significant gains in CDS and RSall, regardless of layout. The threats-to-validity section (VI) mentions only the similarity metric and problem complexity—it misses this.\n\nOther issues are more ordinary: n=20, many t-tests without correction, and no code or data. The multiple-comparison issue is mildly concerning; the confound is the one that matters.\n\nIf the control did in fact receive the same color feedback, the paper should say so plainly. If it didn't, the central claim is not supported. Either way, the current write-up does not establish that layout conversion, rather than the bundled name-color feedback, did the work. That's a fixable problem: separate arms for layout-only and color-only would sort it out.\n\nWho is this for? People building feedback tools for UML modeling exercises. The idea deserves a serious referee; the evidence, as presented, doesn't yet support the abstract's claim. I'd send it back for a redesign or, at minimum, a clear statement of what the control group actually received.\n\nRecommendation: peer review, but with a major-revision bar. If the authors can show the control had color feedback, or run the cleaner experiment, the paper could be a useful small contribution.","headline":"The paper's novel hook—using the instructor's layout as a hint—is worth testing, but this experiment can't attribute the gains to layout because the experimental group also got color-coded name feedback the control may not have had.","tokens_in":14437,"tokens_out":4623,"would_cite":false,"duration_ms":41176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic layout conversion raises novices' class-diagram similarity in a one-hour experiment.","keywords":["class diagram","layout","modeling exercises","novice learners","automatic layout conversion","similarity feedback","UML class diagram","relationship similarity"],"falsifier":"Run the same one-hour exercise with a third group that receives only the red/black/blue name-color feedback and no automatic layout conversion; if that group's CDS and RSall match the experimental group's, the paper's attribution of the improvement to layout conversion is falsified. A complementary check: replace the instructor's coordinates with randomly chosen positions and see whether the benefit disappears.","tokens_in":13455,"feed_emoji":"📐","tokens_out":7847,"duration_ms":69422,"temperature":0.7,"pith_summary":"This paper tries to establish that automatically moving a novice's class diagram onto the instructor's class coordinates is effective feedback during a one-hour modeling exercise. The support matters because modeling exercises give learners little real-time feedback, and a bare similarity score can make them abandon and redraw their diagram instead of fixing it. The proposed method rearranges only the learner's existing classes to the instructor's layout while keeping relationships attached, so the learner sees which classes have counterparts and where gaps sit without being handed the correct answer. In a randomized comparison with twenty novices, the group with layout conversion achieved significantly higher overall diagram similarity and relationship similarity than the control group, while class-only similarity did not differ significantly.","feed_headline":"Moving novices' diagrams to the instructor's layout boosts scores","feed_subtitle":"In a one-hour experiment, layout feedback significantly improved overall diagram and relationship similarity.","key_machinery":"The mechanism is automatic layout conversion: the learner's classes are matched to the instructor's classes using per-class similarity CS with a threshold of 0.4 and lexicographic tie-breaking, the matched classes are moved to the coordinates of their counterparts in the correct diagram, unmatched classes are stacked at the top-left corner (0,0), and existing relationships are preserved because they are stored as endpoint references rather than coordinates. A bundled secondary feedback colors each class and attribute name red for an exact name match, black for a partial match, and blue for a mismatch. The layout conversion carries the argument by letting the learner see, from position alone, which of their classes belong in the correct model and where related classes sit, making gaps and candidate relationships visible without displaying the correct diagram.","core_discovery":"The paper's central claim is that the layout of the correct answer is itself a usable feedback channel. Concretely, in the authors' experiment, the experimental group's class-diagram similarity to the instructor's model (CDS) was significantly higher than the control group's ($p = 0.0235$), and so was relationship similarity RSall ($p = 0.0050$); class similarity CSall showed no significant difference ($p = 0.2886$). The authors interpret this as showing that automatic layout conversion helped novices mainly by narrowing the space of possible relationships and by suggesting missing elements through whitespace, rather than by directly teaching which classes exist.","pith_inferences":["Editorial inference: because the red/black/blue name-color feedback was bundled with layout conversion in the experimental tool, the design does not yet isolate layout conversion from color feedback; a follow-up comparing layout conversion with and without the color hints could determine whether the two reinforce each other.","Editorial inference: if the active mechanism is the layout's narrowing of relationship options, then rearranging classes to randomly chosen positions should produce weaker or no gains compared with rearranging to the instructor's coordinates, which is testable.","Editorial inference: the experiment measured similarity to the same instructor's diagram during a single exercise, so it remains open whether the skill transfers to new modeling tasks without layout support; a delayed post-test would answer that.","Editorial inference: the observed correlation between number of uses and final similarity could reflect motivated learners using the function more rather than the function causing improvement; a fixed-feedback-schedule experiment would separate these explanations."],"forward_implications":["Layout can act as a hint that does not reveal the correct answer directly, so learners can check and revise repeatedly during an exercise without being shown the instructor's model.","The benefit is concentrated in relationship creation: relationship similarity RSall differed significantly while class-only similarity CSall did not, so the layout appears to help learners see which associations are possible.","The effect appears in the later phase of the exercise, around and after the 20-minute mark, when learners are correcting their diagrams, suggesting that layout feedback helps resolve the stuck state that similarity scores alone did not.","The number of times learners invoked the conversion correlated strongly with final diagram similarity ($r = 0.7877$), indicating that repeated use of the feedback is associated with larger improvements.","The approach can be automated, which matters because manual layout feedback would require roughly an hour of instructor effort per learner for the volume of feedback observed."],"supporting_citations":[{"why":"Defines the class-diagram similarity metric CDS with its CSall and RSall components, which the experiment uses as its outcome measures.","marker":"[27]"},{"why":"Introduced similarity-score feedback for modeling exercises, the approach this paper extends and the baseline that motivated layout feedback.","marker":"[11]"},{"why":"Provides empirical evidence that layout quality affects diagram comprehension, with stronger benefits for novices.","marker":"[26]"},{"why":"Supplies the nameSimilarity function used inside the similarity metric for matching class and attribute names.","marker":"[31]"},{"why":"Describes the drawing tool whose snapshot mechanism the automatic layout conversion is built into.","marker":"[28]"},{"why":"Documents the high rate of mistakes in learners' UML class diagrams, motivating the need for feedback support.","marker":"[5]"}],"fun_headline_variants":["Layout feedback reshapes novices' class diagrams","Instructor layout as feedback improves diagram similarity","Auto layout conversion aids novice diagram relationships","Layout conversion feedback narrows relationship errors","Teaching class diagrams via layout conversion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the higher similarity scores came from the automatic layout conversion rather than from the red/black/blue color feedback on class and attribute names, because the experiment's control group did not receive that color feedback separately.","fun_headline_variants_meta":{"raw":{"variants":["Layout feedback reshapes novices' class diagrams","Instructor layout as feedback improves diagram similarity","Auto layout conversion aids novice diagram relationships","Layout conversion feedback narrows relationship errors","Teaching class diagrams via layout conversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":2046,"prompt_tokens":792,"completion_tokens":1254,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":1191}},"tokens_in":408,"tokens_out":1254,"duration_ms":9225,"temperature":1.0,"reasoning_tokens":1191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:39:31.039786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same one-hour exercise with a third group that receives only the red/black/blue name-color feedback and no automatic layout conversion; if that group's CDS and RSall match the experimental group's, the paper's attribution of the improvement to layout conversion is falsified. A complementary check: replace the instructor's coordinates with randomly chosen positions and see whether the benefit disappears.","supporting_citations":[{"cited_title":"of the IPSJ/SIGSE Software Engineering Symposium (SES2015), Aug","cited_arxiv_id":null,"evidence_quote":"Defines the class-diagram similarity metric CDS with its CSall and RSall components, which the experiment uses as its outcome measures."},{"cited_title":"of the 12th Joint Conference on Knowledge-Based Software Engineering (JCKBSE 2018) Corfu, Greece","cited_arxiv_id":null,"evidence_quote":"Introduced similarity-score feedback for modeling exercises, the approach this paper extends and the baseline that motivated layout feedback."},{"cited_title":"of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC2012) , Oct","cited_arxiv_id":null,"evidence_quote":"Provides empirical evidence that layout quality affects diagram comprehension, with stronger benefits for novices."},{"cited_title":"and Stroulia, E.: UMLDiff: an algorithm for object-oriented design differencing, Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the nameSimilarity function used inside the similarity metric for matching class and attribute names."},{"cited_title":"E101.D, No","cited_arxiv_id":null,"evidence_quote":"Describes the drawing tool whose snapshot mechanism the automatic layout conversion is built into."},{"cited_title":"of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) , May","cited_arxiv_id":null,"evidence_quote":"Documents the high rate of mistakes in learners' UML class diagrams, motivating the need for feedback support."}],"review_version":1}