{"id":"c9f7695e-66ec-43b0-8319-091bf390179b","arxiv_id":"2412.15254","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"RIRO combines input reformulation and output reshaping to report higher BLEU and ROUGE scores on a small user story to test case generation task.","lead":"This paper introduces RIRO, a two-layer setup that rewrites inputs and touch-up outputs to help fine-tuned language models perform better when training data is scarce. It is tested on turning user stories into software test cases, and the authors report better scores than fine-tuning alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The output-reshaping layer in §4.4 is unspecified and may rewrite generated test cases toward the reference strings, so the Table 1 BLEU/ROUGE gains could be circular; a reference-blind rerun is needed.","rationale":"Reading the paper in good faith, the authors propose a layered pipeline for data-scarce generation: reformulate inputs, QLoRA-fine-tune, then reshape outputs. The central empirical claim is that the full pipeline is superior on all metrics in Table 1. For that claim to be meaningful, the output-reshaping layer must not trivially move the generated text toward the reference string. The paper gives no implementation details for Reshape, no prompts, no failure examples, and no ablation numbers for the variants defined in Section 4.5. This is the weakest load-bearing point because BLEU and ROUGE reward lexical overlap, and an unspecified post-processor can game those metrics directly. The Levenshtein result strengthens the concern, since the largest drop appears exactly where reference-aware rewriting would show up. The reader's verdict identifies essentially the same circularity risk, and the correction would be to expose the reshaper's behavior and rerun evaluation with references withheld. Other issues (missing baseline comparisons, no error bars, label contradictions in Figure 1 versus Section 5, no code) reinforce rejection but are secondary; the reference-blind reshaper check is the one test that would settle whether the headline result is real.","tokens_in":8013,"tokens_out":3965,"duration_ms":35500,"concrete_test":"Require the authors to release the exact prompts, checkpoints, and inference code for both fr(x) and Reshape(y), then rerun Table 1 under a reference-blind condition: keep the reference test cases out of every prompt, few-shot example, and training set used by the reshaper, and compare the LLM-RF (no reshaping) and LLM-RFR (full pipeline) variants from Section 4.5. If the BLEU/ROUGE gap between LLM-RF and LLM-RFR shrinks to noise under this condition, or if Reshape's output is lexically closer to the reference than to the raw model output, the central claim is driven by output rewriting rather than by the RIRO architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the full RIRO pipeline outperforms fine-tuning alone (Table 1, Section 5) is load-bearing on a single unexamined component: the output-reshaping step y' = Reshape(y) defined only symbolically in Section 4.4. The paper does not state what Reshape is, whether it is an LLM call, what prompt it uses, whether it has any access to reference test cases, or how it is trained. Because BLEU and ROUGE are n-gram overlap metrics, a reshaper that rewrites outputs toward the format or lexical content of the references can inflate all reported gains without any improvement in the underlying fine-tuned model. The Levenshtein Distance result in Table 1 is especially telling: RIRO drops from 1157.620 (Phi-2) and 1157.080 (Reshaping) to 1000.880, meaning its outputs are far closer to the reference strings at the character level, exactly the signature of a post-processor that has reference-like target text available. The ablation variants in Section 4.5 (LLM-RF without reshaping and LLM-FR without reformulation) would isolate this effect, but their results are never reported; Table 1's columns are instead named Reshaping, Refining, and RIRO, with labels that conflict with Figure 1. Without the implementation of Reshape and a reference-blind evaluation, the comparison is circular rather than evidence that input normalization plus QLoRA fine-tuning is what drives the improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RIRO, a pipeline intended to improve LLM fine-tuning in data-scarce settings, applied to generating test cases from user stories. The pipeline consists of an LLM-based input reformulation step (x' = fr(x)), QLoRA fine-tuning, and an LLM-based output reshaping step (y' = Reshape(y)). The paper claims that the full pipeline (RIRO) outperforms its own variants and the fine-tuned baseline across BLEU, ROUGE-1/2/L, Levenshtein distance, and cosine similarity, based on Table 1. The manuscript also introduces a benchmark built from a subset of the 'user story neodataset.' No code, data, or detailed experimental protocol is provided.","tokens_in":8314,"tokens_out":5966,"duration_ms":45192,"significance":"If the claimed gains were established, the idea of wrapping a fine-tuned LLM with input normalization and output reshaping would be a pragmatic contribution for low-resource applications, and the introduced benchmark could be useful to the community. However, the manuscript as written does not establish the central claim: the output-reshaping step is undefined, the ablation variants are not reported, the evaluation lacks any protocol details, and the table's component labels contradict the architecture diagram. The paper provides no derivations, machine-checked artifacts, or reproducible code; its only falsifiable output is Table 1, which cannot be checked or interpreted. The significance of the work is therefore currently unassessable.","major_comments":[{"comment":"The output-reshaping function y' = Reshape(y) is never specified: the paper does not state whether it is an LLM call, what prompt or model it uses, whether it is fine-tuned, or whether it has access to the reference test cases. This is load-bearing because RIRO's Levenshtein distance drops from 1157.080 (Reshaping) and 1420.500 (Refining) to 1000.880, which is exactly the pattern expected if a post-processor rewrites outputs toward reference-like strings. A reference-blind evaluation (or at minimum a generic post-processing baseline without references) is required to rule out circularity in the reported BLEU, ROUGE, and Levenshtein gains.","section":"§4.4, Table 1"},{"comment":"The evaluation is not reproducible: no dataset size, train/validation/test split, number of test examples, QLoRA rank, learning rate, batch size, epochs, or decoding parameters are reported, and there are no error bars or statistical significance tests. The differences between some configurations are very small (e.g., ROUGE-2: 0.147 vs 0.149), so the claim that RIRO 'demonstrates superior performance across all metrics' is unsupported. The abstract also promises results for Falcon 7B and Falcon 1B, but only Phi-2 results appear in Table 1.","section":"§5, Table 1"},{"comment":"The ablation study is described in Section 4.5 with variants LLM-RFR, LLM-RF, and LLM-FR, but none of these variants appears in Table 1. Instead, Table 1 lists 'Reshaping,' 'Refining,' and 'RIRO,' and Section 5 labels Reshaping as input-focused and Refining as output-focused, which is the opposite of Figure 1, where (a) Refining LLM performs input normalization and (b) Reshaping LLM performs output reshaping. This contradiction makes it impossible to determine which component drives the improvement.","section":"§4.5, §5, Figure 1"},{"comment":"The dataset description is incomplete: 'a subset of the user story neodataset' with no size, split, filtering criteria, or an explanation of how the issue fields (title, description, story points) are converted into user stories and reference test cases. Similarly, the reformulation function fr is defined only symbolically, with no implementation, prompt, target format, or example. Without these details, the benchmark is ill-specified and the experiment cannot be replicated or independently evaluated.","section":"§3, §4.2"}],"minor_comments":[{"comment":"The sentence 'Through fine-tuning models like Phi-2, Falcon 7B, and Falcon 1B, with Phi-2 outperforming the others' is a sentence fragment and should be revised.","section":"Abstract"},{"comment":"The phrase 'working in Adjective way' appears to be a typo; likely 'additive' or 'complementary' was intended.","section":"§1"},{"comment":"The introduction says 'RIRO is composed of three model architectures' but the abstract and methodology describe a two-layer architecture; the terminology should be harmonized across sections.","section":"§1, §4.5"},{"comment":"The table caption and surrounding text should use the same names as Figure 1 for the three architectures, or explicitly state why different names are used.","section":"§5"},{"comment":"The paper claims superiority over 'existing state-of-the-art methods' but provides no external baseline; at least one non-LLM or standard LLM baseline from the related work should be included for comparison.","section":"§5"},{"comment":"Reference 9 (QLoRA) contains a URL with a line break ('URLhttps://arxiv.org/abs/2305.14314'); the formatting should be corrected.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript is far below the standard for a peer-reviewed NLP venue: the core experimental evidence is a single table with no protocol, and the main pipeline component is underspecified. The circularity concern raised by the reader is real and should be addressed before any resubmission. The citation list contains many self-citations to unrelated topic areas (e.g., Arabic sentiment analysis, stance detection), which distracts from the technical content. I do not see a path to acceptance within the current manuscript's scope; a new, fully specified empirical study would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I agree with the reject verdict, and I want to underline the stress-test's point about Reshape: it's not just a missing detail, it's the load-bearing joint. The paper defines y'=Reshape(y) in §4.4 and never says what this is—LLM call, prompt, trained or frozen, whether it sees the reference. Because BLEU, ROUGE, and Levenshtein all measure overlap with the reference, a post-processor that rewrites outputs toward the reference format would inflate every metric in Table 1 without improving the fine-tuned model. The Levenshtein drop from ~1157 to ~1000 is exactly what you'd expect if the reshaper produces reference-like strings. So the central comparison is circular until they run a reference-blind rerun or specify the reshaper's inputs.\n\nI don't want to dismiss the underlying idea. Input reformulation to match the training distribution, and output post-processing, are both known techniques, and stacking them for user-story test case generation is a reasonable small contribution. The paper also defines the right ablation (LLM-RF and LLM-FR) to isolate the effects—but then never reports those results. That's the most damning absence: they built the experiment to answer the circularity question and didn't run it.\n\nOther soft spots in proportion: the labels for Reshaping and Refining are swapped between Figure 1 and Section 5, which makes the Table 1 columns ambiguous. The abstract mentions Falcon 7B and 1B fine-tuning, but Table 1 only shows Phi-2, and we don't know dataset size, split, or hyperparameters. There are no error bars or significance tests. Refining's Levenshtein is worse than the Phi-2 baseline, which is odd given the narrative. No code or data are provided, so none of this is reproducible.\n\nWho gets value: someone thinking about cheap LLM post-processing tricks for low-resource test generation might get a list of prompts to try, but as a research claim the paper isn't there. The internal contradictions and missing ablations are not minor copy-edits; they are the evidence the claim depends on.\n\nRecommendation: don't send to peer review yet. I'd aim this at a workshop paper or a strong revision with the ablation reported, the reshaper specified, error bars added, and the figure/text labels fixed. If they can show that reformulation alone helps under reference-blind reshaping, there might be a publishable ten-page result. The current version doesn't deserve referee time.","headline":"A plausible pipeline undermined by an unspecified output reshaper that makes Table 1 circular; reject in current form, but the ablation design hints at a fixable idea.","tokens_in":8853,"tokens_out":2693,"would_cite":false,"duration_ms":23821,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reformulating inputs and reshaping outputs lifts fine-tuned LLM scores","keywords":["large language models","data scarcity","fine-tuning","QLoRA","input reformulation","output reshaping","test case generation","user stories"],"falsifier":"Give the reshaping layer the fine-tuned model's raw outputs but withhold all reference test cases, then compare its edits to the references; if BLEU and ROUGE still jump, the gain is genuine repair, whereas if the layer's edits match reference phrasing only when it has seen the reference distribution, the improvement is an artifact of the evaluation setup.","tokens_in":7794,"feed_emoji":"🧪","tokens_out":5465,"duration_ms":44717,"temperature":0.7,"pith_summary":"This paper tries to show that wrapping a fine-tuned language model in two extra LLM layers—one that normalizes the input and one that reshapes the output—makes the model produce better results when training data is scarce. The test bed is generating software test cases from user stories, with only a small domain-specific dataset for fine-tuning. The paper reports that the full two-layer pipeline outperforms fine-tuning alone on every metric it measures, including a BLEU increase from 0.55 to 0.72 and a cosine similarity increase from 0.816 to 0.891. A sympathetic reader would care because data scarcity is common in specialized fields, and this is a practical recipe for squeezing more accuracy out of a small fine-tuning set without retraining the base model.","feed_headline":"Reformulating inputs and reshaping outputs lifts fine-tuned LLM scores","feed_subtitle":"A user-story test-case pipeline reports BLEU up from 0.55 to 0.72 and cosine similarity up to 0.89","key_machinery":"The machinery is a two-layer architecture around a QLoRA-fine-tuned base LLM (Phi-2 in the main experiments). The first layer is a reformulation function $x' = f_r(x)$ that rewrites a raw user story into the 'Action, Condition, Result' structure used during fine-tuning. The second layer is a reshaping function $y' = \\text{Reshape}(y)$ that edits the model's raw output into a clean, test-case format. QLoRA keeps the fine-tune affordable by quantizing the base weights to 4-bit and updating only a low-rank adaptation $W_q + UV^T$. The ablation compares the full stack (Reformulation–Fine-tuning–Reshaping) with each one-layer variant to attribute the gains.","core_discovery":"The central claim is that the RIRO pipeline—reformulate input, fine-tune with QLoRA, reshape output—demonstrates superior performance across all metrics compared with fine-tuning alone. On the user-story benchmark, the full Stacked LLM version raises BLEU from 0.55 to 0.72, ROUGE-1 from 0.265 to 0.402, ROUGE-2 from 0.128 to 0.149, ROUGE-L from 0.172 to 0.257, and cosine similarity from 0.816 to 0.891, while reducing Levenshtein distance from 1157.620 to 1000.880. The paper interprets these numbers as evidence that input normalization aligns the test input with the training distribution and that output reshaping fixes residual inconsistencies, so the two operations are complementary rather than redundant. It also reports that the full pipeline beats either layer applied alone in the ablation study.","pith_inferences":["A natural extension, not explored in the paper, is to measure whether the reformulation layer preserves meaning under paraphrased inputs; if it does, the same wrapper could serve as a drop-in robustness module for out-of-distribution inputs.","The architecture invites a fair-comparison experiment against deterministic template normalization: if a rule-based normalizer matches the LLM reformulation's gains, the improvement is about format alignment rather than LLM understanding.","Testing the pipeline on a second data-scarce domain, such as legal clause summarization, with the same metrics would show whether the gains generalize beyond user stories."],"forward_implications":["If the reported gains hold, practitioners with small domain datasets can improve generation quality by adding an input-normalization prompt before fine-tuning, without changing the base model.","The ablation implies that each layer contributes: skipping either reformulation or reshaping lowers at least some of the metrics, so the two operations are not interchangeable.","Because QLoRA is used for the fine-tune, the added layers do not require full retraining of the backbone, making the recipe affordable for modest compute budgets.","The same wrapper could transfer to other structured-output tasks where a small reference corpus defines the expected format, such as report generation or requirement-to-specification conversion."],"supporting_citations":[{"why":"Supplies the QLoRA fine-tuning method that makes low-cost adaptation of quantized LLMs possible.","marker":"[9]"},{"why":"Supplies the transformer and attention mechanism on which the Phi-2 architecture is based.","marker":"[36]"},{"why":"Provides prior work on LLM-based unit test improvement that this paper extends.","marker":"[1]"},{"why":"Offers an empirical evaluation of LLM unit test generation that serves as a comparison point for automated test generation.","marker":"[33]"},{"why":"Presents LLM-based unit test co-evolution and repair, a related output-refinement idea.","marker":"[13]"},{"why":"Describes data augmentation techniques whose drawbacks motivate the RIRO alternative.","marker":"[29]"},{"why":"Reviews LLM challenges and open issues, including small-data limitations, which motivate the proposed architecture.","marker":"[30]"},{"why":"Benchmarks out-of-distribution robustness in NLP, supporting the paper's focus on generalization.","marker":"[40]"}],"fun_headline_variants":["RIRO's two-step fix boosts LLM fine-tuning in low-data settings","Reformulate inputs, refine outputs: RIRO boosts LLM scores on scarce data","RIRO pipeline lifts BLEU to 0.72 and cosine similarity to 0.89","Two-layer RIRO beats plain fine-tuning for data-scarce LLMs","RIRO's input reshape and output refine lift LLM on tiny datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole improvement rests on the assumption that a user story can be rewritten into the training format without changing its requirements, and that the reshaping layer can edit the output without adding or dropping content.","fun_headline_variants_meta":{"raw":{"variants":["RIRO's two-step fix boosts LLM fine-tuning in low-data settings","Reformulate inputs, refine outputs: RIRO boosts LLM scores on scarce data","RIRO pipeline lifts BLEU to 0.72 and cosine similarity to 0.89","Two-layer RIRO beats plain fine-tuning for data-scarce LLMs","RIRO's input reshape and output refine lift LLM on tiny datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3309,"prompt_tokens":972,"completion_tokens":2337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2230}},"tokens_in":588,"tokens_out":2337,"duration_ms":16150,"temperature":1.0,"reasoning_tokens":2230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:08:56.374596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the reshaping layer the fine-tuned model's raw outputs but withhold all reference test cases, then compare its edits to the references; if BLEU and ROUGE still jump, the gain is genuine repair, whereas if the layer's edits match reference phrasing only when it has seen the reference distribution, the improvement is an artifact of the evaluation setup.","supporting_citations":[{"cited_title":"In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp","cited_arxiv_id":null,"evidence_quote":"Provides prior work on LLM-based unit test improvement that this paper extends."},{"cited_title":"IEEE Transactions on Soft- ware Engineering (2023)","cited_arxiv_id":null,"evidence_quote":"Offers an empirical evaluation of LLM unit test generation that serves as a comparison point for automated test generation."},{"cited_title":"Applied Soft Computing132, 109,803 (2023)","cited_arxiv_id":null,"evidence_quote":"Describes data augmentation techniques whose drawbacks motivate the RIRO alternative."},{"cited_title":"IEEE Access (2024)","cited_arxiv_id":null,"evidence_quote":"Reviews LLM challenges and open issues, including small-data limitations, which motivate the proposed architecture."},{"cited_title":"Advances in Neural Information Processing Systems36, 58,478– 58,507 (2023)","cited_arxiv_id":null,"evidence_quote":"Benchmarks out-of-distribution robustness in NLP, supporting the paper's focus on generalization."}],"review_version":1}