{"id":"b9eb4e41-ae95-4243-9e0a-f2cd75d34756","arxiv_id":"2509.03161","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning small LLMs on raw process data with parameter-efficient methods improves predictive process monitoring, with the clearest gains in multi-task remaining-time prediction.","lead":"This paper tests whether small language models can be fine-tuned directly on raw event-log sequences, without converting process traces into natural language, to predict the next activity and remaining time of running business cases. The approach matches or beats recurrent neural networks and narrative-style LLM baselines on five public logs, especially for remaining-time prediction in multi-task models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evaluation without a clear validation split makes the reported LLM-vs-RNN gains statistically unsupported; the central 'outperform' claim may reflect best-epoch selection.","rationale":"The paper's headline contribution is an empirical comparison; the most direct threat is that the reported differences may be noise or selection artifacts. The reader's weakest_assumption about pretrained-weight transfer is plausible but would not, by itself, invalidate the outperform claim if the architecture's inductive bias drives the results. In contrast, the absence of repeated runs and validation-based selection directly undermines the reliability of every number in Table 2. The paper's own Figure 1 shows unstable RT losses, making best-epoch selection especially risky. I therefore focus the stress-test on evaluation robustness. The proposed multi-seed rerun with a proper validation split would either confirm the claimed advantage or reveal it to be ephemeral; either outcome is decision-relevant. The transfer question remains worth investigating, but it is secondary to whether the stated scores can be reproduced.","tokens_in":9961,"tokens_out":9629,"duration_ms":102779,"concrete_test":"For each of the five logs, rerun all backbones (Llama3.2, Qwen2.5, PM-GPT2, ST-RNN, MT-RNN) with at least 5 random seeds. Use a fixed validation split (e.g., 60/20/20 within the unbiased split) to select the best epoch and hyperparameters, and evaluate on the held-out test set. Report mean±std of NA accuracy and RT MSE, and run a paired significance test (e.g., Wilcoxon signed-rank) comparing the best LLM against the best RNN per task and log. If the multi-task RT advantage is not significant (p<0.05) or the validation-selected scores drop substantially from Table 2, the central outperform claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5.2, RQ1) that fine-tuned LLMs outperform RNNs rests on 'only the best scores' in Table 2, obtained from single runs with no seeds or confidence intervals. Section 5.1 describes only an unbiased train/test split; no validation-based early stopping or checkpoint selection is specified. Given the spiky RT loss curves in Figure 1, the 'top' MSE/accuracy is the minimum over epochs and is likely to be optimistic and unstable. For example, on BPI20RfP, the RT MSE gap between PM-GPT2 (0.6366) and ST-RNN (0.6446) is ~1.2%, while on the same log the MT-RNN (0.6456) is within 1.4% of the best LLM; without variance estimates these differences are not interpretable. If the LLMs' best epochs were selected using the test set (as the absence of a validation set implies), the comparison is biased in their favor. The paper's own conclusion overstates the table by claiming gains in 'both single- and multi-task NA and RT prediction' when ST-RNN beats all LLMs on NA for BPI12 and BPI17. Thus the empirical foundation for the central claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies direct fine-tuning of small open-weights LLMs (GPT-2, Qwen2.5, Llama3.2) on raw event-log tokens for predictive process monitoring, bypassing natural-language reformulation. The methodology replaces standard text tokenization with process-specific embeddings, then applies PEFT variants (full freezing, partial freezing, LoRA) to the transformer backbone. Experiments on five BPI event logs compare next-activity (NA) accuracy and remaining-time (RT) MSE against LSTM baselines (single- and multi-task) and the narrative-style S-NAP method. The central claim is that PEFT-adapted LLMs outperform recurrent and narrative baselines, especially in multi-task RT prediction, while converging faster and requiring less hyperparameter tuning.","tokens_in":10261,"tokens_out":5877,"duration_ms":66447,"significance":"If the results hold, this is a useful systematic study of an overlooked direction: parameter-efficient adaptation of LLMs to structured process data rather than prompt engineering or narrative-style reformulation. The use of open-source models, a public code repository, and an effort to adapt S-NAP to avoid future-information leakage are concrete strengths. The multi-task RT improvements are practically relevant for PPM. However, the current empirical support is weakened by single-run comparisons, test-set-based model selection, and a mismatch between the stated single-task claims and the actual experimental design. The paper's central comparative claims need stronger statistical grounding before the conclusions can be accepted.","major_comments":[{"comment":"The evaluation reports only 'the best scores' from single runs, with no seeds, confidence intervals, or significance tests. Section 5.1 describes an unbiased train/test split but does not describe a validation set or a model-selection procedure. Since the grid spans freezing configurations and LoRA settings, the reported 'best' configuration may have been selected on the test set, which biases results in favor of the reported methods. Some key differences are small: on BPI20RfP, RT MSE is 0.6366 for PM-GPT2 versus 0.6446/0.6456 for the RNNs, a gap of about 1.2-1.4%, which is not interpretable without variance estimates. The central RQ1 claim that LLMs 'outperform' existing methods is therefore statistically unsupported. Please add multiple seeds with means and standard deviations, a fixed validation split (or nested cross-validation) for configuration selection, and paired significance t","section":"§5.1, §5.2, Table 2"},{"comment":"The paper's conclusion that LLMs outperform 'traditional PPM models and narrative-style approaches in both single- and multi-task NA and RT prediction' is not supported by Table 2. Section 5.1 states that, due to cost, LLMs are trained only in the multi-task setup, yet Table 2 also reports single-task ST-RNN results; thus the 'single-task' comparison is not like-for-like. Moreover, in single-task NA, ST-RNN beats every LLM on BPI12 (0.8358 vs 0.8162) and on BPI17 (0.8855 vs 0.8730). The more balanced wording in Section 5.2 — ST-RNN remains strong for NA, LLMs are stronger for RT — is accurate, whereas the abstract/conclusion overstate the case. The claims should be restricted to what the experiments actually compare: multi-task LLM adaptation versus single-/multi-task RNNs on these datasets.","section":"§5.2, RQ1; §6"},{"comment":"The claim that pretrained LLM knowledge transfers to process data is not isolated experimentally. Replacing the token embedding layer with randomly initialized, process-specific embeddings removes the word-level pretrained representations; the remaining value of pretraining comes from positional and upper-layer representations, but this is an assumption rather than a tested result. No from-scratch transformer or randomly initialized same-size backbone is included as a control. Without such an ablation, the good results could be attributed to the transformer architecture and trainable I/O layers rather than to the pretrained backbone. Given the paper's stated goal of 'domain adaptation of LLMs' and its claim of an 'inherent capacity' to interpret sequences when detached from natural language, this control is needed. Please add a random-init backbone of the same architecture, or at least a","section":"§4, PEFT of LLMs; §1"},{"comment":"The claim that LLMs require 'significantly less hyperparameter optimization' is confounded with tuning effort. The RNN baselines are searched over a large grid (1-6 layers; three learning rates; four embedding sizes; three hidden sizes; three batch sizes), while LLM LoRA uses r=256 and alpha=512 from a reference and 10 epochs. The 'effort' comparison is therefore not on equal terms: the RNN budget is much larger, and the LLM freezing configurations are also selected in the same grid. Please either match tuning budgets, report performance as a function of number of trials, or qualify the claim accordingly.","section":"§5.1, Hyperparameter Search Space; §5.2, RQ1"}],"minor_comments":[{"comment":"The text says 'The cross-entropy loss and mean squared error (MSE) are employed as optimizers during training'; these are loss functions, not optimizers. Also, 'the runtime for training and validation are also reported' is inconsistent with the absence of a described validation set.","section":"§5.1"},{"comment":"The caption and the '# params (%trainable)' column are ambiguous for RNNs. Specify whether the RNN parameter counts are total parameters, and make clear what the '(RNNs: 100%)' annotation means.","section":"Table 2"},{"comment":"The figure caption says 'loss distributions' but does not state what the box plots summarize (datasets? runs? epochs?) or how many data points each box represents. Since Table 2 reports single runs, clarify whether these are distributions over event logs or configurations.","section":"Figure 2"},{"comment":"The S-NAP baseline is adapted from its original version, and the authors correctly note that the original prompt includes future information. Still, the phrase 'narrative-style-based solutions' in the abstract is broader than what is actually evaluated, which is one adapted S-NAP configuration. Consider naming the baseline explicitly throughout.","section":"§3, §5.2"},{"comment":"The sentence 'Due to the LLM cost, we only train them in the multi-task setup' is important and should appear earlier, because it directly affects how readers interpret the single-task comparisons in Table 2.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is on a relevant and timely topic, and the authors have made their code public, which is a strength. However, the main comparative claims are built on single-run, validation-free evaluations and on comparisons that are not always like-for-like. The requested controls (seeds, validation split, random-init backbone, single-task LLM runs) require additional experiments but are feasible within the scope of the manuscript. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee, but take the headline numbers with a grain of salt until the evaluation is tightened. The direct adaptation of LLMs to raw process tokens—no narrative prompts, no natural-language reformulation—is a genuinely new and sensible direction for predictive process monitoring. Replacing the embedding layer with process-specific, randomly initialized embeddings and then applying freezing or LoRA is a plausible design, and the multi-task remaining-time gains over the reported RNN baselines are consistent across all five logs. Credit is due for fixing the unrealistic leak in the S-NAP baseline (it originally used the future process variant's activity list), for using public real-world logs, and for shipping code. The paper also makes an honest point about LLMs being next-token classifiers that struggle with regression unless explicitly adapted via LoRA.\n\nThat said, the empirical foundation for the central claim is not there yet. Table 2 reports only the best scores per model—no error bars, no repeated runs, no seeds. The experimental section describes an unbiased train/test split but no validation set; given the grid search over freezing layers and LoRA settings, the best configurations may have been selected on the test set. The loss curves in Figure 1 are spiky, especially for RT, which means picking the minimum over epochs can be optimistic and unstable. On BPI20RfP the gap between the best LLM and ST-RNN is about 1% in RT MSE—meaningless without variance estimates. The paper's own conclusion overstates the table: it claims gains in 'both single- and multi-task NA and RT prediction,' but ST-RNN beats all LLMs on NA for BPI12 and BPI17. So the 'outperform' claim holds mainly for RT and for the weaker MT-RNN baseline; it does not hold uniformly as stated. These issues are addressable with proper validation, multiple seeds, and significance tests, and the central multi-task RT trend may well survive. But as it stands, the comparison is not rigorous enough to accept the results at face value.\n\nThis paper is aimed at process mining practitioners and anyone interested in adapting pretrained LLMs to structured, non-linguistic sequences. It deserves peer review—the core idea is relevant and the work is honest—but I would expect a major revision to add repeated runs, a clear validation protocol, and a more calibrated conclusion. I would not cite the current version as evidence for the performance claims, but I would keep an eye on the repository and a possible revised version.","headline":"Worth a serious look and a major revision: the direct PEFT-adaptation idea is genuinely new for PPM, but the headline 'outperforms RNNs' claim is not yet statistically supported by single-run, best-epoch reporting.","tokens_in":10743,"tokens_out":1824,"would_cite":false,"duration_ms":22989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Directly fine-tuning small LLMs on raw event-log sequences, with process-specific embeddings and lightweight adapters, beats recurrent baselines and narrative-style LLM pipelines on next-activity and remaining-time prediction.","keywords":["Predictive process monitoring","Large language models","Parameter-efficient fine-tuning","LoRA","Event logs","Next-activity prediction","Remaining-time prediction","Domain adaptation"],"falsifier":"Train the same model architecture from scratch (random initialization) with the same PEFT and data pipeline. If the pretrained and randomly initialized models achieve statistically indistinguishable next-activity accuracy and remaining-time MSE across the five event logs, the improvement attributed to domain adaptation of pretrained LLMs is an artifact of the architecture or the input/output layers rather than of transfer from language pretraining.","tokens_in":9875,"feed_emoji":"🤖","tokens_out":8017,"duration_ms":74370,"temperature":0.7,"pith_summary":"This paper argues that large language models can be adapted to predict business-process behavior directly from raw event logs—sequences of activity labels and timestamps—without translating the logs into natural-language prompts or narratives. The authors fine-tune small open LLMs with parameter-efficient methods (LoRA adapters or partial layer freezing) after replacing the language model's token embedding layer with a process-specific embedding layer trained from scratch. On five real-world event logs, these adapted LLMs match or beat state-of-the-art LSTM baselines and a narrative-style LLM pipeline (S-NAP) on next-activity and remaining-time prediction, with the clearest gains in multi-task remaining-time prediction. The adapted models converge within a few epochs and need noticeably less hyperparameter tuning than RNNs. The central claim is that the transfer of a pretrained backbone's hidden representations, not its semantic understanding of text, is what carries the improvement.","feed_headline":"Fine-tuned LLMs beat RNNs on raw process logs","feed_subtitle":"Process-specific embeddings outperform narrative prompts and LSTMs, with less tuning","key_machinery":"The central mechanism is the replacement of the LLM's language-based tokenizer and embedding layer with a task-specific input layer that embeds activity-label indices and normalized timestamp features into the model's latent dimension, followed by PEFT of the backbone (either freezing selected layers or inserting LoRA low-rank adapters) while training the new input and output layers from scratch. This lets the pretrained transformer operate on the process's native token alphabet, and it is what distinguishes the approach from prompt-based and narrative-style methods that rely on natural-language text.","core_discovery":"The paper's central claim is that a pretrained LLM backbone, when its token embeddings are replaced by randomly initialised embeddings for activity labels and time features and then adapted with PEFT, can learn predictive process monitoring tasks from raw event-log sequences and outperform both dedicated RNN models and LLMs that receive logs as narrative text. The authors report that these adapted LLMs achieve higher next-activity accuracy and lower remaining-time error than the best LSTM baselines in most settings, and that the advantage is most pronounced in multi-task remaining-time prediction. They further claim that the adapted models converge in under five epochs, require no task-speci","pith_inferences":["A direct testable extension: fine-tune the same PEFT pipeline on a randomly initialized transformer of identical architecture; if the pretrained model still wins, the value comes from pretrained hidden representations rather than from architecture alone.","The method's reliance on trace encoding with teacher forcing suggests it could be combined with prefix-based online prediction, where the model sees partial traces, but the paper does not evaluate that setup.","Because the adapted models converge so quickly, a natural next step is to use them as general-purpose process encoders and probe their learned representations for other PM tasks such as anomaly detection or outcome prediction.","The reported sensitivity of the narrative-style baseline to label language implies that prompt-based approaches may be brittle in multilingual logs; the proposed direct adaptation avoids translation entirely, which could be exploited for cross-lingual process mining."],"forward_implications":["If the central claim holds, predictive process monitoring can move away from prompt engineering: practitioners would fine-tune on raw event logs and get better accuracy with less human effort.","Multi-task monitoring (next activity plus remaining time in one model) becomes practical with LLM backbones, since the adapted models handle both tasks better than multi-task RNNs.","The finding that LoRA adapters are needed for the regression task but layer freezing suffices for classification suggests a simple rule for choosing a PEFT method based on the prediction target.","Small open-weights LLMs (0.1 to 1 billion parameters) suffice to outperform much larger narrative-style pipelines, reducing runtime and deployment cost.","Because the approach works without natural language, it applies to event logs with arbitrary or multilingual label alphabets, sidestepping label-translation issues."],"supporting_citations":[{"why":"Introduces the LoRA low-rank adapter method used as the adapter-based PEFT configuration.","marker":"[4]"},{"why":"Provides the input-layer, backbone, output-layer evaluation framework that the paper's experimental setup is inspired by.","marker":"[5]"},{"why":"Shows that retraining embedding layers is needed when transferring transformers between event logs, motivating the replacement of the token embedding layer.","marker":"[9]"},{"why":"Benchmark of deep learning for predictive business process monitoring; supplies the LSTM baseline context.","marker":"[14]"},{"why":"The narrative-style S-NAP baseline that the paper adapts and compares against; its prompt leak is the methodological flaw addressed.","marker":"[15]"},{"why":"Introduces trace encoding with teacher forcing, the sequence-to-sequence input setup used to train the models.","marker":"[16]"},{"why":"Defines the multi-task next-activity and remaining-time prediction setup for LSTM networks.","marker":"[18]"},{"why":"Provides the unbiased data split used to create train and test sets, load-bearing for the reported results.","marker":"[19]"}],"fun_headline_variants":["Adapter-tuned LLMs beat RNNs on raw process logs","Raw event logs outperform narrative prompts for LLMs","Fine-tuned LLMs outpace LSTMs in process prediction","Parameter-efficient tuning boosts LLM process mining","Adapted LLMs converge faster, beat RNN baselines"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that hidden representations learned on natural-language text transfer to structured event-log sequences even after the input embeddings are replaced with randomly initialized, process-specific ones; if that transfer fails, the reported gains over RNNs would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Adapter-tuned LLMs beat RNNs on raw process logs","Raw event logs outperform narrative prompts for LLMs","Fine-tuned LLMs outpace LSTMs in process prediction","Parameter-efficient tuning boosts LLM process mining","Adapted LLMs converge faster, beat RNN baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1223,"prompt_tokens":681,"completion_tokens":542,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":425,"tokens_out":542,"duration_ms":5552,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:05:35.617513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model architecture from scratch (random initialization) with the same PEFT and data pipeline. If the pretrained and randomly initialized models achieve statistically indistinguishable next-activity accuracy and remaining-time MSE across the five event logs, the improvement attributed to domain adaptation of pretrained LLMs is an artifact of the architecture or the input/output layers rather than of transfer from language pretraining.","supporting_citations":[{"cited_title":"In: SAC (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the input-layer, backbone, output-layer evaluation framework that the paper's experimental setup is inspired by."},{"cited_title":"In: BPM Workshops (2023)","cited_arxiv_id":null,"evidence_quote":"Shows that retraining embedding layers is needed when transferring transformers between event logs, motivating the replacement of the token embedding layer."},{"cited_title":"IEEE TSC (2023)","cited_arxiv_id":null,"evidence_quote":"Benchmark of deep learning for predictive business process monitoring; supplies the LSTM baseline context."},{"cited_title":"In: ICPM (2024)","cited_arxiv_id":null,"evidence_quote":"The narrative-style S-NAP baseline that the paper adapts and compares against; its prompt leak is the methodological flaw addressed."},{"cited_title":"In: BPM (2024)","cited_arxiv_id":null,"evidence_quote":"Introduces trace encoding with teacher forcing, the sequence-to-sequence input setup used to train the models."},{"cited_title":"In: CAiSE (2017)","cited_arxiv_id":null,"evidence_quote":"Defines the multi-task next-activity and remaining-time prediction setup for LSTM networks."},{"cited_title":"In: BPM (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the unbiased data split used to create train and test sets, load-bearing for the reported results."}],"review_version":1}