Domain Adaptation of LLMs for Process Data

Jari Peeperkorn; Jochen De Weerdt; Johannes De Smedt; Rafael Seidi Oyamada

REVIEW 4 major objections 5 minor 19 references

Domain Adaptation of LLMs for Process Data

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Directly fine-tuning small LLMs on raw event-log sequences, with process-specific embeddings and lightweight adapters, beats recurrent baselines and narrative-style LLM pipelines on next-activity and remaining-time prediction.

desk verdict Worth a serious look and a major revision: the direct PEFT-adaptation idea is genuinely new for PPM, but the headline 'outperforms RNNs' claim is not yet statistically supported by single-run, best-epoch reporting. read the letter →

arxiv 2509.03161 v1 pith:GXO4RELX submitted 2025-09-03 cs.CL cs.AI

Rafael Seidi Oyamada , Jari Peeperkorn , Jochen De Weerdt , Johannes De Smedt This is my paper · ORCID

classification cs.CLcs.AI

keywords PredictiveprocessmonitoringLargelanguagemodelsParameter-efficientfine-tuningLoRAEventlogsNext-activitypredictionRemaining-timeDomainadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models can be adapted to predict business-process behavior directly from raw event logs—sequences of activity labels and timestamps—without translating the logs into natural-language prompts or narratives. The authors fine-tune small open LLMs with parameter-efficient methods (LoRA adapters or partial layer freezing) after replacing the language model's token embedding layer with a process-specific embedding layer trained from scratch. On five real-world event logs, these adapted LLMs match or beat state-of-the-art LSTM baselines and a narrative-style LLM pipeline (S-NAP) on next-activity and remaining-time prediction, with the clearest gains in multi-task remaining-time prediction. The adapted models converge within a few epochs and need noticeably less hyperparameter tuning than RNNs. The central claim is that the transfer of a pretrained backbone's hidden representations, not its semantic understanding of text, is what carries the improvement.

What carries the argument

The central mechanism is the replacement of the LLM's language-based tokenizer and embedding layer with a task-specific input layer that embeds activity-label indices and normalized timestamp features into the model's latent dimension, followed by PEFT of the backbone (either freezing selected layers or inserting LoRA low-rank adapters) while training the new input and output layers from scratch. This lets the pretrained transformer operate on the process's native token alphabet, and it is what distinguishes the approach from prompt-based and narrative-style methods that rely on natural-language text.

What would settle it

Train the same model architecture from scratch (random initialization) with the same PEFT and data pipeline. If the pretrained and randomly initialized models achieve statistically indistinguishable next-activity accuracy and remaining-time MSE across the five event logs, the improvement attributed to domain adaptation of pretrained LLMs is an artifact of the architecture or the input/output layers rather than of transfer from language pretraining.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pretrained LLM backbone, when its token embeddings are replaced by randomly initialised embeddings for activity labels and time features and then adapted with PEFT, can learn predictive process monitoring tasks from raw event-log sequences and outperform both dedicated RNN models and LLMs that receive logs as narrative text. The authors report that these adapted LLMs achieve higher next-activity accuracy and lower remaining-time error than the best LSTM baselines in most settings, and that the advantage is most pronounced in multi-task remaining-time prediction. They further claim that the adapted models converge in under five epochs, require no task-speci

Load-bearing premise

The load-bearing premise is that hidden representations learned on natural-language text transfer to structured event-log sequences even after the input embeddings are replaced with randomly initialized, process-specific ones; if that transfer fails, the reported gains over RNNs would not hold.

Editorial extensions

If this is right

If the central claim holds, predictive process monitoring can move away from prompt engineering: practitioners would fine-tune on raw event logs and get better accuracy with less human effort.
Multi-task monitoring (next activity plus remaining time in one model) becomes practical with LLM backbones, since the adapted models handle both tasks better than multi-task RNNs.
The finding that LoRA adapters are needed for the regression task but layer freezing suffices for classification suggests a simple rule for choosing a PEFT method based on the prediction target.
Small open-weights LLMs (0.1 to 1 billion parameters) suffice to outperform much larger narrative-style pipelines, reducing runtime and deployment cost.
Because the approach works without natural language, it applies to event logs with arbitrary or multilingual label alphabets, sidestepping label-translation issues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct testable extension: fine-tune the same PEFT pipeline on a randomly initialized transformer of identical architecture; if the pretrained model still wins, the value comes from pretrained hidden representations rather than from architecture alone.
The method's reliance on trace encoding with teacher forcing suggests it could be combined with prefix-based online prediction, where the model sees partial traces, but the paper does not evaluate that setup.
Because the adapted models converge so quickly, a natural next step is to use them as general-purpose process encoders and probe their learned representations for other PM tasks such as anomaly detection or outcome prediction.
The reported sensitivity of the narrative-style baseline to label language implies that prompt-based approaches may be brittle in multilingual logs; the proposed direct adaptation avoids translation entirely, which could be exploited for cross-lingual process mining.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a serious referee, but take the headline numbers with a grain of salt until the evaluation is tightened. The direct adaptation of LLMs to raw process tokens—no narrative prompts, no natural-language reformulation—is a genuinely new and sensible direction for predictive process monitoring. Replacing the embedding layer with process-specific, randomly initialized embeddings and then applying freezing or LoRA is a plausible design, and the multi-task remaining-time gains over the reported RNN baselines are consistent across all five logs. Credit is due for fixing the unrealistic leak in the S-NAP baseline (it originally used the future process variant's activity list), for using public real-world logs, and for shipping code. The paper also makes an honest point about LLMs being next-token classifiers that struggle with regression unless explicitly adapted via LoRA.

That said, the empirical foundation for the central claim is not there yet. Table 2 reports only the best scores per model—no error bars, no repeated runs, no seeds. The experimental section describes an unbiased train/test split but no validation set; given the grid search over freezing layers and LoRA settings, the best configurations may have been selected on the test set. The loss curves in Figure 1 are spiky, especially for RT, which means picking the minimum over epochs can be optimistic and unstable. On BPI20RfP the gap between the best LLM and ST-RNN is about 1% in RT MSE—meaningless without variance estimates. The paper's own conclusion overstates the table: it claims gains in 'both single- and multi-task NA and RT prediction,' but ST-RNN beats all LLMs on NA for BPI12 and BPI17. So the 'outperform' claim holds mainly for RT and for the weaker MT-RNN baseline; it does not hold uniformly as stated. These issues are addressable with proper validation, multiple seeds, and significance tests, and the central multi-task RT trend may well survive. But as it stands, the comparison is not rigorous enough to accept the results at face value.

This paper is aimed at process mining practitioners and anyone interested in adapting pretrained LLMs to structured, non-linguistic sequences. It deserves peer review—the core idea is relevant and the work is honest—but I would expect a major revision to add repeated runs, a clear validation protocol, and a more calibrated conclusion. I would not cite the current version as evidence for the performance claims, but I would keep an eye on the repository and a possible revised version.

Referee Report

4 major / 5 minor

Summary. The paper studies direct fine-tuning of small open-weights LLMs (GPT-2, Qwen2.5, Llama3.2) on raw event-log tokens for predictive process monitoring, bypassing natural-language reformulation. The methodology replaces standard text tokenization with process-specific embeddings, then applies PEFT variants (full freezing, partial freezing, LoRA) to the transformer backbone. Experiments on five BPI event logs compare next-activity (NA) accuracy and remaining-time (RT) MSE against LSTM baselines (single- and multi-task) and the narrative-style S-NAP method. The central claim is that PEFT-adapted LLMs outperform recurrent and narrative baselines, especially in multi-task RT prediction, while converging faster and requiring less hyperparameter tuning.

Significance. If the results hold, this is a useful systematic study of an overlooked direction: parameter-efficient adaptation of LLMs to structured process data rather than prompt engineering or narrative-style reformulation. The use of open-source models, a public code repository, and an effort to adapt S-NAP to avoid future-information leakage are concrete strengths. The multi-task RT improvements are practically relevant for PPM. However, the current empirical support is weakened by single-run comparisons, test-set-based model selection, and a mismatch between the stated single-task claims and the actual experimental design. The paper's central comparative claims need stronger statistical grounding before the conclusions can be accepted.

major comments (4)

[§5.1, §5.2, Table 2] The evaluation reports only 'the best scores' from single runs, with no seeds, confidence intervals, or significance tests. Section 5.1 describes an unbiased train/test split but does not describe a validation set or a model-selection procedure. Since the grid spans freezing configurations and LoRA settings, the reported 'best' configuration may have been selected on the test set, which biases results in favor of the reported methods. Some key differences are small: on BPI20RfP, RT MSE is 0.6366 for PM-GPT2 versus 0.6446/0.6456 for the RNNs, a gap of about 1.2-1.4%, which is not interpretable without variance estimates. The central RQ1 claim that LLMs 'outperform' existing methods is therefore statistically unsupported. Please add multiple seeds with means and standard deviations, a fixed validation split (or nested cross-validation) for configuration selection, and paired significance t
[§5.2, RQ1; §6] The paper's conclusion that LLMs outperform 'traditional PPM models and narrative-style approaches in both single- and multi-task NA and RT prediction' is not supported by Table 2. Section 5.1 states that, due to cost, LLMs are trained only in the multi-task setup, yet Table 2 also reports single-task ST-RNN results; thus the 'single-task' comparison is not like-for-like. Moreover, in single-task NA, ST-RNN beats every LLM on BPI12 (0.8358 vs 0.8162) and on BPI17 (0.8855 vs 0.8730). The more balanced wording in Section 5.2 — ST-RNN remains strong for NA, LLMs are stronger for RT — is accurate, whereas the abstract/conclusion overstate the case. The claims should be restricted to what the experiments actually compare: multi-task LLM adaptation versus single-/multi-task RNNs on these datasets.
[§4, PEFT of LLMs; §1] The claim that pretrained LLM knowledge transfers to process data is not isolated experimentally. Replacing the token embedding layer with randomly initialized, process-specific embeddings removes the word-level pretrained representations; the remaining value of pretraining comes from positional and upper-layer representations, but this is an assumption rather than a tested result. No from-scratch transformer or randomly initialized same-size backbone is included as a control. Without such an ablation, the good results could be attributed to the transformer architecture and trainable I/O layers rather than to the pretrained backbone. Given the paper's stated goal of 'domain adaptation of LLMs' and its claim of an 'inherent capacity' to interpret sequences when detached from natural language, this control is needed. Please add a random-init backbone of the same architecture, or at least a
[§5.1, Hyperparameter Search Space; §5.2, RQ1] The claim that LLMs require 'significantly less hyperparameter optimization' is confounded with tuning effort. The RNN baselines are searched over a large grid (1-6 layers; three learning rates; four embedding sizes; three hidden sizes; three batch sizes), while LLM LoRA uses r=256 and alpha=512 from a reference and 10 epochs. The 'effort' comparison is therefore not on equal terms: the RNN budget is much larger, and the LLM freezing configurations are also selected in the same grid. Please either match tuning budgets, report performance as a function of number of trials, or qualify the claim accordingly.

minor comments (5)

[§5.1] The text says 'The cross-entropy loss and mean squared error (MSE) are employed as optimizers during training'; these are loss functions, not optimizers. Also, 'the runtime for training and validation are also reported' is inconsistent with the absence of a described validation set.
[Table 2] The caption and the '# params (%trainable)' column are ambiguous for RNNs. Specify whether the RNN parameter counts are total parameters, and make clear what the '(RNNs: 100%)' annotation means.
[Figure 2] The figure caption says 'loss distributions' but does not state what the box plots summarize (datasets? runs? epochs?) or how many data points each box represents. Since Table 2 reports single runs, clarify whether these are distributions over event logs or configurations.
[§3, §5.2] The S-NAP baseline is adapted from its original version, and the authors correctly note that the original prompt includes future information. Still, the phrase 'narrative-style-based solutions' in the abstract is broader than what is actually evaluated, which is one adapted S-NAP configuration. Consider naming the baseline explicitly throughout.
[§5.1] The sentence 'Due to the LLM cost, we only train them in the multi-task setup' is important and should appear earlier, because it directly affects how readers interpret the single-task comparisons in Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark comparison with no derivation chain that reduces to its own inputs; self-citations are non-load-bearing method references.

full rationale

The paper makes no mathematical derivation or first-principles prediction; its central claim is an empirical comparison of fine-tuned LLMs against RNN and narrative-style baselines on held-out event log splits. There is no equation-level equivalence between an input assumption and a reported result, and no fitted parameter is renamed as a prediction. The self-citations present are to the unbiased split methodology [19] and to time-related feature engineering [11], both of which are auxiliary, externally described methods and are not load-bearing for the main claim. The use of 'only the best scores' from a hyperparameter search is a test-set selection concern, not a definitional circularity, and the paper does not claim to derive these scores from its inputs. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its results rest on a set of empirical choices: the transferability of pretrained transformers to non-textual tokens, the trace encoding format, the split protocol, and the hyperparameter grids for both LLMs and RNNs. None of these are derived; they are assumptions of the experimental design.

free parameters (4)

LoRA rank r and alpha = r=256, alpha=512
Chosen as suggested in [3], not tuned per dataset; affects the trade-off between trainable parameters and predictive performance.
Fine-tuning epochs = 10 for LLMs, 25 for RNNs
Chosen to limit LLM cost; the comparison of convergence and required tuning depends on this choice.
Freezing layer selection = Full freezing, (0), (0,1), (-1), (-1,-2)
Grid searched per model and dataset; the best configuration is selected and reported, effectively a fitted experimental choice.
RNN hyperparameters = layers 1-6, LR 5e-4/1e-4/5e-5, embedding 32/128/256/512, hidden 128/256/512, batch 32/64/256
Large grid search; the best per dataset is reported, giving RNNs a tuning advantage that makes the comparison asymmetric.

assumptions (6)

domain assumption Pretrained transformer weights transfer to a new input embedding space
Section 4 replaces the LLM embedding layers with random process-specific embeddings while keeping the backbone; the method's success depends on this transfer.
domain assumption Trace encoding with teacher forcing is a valid training format for NA and RT
Section 5.1 adopts the many-to-many encoding of [16]; the predictive quality depends on this choice.
domain assumption The unbiased split from [19] prevents data leakage
Section 5.1 relies on [19] without detailing the split; if leakage occurs, the reported scores are inflated.
domain assumption The public BPI logs are representative of process mining practice
Section 5.1 selects five BPI logs; generalizability to other logs is assumed.
domain assumption The narrative-style baseline S-NAP is faithfully reimplemented
Section 5.1 modifies S-NAP's prompt to include all training activities and later standardizes labels; the baseline comparison depends on these adjustments.
domain assumption Event and case attributes can be ignored
Section 2.1 limits features to activity labels and timestamps; richer attributes could change the ranking.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Domain Adaptation of LLMs for Process Data." pith.science (2026). https://pith.science/paper/GXO4RELX

@misc{pith2026250903161,
  author       = {Pith},
  title        = {Pith review of: Domain Adaptation of LLMs for Process Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXO4RELX}},
  note         = {Machine review of arXiv:2509.03161}
}

read the original abstract

In recent years, Large Language Models (LLMs) have emerged as a prominent area of interest across various research domains, including Process Mining (PM). Current applications in PM have predominantly centered on prompt engineering strategies or the transformation of event logs into narrative-style datasets, thereby exploiting the semantic capabilities of LLMs to address diverse tasks. In contrast, this study investigates the direct adaptation of pretrained LLMs to process data without natural language reformulation, motivated by the fact that these models excel in generating sequences of tokens, similar to the objective in PM. More specifically, we focus on parameter-efficient fine-tuning techniques to mitigate the computational overhead typically associated with such models. Our experimental setup focuses on Predictive Process Monitoring (PPM), and considers both single- and multi-task predictions. The results demonstrate a potential improvement in predictive performance over state-of-the-art recurrent neural network (RNN) approaches and recent narrative-style-based solutions, particularly in the multi-task setting. Additionally, our fine-tuned models exhibit faster convergence and require significantly less hyperparameter optimization.

Figures

Figures reproduced from arXiv: 2509.03161 by the authors.

**Figure 2.** (a) Accuracy and MSE of each PEFT for each language model on all [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

In: ICPM Workshops (2025)

Berti, A., Kourani, H., van der Aalst, W.M.P.: Pm-llm-benchmark: Evaluating large language models on process mining tasks. In: ICPM Workshops (2025)

work page 2025
[2]

In: BPMDS and EMMSAD (2024)

Berti, A., Kourani, H., Häfke, H., Li, C., Schuster, D.: Evaluating large language models in process mining: Capabilities, benchmarks, and evaluation strategies. In: BPMDS and EMMSAD (2024)

work page 2024
[3]

CoRR (2024)

Biderman, D., Ortiz, J.J.G., Portes, J.P., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., Cunningham, J.P.: Lora learns less and forgets less. CoRR (2024)

work page 2024
[4]

In: ICLR (2022)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. In: ICLR (2022)

2022
[5]

In: SAC (2022)

Ketykó, I., Mannhardt, F., Hassani, M., van Dongen, B.F.: What averages do not tell: predicting real life processes with sequential deep learning. In: SAC (2022)

work page 2022
[6]

In: BPM (2024)

Kubrak, K., Botchorishvili, L., Milani, F., Nolte, A., Dumas, M.: Explanatory capabilities of large language models in prescriptive process monitoring. In: BPM (2024)

work page 2024
[7]

In: BPM (2024)

Lashkevich, K., Milani, F., Avramenko, M., Dumas, M.: Llm-assisted optimization of waiting time in business processes: A prompting method. In: BPM (2024)

work page 2024
[8]

In: ACL (2022)

Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In: ACL (2022)

work page 2022

Show all 19 references

[9]

In: BPM Workshops (2023)

van Luijken, M., Ketykó, I., Mannhardt, F.: An experiment on transfer learning for suffix prediction on event logs. In: BPM Workshops (2023)

2023
[10]

Trends in Cognitive Sciences (2024)

Mahowald, K., Ivanova, A.A., Blank, I.A., Kanwisher, N., Tenenbaum, J.B., Fe- dorenko, E.: Dissociating language and thought in large language models. Trends in Cognitive Sciences (2024)

2024
[11]

In: CAiSE

Oyamada, R.S., Tavares, G.M., Junior, S.B., Ceravolo, P.: Enhancing predictive process monitoring with time-related feature engineering. In: CAiSE. LNCS (2024)

2024
[12]

In: ICPM (2024)

Pasquadibisceglie, V., Appice, A., Malerba, D.: LUPIN: A LLM approach for ac- tivity suffix prediction in business process event logs. In: ICPM (2024)

2024
[13]

In: ICLR (2024)

Petrov, A., Torr, P., Bibi, A.: When do prompting and prefix-tuning work? A theory of capabilities and limitations. In: ICLR (2024)

2024
[14]

IEEE TSC (2023)

Rama-Maneiro, E., Vidal, J.C., Lama, M.: Deep learning for predictive business process monitoring: Review and benchmark. IEEE TSC (2023)

2023
[15]

In: ICPM (2024)

Rebmann, A., Schmidt, F.D., Glavas, G., van der Aa, H.: Evaluating the ability of llms to solve semantics-aware process mining tasks. In: ICPM (2024)

2024
[16]

In: BPM (2024)

Roider, J., Zanca, D., Eskofier, B.M.: Efficient training of recurrent neural networks for remaining time prediction in predictive process monitoring. In: BPM (2024)

2024
[17]

In: ICLR (2024)

Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A.: Quantifying language models’ sensi- tivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In: ICLR (2024)

2024
[18]

In: CAiSE (2017)

Tax, N., Verenich, I., Rosa, M.L., Dumas, M.: Predictive business process moni- toring with LSTM neural networks. In: CAiSE (2017)

2017
[19]

In: BPM (2021)

Weytjens, H., Weerdt, J.D.: Creating unbiased public benchmark datasets with data leakage prevention for predictive process monitoring. In: BPM (2021)

2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

In: ICPM Workshops (2025)

Berti, A., Kourani, H., van der Aalst, W.M.P.: Pm-llm-benchmark: Evaluating large language models on process mining tasks. In: ICPM Workshops (2025)

work page 2025

[2] [2]

In: BPMDS and EMMSAD (2024)

Berti, A., Kourani, H., Häfke, H., Li, C., Schuster, D.: Evaluating large language models in process mining: Capabilities, benchmarks, and evaluation strategies. In: BPMDS and EMMSAD (2024)

work page 2024

[3] [3]

CoRR (2024)

Biderman, D., Ortiz, J.J.G., Portes, J.P., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., Cunningham, J.P.: Lora learns less and forgets less. CoRR (2024)

work page 2024

[4] [4]

In: ICLR (2022)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. In: ICLR (2022)

2022

[5] [5]

In: SAC (2022)

Ketykó, I., Mannhardt, F., Hassani, M., van Dongen, B.F.: What averages do not tell: predicting real life processes with sequential deep learning. In: SAC (2022)

work page 2022

[6] [6]

In: BPM (2024)

Kubrak, K., Botchorishvili, L., Milani, F., Nolte, A., Dumas, M.: Explanatory capabilities of large language models in prescriptive process monitoring. In: BPM (2024)

work page 2024

[7] [7]

In: BPM (2024)

Lashkevich, K., Milani, F., Avramenko, M., Dumas, M.: Llm-assisted optimization of waiting time in business processes: A prompting method. In: BPM (2024)

work page 2024

[8] [8]

In: ACL (2022)

Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In: ACL (2022)

work page 2022