{"id":"4ba5d886-ab7a-4c0a-852d-c65e60e477d1","arxiv_id":"2502.10211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Per-activity costs from alignment-based conformance checking, combined with dimensionality reduction, detect control-flow anomalies in event logs better than fitness-threshold baselines.","lead":"This paper combines process mining with machine learning to detect unusual activity sequences in business processes. The new feature extraction method based on alignment conformance checking improves detection accuracy while keeping explanations tied to the process model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison confounds feature extraction with threshold rule: baselines use minimum validation fitness, framework uses mean reconstruction error; reported F1 gaps may be an artifact.","rationale":"The reader identified the validation-derived threshold as the weakest assumption, which is related but not identical to my concern. The reader worries that the mean reconstruction error may not separate normal from anomalous logs; I agree that threshold calibration is unvalidated, but the more acute issue is the asymmetric threshold rule between baselines (minimum fitness) and framework (mean reconstruction error). This asymmetry alone can explain much of the reported performance gap, independent of the ABCC features. The central claim is therefore conditionally supported: it holds only if the framework also outperforms baselines under a comparable threshold rule. My concrete test would settle this by applying the same deviation-based threshold logic to both sides. Since the existing verdict is CONDITIONAL and my concern is a refinement of the same condition, the verdict remains unchanged. I do not see internal inconsistency or fundamental invalidity in the proposed feature extraction itself; the issue is experimental design and the strength of the comparative claim.","tokens_in":25447,"tokens_out":5039,"duration_ms":56575,"concrete_test":"Recompute Table 5 with the ABCC_B baseline using a threshold that mirrors the framework's mean-squared-error rule: flag a test log as anomalous when |fitness - mean_validation_fitness| exceeds the root-mean-square deviation of validation fitness (i.e., the reconstruction-error threshold for a one-dimensional PCA on fitness). Conversely, recompute the (ABCC, *) framework techniques with a conservative one-sided threshold equal to the maximum validation reconstruction error. If the F1 gap on PDC 2020/2021 narrows to within 10 points, the headline claim is not robust to threshold-rule choice and needs to be re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that framework techniques implementing ABCC outperform baseline conformance checking-based techniques (Abstract; Section 5.3.1, Table 5) rests on a comparison that is not like-for-like. Baselines (Section 5.2.1) set the anomaly threshold at the minimum fitness over the validation set, an extreme order statistic that is maximally conservative and flags only logs with fitness below the worst normal validation log. The framework (Section 4.4) instead sets the threshold at the mean reconstruction error over validation, a central tendency that naturally flags a much larger fraction of logs as anomalous. Figure 8 shows substantial overlap between normal and anomalous fitness distributions, so the minimum-fitness threshold lies in the far left tail, guaranteeing low baseline recall regardless of feature quality. The reported gaps (e.g., 97.3% vs 36.1% F1 on PDC 2020) therefore conflate the choice of threshold statistic with the contribution of the proposed ABCC features. To support the claim that ABCC feature extraction itself is superior, the threshold rule must be held fixed or made comparable across methods. Without this control, the 'outperform baseline' result may be an artifact of threshold calibration rather than feature extraction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for control-flow anomaly detection in event logs that combines process mining-based feature extraction with dimensionality reduction. The central novelty is an alignment-based conformance checking (ABCC) feature extraction that records per-activity alignment costs from alignments between traces and a reference Petri net. The framework is evaluated on four datasets (PDC 2020, PDC 2021, ERTMS, COVAS) with four feature extractors and four dimensionality reduction techniques. The main empirical claim is that the four ABCC-based framework techniques outperform two fitness-thresholding baselines (Table 5) while providing SHAP-based explanations linked to model elements. A secondary claim is that no single feature extractor works best across all datasets (Table 7).","tokens_in":25647,"tokens_out":6012,"duration_ms":57725,"significance":"The framework is well-motivated and the paper is clearly written. If the baseline comparison were made like-for-like, the framework would be a valuable contribution to explainable process mining anomaly detection. Strengths include the public release of code and batch scripts for replication, the use of four datasets including a real-world healthcare case study, the SHAP-based explanation tying per-activity costs to Petri net transitions, and a systematic comparison of framework techniques under a common threshold rule. The analysis of variance (Table 7) appropriately supports the 'no one-size-fits-all' conclusion. However, the headline claim of outperforming baseline conformance checking is currently undermined by a calibration confound.","major_comments":[{"comment":"The baseline threshold is set to the minimum fitness over the validation set, while the framework threshold is the mean reconstruction error (Section 4.4, Definition 4.3). These are fundamentally different statistics: the minimum is an extreme order statistic that lies in the far left tail of the validation fitness distribution, whereas the mean is a central tendency. Given the substantial overlap of normal and anomalous fitness distributions shown in Figure 8, the baseline threshold will classify only logs with fitness below the least-fitting normal validation log as anomalous, guaranteeing low recall regardless of the quality of the underlying conformance checking features. The large F1 gaps in Table 5 (e.g., 97.3% vs 36.1% on PDC 2020) therefore conflate the choice of threshold statistic and the absence of baseline hyperparameter tuning with the contribution of the proposed ABCC feature extraction. To support the claim that ABCC-based framework techniques outperform baseline conformance checking, the authors should hold the threshold calibration mechanism fixed across methods (e.g., use the same quantile of the validation score distribution for both, or report a sensitivity analysis over threshold choices) and give the baselines a comparable tuning budget.","section":"Section 5.2.1 and Table 5"},{"comment":"The validation set is used twice: first for exhaustive search of dimensionality reduction hyperparameters that minimize reconstruction error (Section 4.3, Table 4), and then as the basis for the reconstruction error threshold Eth (Section 4.4). This double use means the threshold is calibrated on the same data used to select the model, so the reported detection performance is not an unbiased estimate for a new test set. Moreover, Definition 4.3 sets Eth from a central tendency (mean error) of validation data, but there is no evidence that normal validation reconstruction errors are concentrated below this mean; if they are not, a large fraction of normal test logs will be misclassified. The authors should either use a separate calibration split or report results across a range of threshold percentiles to demonstrate robustness.","section":"Sections 4.3 and 4.4"},{"comment":"The anomaly score is defined as Et = ||t - t_hat|| (an L2 norm), while the threshold example is Eth = sum_i ||v_i - v_hat_i||^2 / nV (a mean squared error). These quantities are on different scales, so the comparison Et < Eth is not dimensionally consistent. The authors should clarify whether the implementation uses the L2 norm for both the threshold and the score, or squared errors for both; this affects the reproducibility of the reported results.","section":"Section 4.4, Definition 4.3"}],"minor_comments":[{"comment":"The comparison in Table 5 includes only ABCC-based framework techniques; the corresponding comparison for TBCC, NG, and DF is not shown in the same table. Since Table 6 shows, for example, (TBCC, AE) also achieves 95.0% F1 on PDC 2020, the claim that ABCC feature extraction specifically drives the improvement over baselines is not established; the authors should state that the improvement is a property of the framework and compare ABCC against the other extractors under identical conditions with appropriate statistical tests.","section":"Section 5.3.1 and Table 6"},{"comment":"The histograms have different bin widths and are not normalized, making visual comparison of overlap difficult; quantitative overlap measures (e.g., Bhattacharyya distance or AUC of the fitness distributions) would strengthen the explanation.","section":"Figure 8"},{"comment":"The Friedman test p-values are reported but the effect size measure I is not defined; the authors should define how the percentage of explained variance is computed.","section":"Section 5.3.2 and Table 7"},{"comment":"The phrase 'worst-case alignment' is introduced before the cost function delta is described; define the cost function explicitly before using it.","section":"Section 3.2, Definition 3.5"},{"comment":"There are several typographical issues, including 'absoulte' in the Figure 7 caption, 'di fferent' in multiple places, and 'outperformed by by' in Section 5.3.2; these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The baseline comparison concern is substantial and will likely require additional experiments or a reanalysis with comparable threshold calibration. The paper's code release should make such experiments feasible. The methodological contribution is sound, and the paper is otherwise well-structured; I see no reason to reject outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The per-activity alignment-cost feature is a real, clearly specified idea, and the code is public, so the core is reproducible. But the headline comparison against fitness-threshold baselines is not like-for-like: baselines use the minimum validation fitness as the threshold, while the framework uses the mean validation reconstruction error, so the large F1 gaps (97.3% vs 36.1% on PDC 2020) conflate the choice of detector threshold with the quality of the features.\n\nWhat is genuinely new: Definitions 3.5–3.7 are clean, and extracting per-activity costs from alignment-based conformance checking is a structured step beyond the token-based feature extraction in the cited prior work. The framework is flexible and the public code means the experiments can be rerun. Table 6 and the ANOVA give an honest picture: ABCC is strong everywhere but not uniformly best, TBCC wins on PDC 2021 and ERTMS, and the no-one-size-fits-all conclusion is consistent with the interaction effect. The SHAP explanation linking feature importances to transitions of the Petri net is a useful addition.\n\nThe soft spots are real but addressable. First, the baseline comparison is confounded. The minimum fitness over normal validation logs is an extreme order statistic; it sits in the far left tail whenever normal and anomalous fitness overlap, as Figure 8 shows, and it will suppress baseline recall no matter what features are used. The fact that TBCC framework techniques using the same reconstruction-error classifier also beat the baselines by large margins confirms that much of the gap is the detector and decision rule, not ABCC features. To support the claim that ABCC feature extraction itself is superior, the threshold rule should be held fixed across feature-extraction methods, or the baselines should receive a comparable calibration budget. Second, some headline numbers are the best of four dimensionality reduction choices per dataset; that selection is not fatal, but the strong conclusion should not rest on the best cell of Table 6. Third, the reconstruction-error threshold is set as the validation mean without independent calibration; it is a reasonable weakly-supervised choice, but the paper should justify it as a principled rule or test its sensitivity to other quantiles.\n\nNone of this is a load-bearing flaw in the central idea. The math is correct, the data provenance from prior work is acceptable, and the framework comparison is genuinely informative. This paper deserves a serious referee, but I would request a controlled comparison and some threshold-sensitivity analysis before accepting the empirical claims.\n\nWho should read it: process mining researchers working on conformance checking and anomaly detection. It gives them a reproducible feature-construction recipe and a fair warning that fitness-threshold detectors are brittle. I would bring it to a reading group and cite it for the ABCC feature extraction, not for the headline baseline comparison.","headline":"The per-activity alignment-cost feature is a real contribution, but the headline F1 gap over fitness-threshold baselines mostly reflects an unfair threshold comparison, not feature superiority.","tokens_in":26206,"tokens_out":2551,"would_cite":true,"duration_ms":33404,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Alignment-based conformance checking extracts per-activity mismatch statistics that, after dimensionality reduction, detect control-flow anomalies far more accurately than fitness-threshold baselines.","keywords":["process mining","conformance checking","alignment-based conformance checking","control-flow anomaly detection","feature extraction","dimensionality reduction","reconstruction error","event logs"],"falsifier":"Re-run the ABCC framework techniques on PDC 2020 and COVAS with the anomaly threshold set to a different rule, such as the 95th percentile of validation reconstruction errors, or a threshold fixed on a separate calibration split that was not used for hyperparameter selection. If F1 drops toward the baseline values, the mean-threshold rule is doing the detection work rather than the alignment-based features.","tokens_in":25191,"feed_emoji":"🛡️","tokens_out":6501,"duration_ms":70281,"temperature":0.7,"pith_summary":"The paper claims that the reason fitness-threshold conformance checking fails is that normal and anomalous event logs often have overlapping fitness distributions, so a single threshold cannot separate them. Its proposal is to use alignment-based conformance checking not as a scorer but as a feature extractor: from each event log, compute alignment-based fitness and per-activity costs, which count how often each activity caused a mismatch against a reference Petri net. These tabular features are fed to dimensionality reduction techniques (PCA, SPCA, KPCA, autoencoder), and an event log is anomalous if its reconstruction error exceeds a threshold derived from normal validation logs. On four datasets the resulting techniques reach F1 scores of 97.3%, 87.2%, 85.2%, and 88.5%, against best baseline F1 scores of 36.1%, 56.6%, 78.8%, and 79.1%, while SHAP values still trace each anomaly back to specific model transitions. A sympathetic reader would take the core claim to be that explainable conformance checking can be competitive with black-box anomaly detection once its diagnostics are used as features rather than collapsed into a fitness number.","feed_headline":"Alignment features beat fitness thresholds in anomaly detection","feed_subtitle":"Alignment-based mismatch features plus reconstruction error lift F1 from 36% to 97% on the PDC 2020 benchmark.","key_machinery":"The load-bearing object is the alignment-based conformance checking diagnosis: a matrix with one row per event log and one column per activity, holding the per-activity cost under the optimal alignment, together with the alignment-based fitness column. The per-activity cost for a trace is the sum of alignment move costs involving that activity, so it localizes deviations to model transitions such as t21 or t75. This matrix is then the input to a dimensionality reduction technique whose encoder/decoder pair defines reconstruction error; the classification rule is simply whether the reconstruction error of a test event log exceeds a threshold computed from validation event logs, and SHAP values on the reconstruction error connect the decision back to specific activities and model elements.","core_discovery":"The central discovery is that the information produced by alignment-based conformance checking is enough to separate normal from anomalous event logs, provided it is represented as per-activity cost features and processed by a reconstruction-based detector. The paper defines alignment-based conformance checking diagnoses as tabular data whose columns are the alignment-based fitness plus, for each activity appearing in the event logs or the model, the total cost contributed by that activity in optimal alignments. When these features are projected and reconstructed by PCA, SPCA, KPCA, or an autoencoder, normal event logs are reconstructed with small error while anomalous event logs produce large reconstruction error. The reported F1 values on PDC 2020, PDC 2021, ERTMS, and COVAS are 97.3%, 87.2%, 85.2%, and 88.5%, versus 36.1%, 56.6%, 78.8%, and 79.1% for the best fitness-threshold baseline, with the single exception of recall on COVAS for one baseline. The paper also shows that the overlap between normal and anomalous fitness distributions explains why threshold-based baselines fail, and that no single feature-extraction approach dominates across all datasets.","pith_inferences":["A direct extension the paper does not run: replace the mean validation reconstruction error with a threshold quantile chosen on a separate calibration split; if the reported F1 numbers persist, the alignment features themselves are doing the work, rather than the threshold rule.","Per-activity costs are natural input for root-cause localization in live monitoring: the same features that drive classification could flag which transitions to inspect when an event log is declared anomalous.","Because no feature extraction dominates all datasets, a portfolio or ensemble across ABCC, TBCC, NG, and DF with per-dataset selection would likely smooth the 85-97% F1 range, at the cost of some explainability.","The weakly-supervised setting uses only normal training data; if labeled anomalies are available, the same tabular diagnoses could be fed to supervised classifiers, which may exceed the reconstruction-based results reported here."],"forward_implications":["Conformance checking can be repurposed from a pass/fail scorer into a feature source, so low-quality reference models and noisy logs no longer make fitness thresholding a bottleneck.","Anomaly explanations are available at the level of individual model transitions: SHAP values on per-activity costs tell which activities were responsible for the anomalous reconstruction error.","The framework supports fair comparison of conformance checking-based and conformance checking-independent feature extraction, and the variance analysis shows the dataset and feature-extraction interaction matters, so practitioners should not expect one best combination.","On the PDC 2020 benchmark the gap is large, 97.3% F1 versus 36.1% for the strongest baseline, suggesting the feature-based route captures signal that fitness discards."],"supporting_citations":[{"why":"Supplies the definitions of Petri nets, fitness, alignments, and moves that the proposed feature extraction builds on.","marker":"[3]"},{"why":"Prior token-based conformance checking feature extraction that the alignment-based diagnoses extend; one is the ERTMS proof-of-concept the experiments replicate.","marker":"[45, 16]"},{"why":"Autoencoder and reconstruction-based anomaly detection works that motivate the dimensionality reduction stage of the framework.","marker":"[37, 50, 18, 15, 22]"},{"why":"Provides the anomaly detection guidelines the framework follows to define the data, anomaly types, and weakly-supervised scenario.","marker":"[14]"},{"why":"The SHAP method used to explain reconstruction errors in terms of per-activity cost features and reference model elements.","marker":"[32]"},{"why":"Conformance checking-based techniques relying on fitness thresholding that serve as the baselines the proposed framework techniques are compared against.","marker":"[11, 30, 35, 40]"},{"why":"Provides the COVAS healthcare case-study data and the discovered first-wave Petri net used as reference model.","marker":"[10]"}],"fun_headline_variants":["Alignment cost features outclass fitness thresholds for anomaly detection","Reconstruction error on alignment features beats fitness thresholds","Per-activity mismatch costs yield 97% F1, beating fitness cutoffs","Alignment features, not fitness, spot control-flow anomalies","Why fitness thresholds fail: alignment costs expose anomalies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The detection pipeline assumes that the mean reconstruction error over the validation set is a reliable boundary between normal and anomalous event logs, and the same validation set is used to pick the dimensionality-reduction hyperparameters; if normal validation errors are not concentrated below that mean, the reported accuracy depends on this threshold rule rather than on the alignment features.","fun_headline_variants_meta":{"raw":{"variants":["Alignment cost features outclass fitness thresholds for anomaly detection","Reconstruction error on alignment features beats fitness thresholds","Per-activity mismatch costs yield 97% F1, beating fitness cutoffs","Alignment features, not fitness, spot control-flow anomalies","Why fitness thresholds fail: alignment costs expose anomalies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2958,"prompt_tokens":1024,"completion_tokens":1934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":640,"tokens_out":1934,"duration_ms":15457,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T18:57:31.693848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ABCC framework techniques on PDC 2020 and COVAS with the anomaly threshold set to a different rule, such as the 95th percentile of validation reconstruction errors, or a threshold fixed on a separate calibration split that was not used for hyperparameter selection. If F1 drops toward the baseline values, the mean-threshold rule is doing the detection work rather than the alignment-based features.","supporting_citations":[{"cited_title":"Anomaly Detection: A Sur- vey","cited_arxiv_id":null,"evidence_quote":"Provides the anomaly detection guidelines the framework follows to define the data, anomaly types, and weakly-supervised scenario."},{"cited_title":"A unified approach to interpreting model predictions, in: Proceedings of the 31st International Confer- ence on Neural Information Processing Systems, p","cited_arxiv_id":null,"evidence_quote":"The SHAP method used to explain reconstruction errors in terms of per-activity cost features and reference model elements."},{"cited_title":"Process Modeling and Conformance Checking in Healthcare: A COVID-19 Case Study, in: Montali, M., Senderovich, A., Weidlich, M","cited_arxiv_id":null,"evidence_quote":"Provides the COVAS healthcare case-study data and the discovered first-wave Petri net used as reference model."}],"review_version":1}