{"id":"97aa6836-1113-404d-996e-9158a5d98057","arxiv_id":"2411.08814","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Aligning neural network predictions with a discovered eating process model improves activity recognition accuracy and Macro F1 on the EatSense dataset in a small two-category study.","lead":"Can a computer recognize what you are doing better if it knows the usual order of things? This paper tests adding a process model of eating behavior to a standard neural network and reports higher accuracy on two small sets of eating videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 gains may reflect generic temporal smoothing; no control baseline isolates the process-model contribution.","rationale":"The reader's weakest assumption concerns whether the process model discovered from the training split generalizes to test videos. That is a valid external-validity concern, and I partially agree. However, the more load-bearing issue is internal validity: even if the process model generalizes perfectly, the reported improvements could arise from a generic temporal-smoothing effect rather than the specific process information. The cost function in Eq. (2) with epsilon=0.1 heavily penalizes log moves and low-probability synchronous moves, so ProbCost acts as a strong prior that forces predictions onto frequent training paths. Since the process model is discovered from training labels, it is essentially a frequency-based automaton. A first-order Markov model built from the same labels would capture much of the same sequential structure and could plausibly achieve similar corrections. The paper's evaluation compares only against raw GCN outputs, not against any sequence-level baseline, so the central claim that process information enhances HAR is not isolated. Additionally, the test sets are extremely small (approximately 4 videos for subject 1 and 3 for roti), and no error bars or significance tests are reported, so a handful of corrections can flip the headline numbers. These concerns reinforce the reader's CONDITIONAL verdict: the approach is plausible but the evidence is currently insufficient to support the general claim. The proposed control experiment would directly test whether the process model adds value over a simpler sequence smoother, and would either strengthen or falsify the central claim.","tokens_in":11386,"tokens_out":6005,"duration_ms":55266,"concrete_test":"Re-run the subject-1 and eating-roti experiments replacing the discovered process model and ProbCost with a first-order Markov model: estimate the transition matrix from the training labels, use the GCN softmax outputs as emissions, decode with Viterbi on the same test videos, and tune epsilon on the validation set in the same way. Also compute a per-video majority-class baseline. If the Markov model or majority class attains accuracy/F1 equal to or above the aligned numbers in Table 3, the claimed improvement cannot be attributed to the process-aware alignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 3, where ProbCost alignment raises accuracy/F1 over raw GCN outputs. However, the alignment with the discovered process model and epsilon=0.1 acts as a strong temporal prior: in Eq. (2), a model move costs -log(0.1) ≈ 2.3, while a synchronous move with probability 0.03 costs ≈ 3.5, so the decoder systematically prefers process-model paths over low-confidence events. Because the process model is discovered from the training labels (Section 5, Process discovery), it encodes the most frequent activity and transition patterns. On highly repetitive eating videos, even a simple first-order Markov model (transition probabilities estimated from the same training labels, Viterbi decoding over the same softmax emissions) could produce comparable or better corrections. The paper provides no such control, nor a majority-class per-video baseline. Thus the observed improvement does not establish that process information, rather than generic sequence smoothing or class-prior correction, drives the gain. With only ~4 and ~3 test videos, a few corrections (like the one in Section 5) can account for the entire reported improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a process-aware human activity recognition (HAR) method. Given the softmax probability outputs of a GCN-based HAR model, the method treats these outputs as a probabilistic event trace and aligns it, via the existing ProbCost algorithm, with a process model discovered from ground-truth activity labels using the Heuristic Miner. The alignment is controlled by a confidence threshold epsilon, and the aligned path is used as the final activity labels. Experiments on the EatSense dataset, restricted to two categories (videos of subject 1 and videos of eating roti), compare the original outputs of 2s-AGCN and CTR-GCN with the post-alignment outputs, reporting higher accuracy and Macro F1-scores after alignment. The central claim is that incorporating process information improves HAR performance over the baseline GCN outputs.","tokens_in":11488,"tokens_out":4328,"duration_ms":39419,"significance":"If the claimed improvements are real, the paper offers a concrete pipeline for injecting structured process knowledge into HAR outputs, which is a plausible and useful direction. The manuscript is clearly written, the pipeline is simple and reproducible, and it is a strength that the test set is held out, so the reported gains are not fitted by construction. However, the evaluation is currently too weak to support the claim as stated: the test sets contain only about 4 and 3 videos in the two categories, there are no error bars or significance tests, and no control baseline isolates the process-model contribution from generic temporal smoothing. The paper is therefore better viewed as a preliminary demonstration than as a confirmatory experimental study.","major_comments":[{"comment":"The evaluation is based on very small test sets: subject 1 has 19 videos and eating roti has 13 videos, so with a 60/20/20 split the test sets contain roughly 4 and 3 videos, respectively. The reported improvements, such as CTR-GCN accuracy rising from 0.57 to 0.71 on subject 1, can be produced by correcting only a few individual events. No confidence intervals, error bars, per-video results, or significance tests are reported. Because the central claim is that the approach 'achieves better accuracy and Macro F1-score compared to baseline models,' the evidence in Table 3 is not strong enough without some measure of variability or a significance test.","section":"Table 3 and Section 5, Experimental design"},{"comment":"The experiment does not isolate the contribution of the process model from that of generic temporal smoothing. Since epsilon is tuned to 0.1, the cost of a model move is -log(0.1) ≈ 2.3, while a synchronous move with probability 0.03 costs about 3.5, so the alignment systematically prefers process-model paths over low-confidence events. Because the process model is discovered from the training labels, it encodes the most frequent activities and transitions. A first-order Markov model with Viterbi decoding, using transition probabilities estimated from the same training labels and the same softmax emissions, would serve as a natural control baseline. Without such a control, the observed gains in Table 3 cannot be attributed specifically to process information rather than to generic sequence smoothing or class-prior correction.","section":"Section 5, Results and discussion; Eq. (2)"},{"comment":"The experimental design states that the validation set is used to tune the parameters of process discovery, but the dependency threshold for the Heuristic Miner is in fact selected by maximizing the F-score of fitness and precision computed on the training log (Table 2). This means the process model is selected on the same data used to fit it, which can lead to overfitting to the training sequences and does not validate the model's generality on held-out data. The authors should either select the dependency threshold on the validation set or clarify why the training-log F-score is an appropriate criterion.","section":"Section 5, Process discovery"},{"comment":"The paper illustrates the method with one corrected example (chewing corrected to food in hand at table). While the example is useful for intuition, it is anecdotal and does not indicate how often the alignment changes predictions, whether such changes are systematically beneficial, or how many test events were affected. Reporting the number of alignments that changed labels and the confusion-matrix-level breakdown would give a more complete picture of where the improvement comes from.","section":"Section 5, Results and discussion"}],"minor_comments":[{"comment":"There are small language and typographical issues, e.g., 'These HAR methods relies solely' should be 'rely solely'.","section":"Abstract and Introduction"},{"comment":"The phrase 'classify the videos into distinct categories through two strategic approaches' is slightly confusing; it would be clearer to say the authors group videos by subject and by food type.","section":"Section 5, Experimental design"},{"comment":"In the description of Eq. (2), 'ALM' should be 'A LM' (a log move) for readability.","section":"Section 3, Eq. (2)"},{"comment":"The text repeatedly uses 'month' where 'mouth' is meant (e.g., Section 2 and Figure 3 descriptions); these should be corrected.","section":"Throughout"},{"comment":"The y-axis label 'Accuracy' would benefit from specifying that it is the validation-set accuracy, to match the text.","section":"Figure 4"},{"comment":"The reference 'Contributors, M.' should be formatted as the OpenMMLab project contributors; consider using the standard citation for OpenMMLab (mmaction2).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting application of probabilistic conformance checking to HAR, and the direction is worth pursuing. However, the current evidence is too thin for the strong claim in the abstract and conclusion. The central gap is the absence of a control baseline that would distinguish process-model information from generic temporal smoothing; the small test sizes and lack of error bars compound this problem. I would recommend asking the authors to add a Markov/Viterbi baseline and per-video or bootstrapped results, or to substantially narrow the claim if such experiments are not possible. Novelty relative to the existing ProbCost paper is moderate, but the application to HAR is new enough for a journal paper if the evaluation is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a plausible pipeline — take softmax outputs from a HAR model, discover a process model from training labels, align with the authors' ProbCost algorithm, and report accuracy/F1 before and after. The specific application of ProbCost to HAR is new, and the worked example in Section 5 is coherent. But the evaluation is too weak to support the abstract's general claim, and the reported gains probably mix in a large dose of generic temporal smoothing.\n\nWhat is good: the authors acknowledge that eating behaviour is flexible and choose heuristic miner for that reason; they tune epsilon on a validation split rather than on the test set; and the discovered process models for subject 1 and roti look reasonable. The idea of using sequential context to correct isolated frame-level errors is sensible.\n\nWhere it gets soft: only two categories are evaluated, both chosen because they have the most videos — 19 for subject 1 and 13 for roti. After the 60/20/20 split, the test sets are roughly 4 and 2–3 videos. There are no error bars, no significance tests, and no code. More importantly, there is no control baseline. With epsilon=0.1, the ProbCost cost function makes model moves cost -log(0.1) ≈ 2.3 and synchronous matches to low-probability events more expensive, so the decoder strongly prefers process-model paths. A simple first-order Markov model with Viterbi decoding over the same softmax emissions, trained on the same labels, would likely do comparable or better. Without that control, Table 3's improvements cannot be attributed to 'process information' as opposed to sequence smoothing or class-prior correction. On a handful of test videos, a few corrections can account for the entire gain.\n\nI'd also note the 'significant improvement' phrasing in the conclusion is not backed by any statistical test.\n\nBottom line: a potentially useful post-processing hook for HAR in structured settings, but as presented the evidence is suggestive, not conclusive. A serious referee should ask for a larger evaluation, a Markov or majority-class baseline, and per-video results before accepting the general claim.\n\nRecommendation: send it to peer review — it deserves referee time — but the controls need to be added before publication.","headline":"Plausible pipeline, weak evaluation: the reported gains likely include a large dose of generic temporal smoothing, and the tiny test sets cannot support the general claim.","tokens_in":12111,"tokens_out":2336,"would_cite":false,"duration_ms":157988,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligning model predictions with discovered eating-process models improves activity recognition accuracy and Macro F1 over raw model outputs.","keywords":["human activity recognition","process mining","alignment conformance checking","probabilistic events","graph convolutional networks","eating behaviour recognition","domain knowledge integration"],"falsifier":"Run the reported pipeline on a test set of videos in which subjects eat in an unusual order (for example, dessert first) while keeping the training process model unchanged; if accuracy drops below the raw GCN outputs, the method's benefit depends on the process model matching test behaviour.","tokens_in":11074,"feed_emoji":"🍽️","tokens_out":8950,"duration_ms":73141,"temperature":0.7,"pith_summary":"The paper sets out to show that human activity recognition can be improved by adding process knowledge—the typical order in which actions occur—to the outputs of a data-driven classifier, rather than by changing the classifier itself. The authors build a process model from labelled training videos of eating behaviour, then align a graph convolutional network's softmax probability trace against that model using a minimum-cost alignment. On the EatSense eating-behaviour videos, this post-hoc alignment raises accuracy and Macro F1 compared with the same GCNs used alone. The reason to care is that many eating actions look similar in motion data but occupy different positions in a routine, so the process provides disambiguating context.","feed_headline":"Process-aware alignment improves eating-action recognition","feed_subtitle":"Aligning graph-network outputs with eating-process models raises accuracy and Macro F1 on both test categories","key_machinery":"The central object is ProbCost, an alignment-based conformance-checking procedure for probabilistic events. It takes a process model and a trace of probability vectors—each column giving the GCN's softmax scores across the sixteen eating actions—and combines them into a synchronous product net. An A* search then finds the minimum-cost path, where synchronous moves, model moves, and log moves are priced using the event probabilities and a threshold epsilon. Low epsilon lets lower-probability events align with the process model, making the process authoritative; high epsilon keeps the classifier's most probable labels, making the model output authoritative. Tuning epsilon on the validation set is what lets the method adaptively weigh the two information sources.","core_discovery":"On its own terms, the paper claims that incorporating process information through alignment-based conformance checking improves human activity recognition. The pipeline extracts a multiclass probability matrix from a GCN, discovers a process model from the activity labels in a training split, and then aligns the probabilities for each test video with the process model. The alignment selects the activity labels that minimise a cost balancing the model's confidence against conformance to the discovered process, with a threshold epsilon tuned on a validation set. In the reported results, 2s-AGCN accuracy rises from 0.67 to 0.70 on subject 1 and from 0.90 to 0.99 on eating roti; CTR-GCN rises from 0.57 to 0.71 and from 0.81 to 0.99, with Macro F1 improving in every case.","pith_inferences":["A natural next experiment is cross-category transfer: learn the process model from one subject or food and apply it to another, since the reported splits keep the same category for discovery and testing.","Because alignment is post-hoc, one could add a guard that only overrides the classifier when its confidence is low, reducing the risk of disturbing already-correct high-confidence predictions.","The aligned path through the process model could double as an explanation artifact, showing the sequence that justifies a corrected label.","The optimal epsilon likely depends on the category and on classifier confidence, so a deployment might need per-process epsilon selection rather than a single value."],"forward_implications":["The same trained GCN improves without retraining, because the alignment is a post-processing step on the softmax outputs.","Confusable actions that appear in different parts of a routine, such as chewing and holding food at the table, can be corrected using the surrounding sequence.","The threshold epsilon gives practitioners a single, interpretable dial for trusting the process model versus the raw classifier.","Both baseline GCN architectures improve in accuracy and Macro F1 on the subject-1 and roti test sets after alignment.","The framework extends to any activity with a repeatable pattern, since only an event log and a probabilistic classifier are needed."],"supporting_citations":[{"why":"Supplies the ProbCost alignment algorithm that the whole method is built around.","marker":"Zheng, Papapanagiotou, and Fleuriot 2024"},{"why":"Provides the EatSense skeleton dataset of eating videos and the baseline GCN selection used in the experiments.","marker":"Raza et al. 2023"},{"why":"Defines the 2s-AGCN baseline whose softmax outputs are aligned and improved.","marker":"Shi et al. 2019"},{"why":"Defines the CTR-GCN baseline whose softmax outputs are aligned and improved.","marker":"Chen et al. 2021"},{"why":"Supplies the process-mining concepts and fitness/precision metrics used to choose dependency thresholds.","marker":"van der Aalst 2016"},{"why":"Provides the heuristic-miner algorithm used to discover the process models from training videos.","marker":"Weijters, Aalst, and Medeiros 2006"},{"why":"Provides the conformance-checking alignment formulation that ProbCost extends to probabilistic events.","marker":"Carmona et al. 2018"}],"fun_headline_variants":["Process-aware alignment improves HAR accuracy and F1","Aligning model outputs with process models boosts activity recognition","Process-aware conformance checking elevates HAR performance","Process knowledge improves human activity recognition via alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The process model learned from the training videos is representative enough of the test videos that forcing test predictions to conform to it fixes misclassifications rather than introducing new ones.","fun_headline_variants_meta":{"raw":{"variants":["Process-aware alignment improves HAR accuracy and F1","Aligning model outputs with process models boosts activity recognition","Process-aware conformance checking elevates HAR performance","Process knowledge improves human activity recognition via alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":3066,"prompt_tokens":818,"completion_tokens":2248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":2189}},"tokens_in":434,"tokens_out":2248,"duration_ms":15401,"temperature":1.0,"reasoning_tokens":2189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:18:10.524069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reported pipeline on a test set of videos in which subjects eat in an unusual order (for example, dessert first) while keeping the training process model unchanged; if accuracy drops below the raw GCN outputs, the method's benefit depends on the process model matching test behaviour.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ProbCost alignment algorithm that the whole method is built around."},{"cited_title":"A.; Chen, L.; Nanbo, L.; and Fisher, R","cited_arxiv_id":null,"evidence_quote":"Provides the EatSense skeleton dataset of eating videos and the baseline GCN selection used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 2s-AGCN baseline whose softmax outputs are aligned and improved."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CTR-GCN baseline whose softmax outputs are aligned and improved."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the process-mining concepts and fitness/precision metrics used to choose dependency thresholds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the heuristic-miner algorithm used to discover the process models from training videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the conformance-checking alignment formulation that ProbCost extends to probabilistic events."}],"review_version":1}