Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Process-aware Human Activity Recognition

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligning model predictions with discovered eating-process models improves activity recognition accuracy and Macro F1 over raw model outputs.

desk verdict Plausible pipeline, weak evaluation: the reported gains likely include a large dose of generic temporal smoothing, and the tiny test sets cannot support the general claim. read the letter →

arxiv 2411.08814 v1 pith:XEWAC4GE submitted 2024-11-13 cs.AI cs.LG

Jiawei Zheng , Petros Papapanagiotou , Jacques D. Fleuriot , Jane Hillston This is my paper · ORCID
classification cs.AIcs.LG
keywords humanactivityrecognitionprocessminingalignmentconformancecheckingprobabilisticeventsgraphconvolutionalnetworkseatingbehaviourdomainknowledgeintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that human activity recognition can be improved by adding process knowledge—the typical order in which actions occur—to the outputs of a data-driven classifier, rather than by changing the classifier itself. The authors build a process model from labelled training videos of eating behaviour, then align a graph convolutional network's softmax probability trace against that model using a minimum-cost alignment. On the EatSense eating-behaviour videos, this post-hoc alignment raises accuracy and Macro F1 compared with the same GCNs used alone. The reason to care is that many eating actions look similar in motion data but occupy different positions in a routine, so the process provides disambiguating context.

What carries the argument

The central object is ProbCost, an alignment-based conformance-checking procedure for probabilistic events. It takes a process model and a trace of probability vectors—each column giving the GCN's softmax scores across the sixteen eating actions—and combines them into a synchronous product net. An A* search then finds the minimum-cost path, where synchronous moves, model moves, and log moves are priced using the event probabilities and a threshold epsilon. Low epsilon lets lower-probability events align with the process model, making the process authoritative; high epsilon keeps the classifier's most probable labels, making the model output authoritative. Tuning epsilon on the validation set is what lets the method adaptively weigh the two information sources.

What would settle it

Run the reported pipeline on a test set of videos in which subjects eat in an unusual order (for example, dessert first) while keeping the training process model unchanged; if accuracy drops below the raw GCN outputs, the method's benefit depends on the process model matching test behaviour.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that incorporating process information through alignment-based conformance checking improves human activity recognition. The pipeline extracts a multiclass probability matrix from a GCN, discovers a process model from the activity labels in a training split, and then aligns the probabilities for each test video with the process model. The alignment selects the activity labels that minimise a cost balancing the model's confidence against conformance to the discovered process, with a threshold epsilon tuned on a validation set. In the reported results, 2s-AGCN accuracy rises from 0.67 to 0.70 on subject 1 and from 0.90 to 0.99 on eating roti; CTR-GCN rises from 0.57 to 0.71 and from 0.81 to 0.99, with Macro F1 improving in every case.

Load-bearing premise

The process model learned from the training videos is representative enough of the test videos that forcing test predictions to conform to it fixes misclassifications rather than introducing new ones.

Editorial extensions

If this is right

  • The same trained GCN improves without retraining, because the alignment is a post-processing step on the softmax outputs.
  • Confusable actions that appear in different parts of a routine, such as chewing and holding food at the table, can be corrected using the surrounding sequence.
  • The threshold epsilon gives practitioners a single, interpretable dial for trusting the process model versus the raw classifier.
  • Both baseline GCN architectures improve in accuracy and Macro F1 on the subject-1 and roti test sets after alignment.
  • The framework extends to any activity with a repeatable pattern, since only an event log and a probabilistic classifier are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is cross-category transfer: learn the process model from one subject or food and apply it to another, since the reported splits keep the same category for discovery and testing.
  • Because alignment is post-hoc, one could add a guard that only overrides the classifier when its confidence is low, reducing the risk of disturbing already-correct high-confidence predictions.
  • The aligned path through the process model could double as an explanation artifact, showing the sequence that justifies a corrected label.
  • The optimal epsilon likely depends on the category and on classifier confidence, so a deployment might need per-process epsilon selection rather than a single value.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a process-aware human activity recognition (HAR) method. Given the softmax probability outputs of a GCN-based HAR model, the method treats these outputs as a probabilistic event trace and aligns it, via the existing ProbCost algorithm, with a process model discovered from ground-truth activity labels using the Heuristic Miner. The alignment is controlled by a confidence threshold epsilon, and the aligned path is used as the final activity labels. Experiments on the EatSense dataset, restricted to two categories (videos of subject 1 and videos of eating roti), compare the original outputs of 2s-AGCN and CTR-GCN with the post-alignment outputs, reporting higher accuracy and Macro F1-scores after alignment. The central claim is that incorporating process information improves HAR performance over the baseline GCN outputs.

Significance. If the claimed improvements are real, the paper offers a concrete pipeline for injecting structured process knowledge into HAR outputs, which is a plausible and useful direction. The manuscript is clearly written, the pipeline is simple and reproducible, and it is a strength that the test set is held out, so the reported gains are not fitted by construction. However, the evaluation is currently too weak to support the claim as stated: the test sets contain only about 4 and 3 videos in the two categories, there are no error bars or significance tests, and no control baseline isolates the process-model contribution from generic temporal smoothing. The paper is therefore better viewed as a preliminary demonstration than as a confirmatory experimental study.

major comments (4)
  1. [Table 3 and Section 5, Experimental design] The evaluation is based on very small test sets: subject 1 has 19 videos and eating roti has 13 videos, so with a 60/20/20 split the test sets contain roughly 4 and 3 videos, respectively. The reported improvements, such as CTR-GCN accuracy rising from 0.57 to 0.71 on subject 1, can be produced by correcting only a few individual events. No confidence intervals, error bars, per-video results, or significance tests are reported. Because the central claim is that the approach 'achieves better accuracy and Macro F1-score compared to baseline models,' the evidence in Table 3 is not strong enough without some measure of variability or a significance test.
  2. [Section 5, Results and discussion; Eq. (2)] The experiment does not isolate the contribution of the process model from that of generic temporal smoothing. Since epsilon is tuned to 0.1, the cost of a model move is -log(0.1) ≈ 2.3, while a synchronous move with probability 0.03 costs about 3.5, so the alignment systematically prefers process-model paths over low-confidence events. Because the process model is discovered from the training labels, it encodes the most frequent activities and transitions. A first-order Markov model with Viterbi decoding, using transition probabilities estimated from the same training labels and the same softmax emissions, would serve as a natural control baseline. Without such a control, the observed gains in Table 3 cannot be attributed specifically to process information rather than to generic sequence smoothing or class-prior correction.
  3. [Section 5, Process discovery] The experimental design states that the validation set is used to tune the parameters of process discovery, but the dependency threshold for the Heuristic Miner is in fact selected by maximizing the F-score of fitness and precision computed on the training log (Table 2). This means the process model is selected on the same data used to fit it, which can lead to overfitting to the training sequences and does not validate the model's generality on held-out data. The authors should either select the dependency threshold on the validation set or clarify why the training-log F-score is an appropriate criterion.
  4. [Section 5, Results and discussion] The paper illustrates the method with one corrected example (chewing corrected to food in hand at table). While the example is useful for intuition, it is anecdotal and does not indicate how often the alignment changes predictions, whether such changes are systematically beneficial, or how many test events were affected. Reporting the number of alignments that changed labels and the confusion-matrix-level breakdown would give a more complete picture of where the improvement comes from.
minor comments (6)
  1. [Abstract and Introduction] There are small language and typographical issues, e.g., 'These HAR methods relies solely' should be 'rely solely'.
  2. [Section 5, Experimental design] The phrase 'classify the videos into distinct categories through two strategic approaches' is slightly confusing; it would be clearer to say the authors group videos by subject and by food type.
  3. [Section 3, Eq. (2)] In the description of Eq. (2), 'ALM' should be 'A LM' (a log move) for readability.
  4. [Throughout] The text repeatedly uses 'month' where 'mouth' is meant (e.g., Section 2 and Figure 3 descriptions); these should be corrected.
  5. [Figure 4] The y-axis label 'Accuracy' would benefit from specifying that it is the validation-set accuracy, to match the text.
  6. [References] The reference 'Contributors, M.' should be formatted as the OpenMMLab project contributors; consider using the standard citation for OpenMMLab (mmaction2).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the process model and epsilon are fit only to training/validation data, and the reported gains are measured on held-out test videos.

full rationale

The central claim is evaluated on a held-out test split. The probabilistic event traces come from the softmax outputs of GCN models trained on the 60% training split; the process model is discovered by Heuristic Miner from the same training split's activity labels; and epsilon is tuned on a separate validation split to maximize accuracy. Table 3 then compares original GCN outputs with ProbCost-aligned outputs on the remaining test videos, so no reported accuracy or Macro F1 number is a fitted quantity reused as a prediction. The cost function in Eq. (2) is explicitly stated rather than imported as an unexamined black box, and the self-citation to the authors' prior ProbCost work supplies an algorithm, not a conclusion that the present experiments are forced to reproduce. Concerns about temporal smoothing or the small number of test videos are validity or generalization risks, not circularity, and no equation in the paper reduces a predicted result to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two fitted scalar parameters (epsilon and the dependency threshold), a data-fitted process model, and the domain assumption that eating sequences are structurally repeatable. No new physical or conceptual entities are introduced.

free parameters (2)
  • epsilon (confidence threshold in ProbCost cost function) = 0.1 for both subject 1 and eating roti categories
    Tuned on the validation set to maximize activity recognition accuracy; controls the trade-off between trusting the ML probabilities and the process model (Section 5, Results and discussion).
  • dependency threshold for Heuristic Miner = 0.85 for subject 1, 0.9 for eating roti
    Selected by comparing fitness, precision, and F-score of discovered process models on the training log; determines which less-frequent paths are kept (Table 2).
assumptions (4)
  • domain assumption Eating behavior follows repeatable, discoverable process patterns, and a process model learned from training videos transfers to test videos.
    The entire method depends on this; see Section 5, Process discovery, where the Heuristic Miner builds a model from a small training set and the model is then applied to test predictions.
  • domain assumption Softmax probabilities from the GCN models are meaningful confidence values usable as probabilistic events in the cost function.
    The alignment cost in equation (2) treats each class probability w(t) as a measure of likelihood; no calibration or uncertainty quantification is performed (Section 4, Extracting probabilistic traces).
  • ad hoc to paper The minimal-cost alignment path found by A* corresponds to the most accurate activity labeling after tuning epsilon.
    The method equates lower alignment cost with higher recognition accuracy; this is operationalized by tuning epsilon on validation accuracy rather than derived from first principles (Section 4, Alignment).
  • standard math A* search and the process mining algorithms (Heuristic Miner) work as described in the cited literature.
    Relies on Russell and Norvig (2016), Weijters et al. (2006), and the PM4Py implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Process-aware Human Activity Recognition." pith.science (2026). https://pith.science/paper/XEWAC4GE

@misc{pith2026241108814,
  author       = {Pith},
  title        = {Pith review of: Process-aware Human Activity Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEWAC4GE}},
  note         = {Machine review of arXiv:2411.08814}
}
read the original abstract

Humans naturally follow distinct patterns when conducting their daily activities, which are driven by established practices and processes, such as production workflows, social norms and daily routines. Human activity recognition (HAR) algorithms usually use neural networks or machine learning techniques to analyse inherent relationships within the data. However, these approaches often overlook the contextual information in which the data are generated, potentially limiting their effectiveness. We propose a novel approach that incorporates process information from context to enhance the HAR performance. Specifically, we align probabilistic events generated by machine learning models with process models derived from contextual information. This alignment adaptively weighs these two sources of information to optimise HAR accuracy. Our experiments demonstrate that our approach achieves better accuracy and Macro F1-score compared to baseline models.

Figures

Figures reproduced from arXiv: 2411.08814 by the authors.

Figure 1
Figure 1. Example of a process model. interrupted by a phone call or need to attend to a child, caus￾ing them to put down their cutlery unexpectedly and alter their usual sequence. Moreover, individuals often have personal eating habits that vary widely, such as the order in which they eat their food. Some might prefer starting with lighter items like soup before progressing to heavier main dishes, while oth￾ers might dive st… view at source ↗
Figure 2
Figure 2. The framework of process-driven human activity recognition. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Discovered process models. The number in the node label indicates the frequency of the corresponding activity. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy of activity recognition across varying [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Aminikhanghahi, S.; and Cook, D. J. 2019. Enhancing Activity Recognition Using CPD-based Activity Segmentation. Pervasive and Mobile Computing, 53: 75--89

    work page 2019

  4. [4]

    A.; Ehatisham-ul-Haq , M.; Naeem, U.; and Khalid, A

    Asim, Y.; Azam, M. A.; Ehatisham-ul-Haq , M.; Naeem, U.; and Khalid, A. 2020. Context- Aware Human Activity Recognition ( CAHAR ) in-the- Wild Using Smartphone Accelerometer . IEEE Sensors Journal, 20(8): 4361--4371

    work page 2020

  5. [5]

    J.; and van der Aalst, W

    Berti, A.; van Zelst, S. J.; and van der Aalst, W. M. P. 2019. Process Mining for Python (PM4Py): Bridging the Gap Between Process- and Data Science. In ICPM Demo Track 2019, volume 2374, 13--16

    work page 2019

  6. [6]

    Carmona, J.; van Dongen , B.; Solti, A.; and Weidlich, M. 2018. Conformance Checking : Relating Processes and Models . Cham: Springer International Publishing. ISBN 978-3-319-99413-0 978-3-319-99414-7

    work page 2018

  7. [7]

    D.; and Wang, H

    Chen, L.; Nugent, C. D.; and Wang, H. 2012. A Knowledge-Driven Approach to Activity Recognition in Smart Homes . IEEE Trans.\ Knowledge and Data Eng., 24(6): 961--974

    work page 2012

  8. [8]

    Chen, Y.; Zhang, Z.; Yuan, C.; Li, B.; Deng, Y.; and Hu, W. 2021. Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition. In IEEE/CVF Int.\ Conf.\ on Computer Vision, ICCV , 13339--13348

    work page 2021

Show all 26 references
  1. [9]

    Contributors, M. 2020. OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark. https://github.com/open-mmlab/mmaction2

    2020

  2. [10]

    Deep, S.; and Zheng, X. 2019. Leveraging CNN and Transfer Learning for Vision-based Human Activity Recognition . In ITNAC 2019 , 1--4

    2019

  3. [11]

    Guo, S.; Wang, D.; Feng, Z.; Chen, J.; and Guo, W. 2023. Di- CNN : Domain-Knowledge-Informed Convolutional Neural Network for Manufacturing Quality Prediction . Sensors (Basel, Switzerland), 23(11): 5313

    2023

  4. [12]

    Gupta, A.; and Davis, L. S. 2007. Objects in Action : An Approach for Combining Action Understanding and Object Perception . In 2007 IEEE CVPR , 1--8

    2007

  5. [13]

    D.; and Sheng, V

    Gupta, H. D.; and Sheng, V. S. 2020. A Roadmap to Domain Knowledge Integration in Machine Learning . In IEEE Int.\ Conf.\ on Knowledge Graph ( ICKG ) , 145--151

    2020

  6. [14]

    K.; Pathak, R

    Gupta, N.; Gupta, S. K.; Pathak, R. K.; Jain, V.; Rashidi, P.; and Suri, J. S. 2022. Human Activity Recognition in Artificial Intelligence Framework: A Narrative Review. Artificial Intelligence Review, 55(6): 4755--4808

    2022

  7. [15]

    James, G.; Witten, D.; Hastie, T.; Tibshirani, R.; and Taylor, J. 2023. An Introduction to Statistical Learning: with Applications in Python. Springer

    2023

  8. [16]

    J.; Fahland, D.; and Van Der Aalst, W

    Leemans, S. J.; Fahland, D.; and Van Der Aalst, W. M. 2014. Process and deviation exploration with inductive visual miner. In 12th Int.\ Conf.\ on Business Process Management, BPM 2014, 46--50

    2014

  9. [17]

    A.; Zhang, H.; Henricson, E.; and Liu, X

    Liu, R.; Ramli, A. A.; Zhang, H.; Henricson, E.; and Liu, X. 2022. An Overview of Human Activity Recognition Using Wearable Sensors : Healthcare and AI . In Internet of Things -- ICIOT 2021 , LNCS , 1--14

    2022

  10. [18]

    A.; Chen, L.; Nanbo, L.; and Fisher, R

    Raza, M. A.; Chen, L.; Nanbo, L.; and Fisher, R. B. 2023. EatSense : Human Centric, Action Recognition and Localization Dataset for Understanding Eating Behaviors and Quality of Motion Assessment. Image and Vision Computing, 137: 104762

    2023

  11. [19]

    J.; and Norvig, P

    Russell, S. J.; and Norvig, P. 2016. Artificial intelligence: a modern approach. Pearson

    2016

  12. [20]

    Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019. Two- Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition . In IEEE CVPR , 12026--12035

    2019

  13. [21]

    Tufano, M.; Lasschuijt, M.; Chauhan, A.; Feskens, E. J. M.; and Camps, G. 2022. Capturing Eating Behavior from Video Analysis : A Systematic Review . Nutrients, 14(22): 4847

    2022

  14. [22]

    van der Aalst , W. 2016. Process Mining . Springer. ISBN 978-3-662-49850-7 978-3-662-49851-4

    2016

  15. [23]

    Weijters, A.; Aalst, V.; and Medeiros, A. K. a. D. 2006. Process Mining with the HeuristicsMiner Algorithm. Eindhoven University of Technology

    2006

  16. [24]

    Wu, Y.-H.; Chen, Y.; Shirmohammadi, S.; and Hsu, C.-H. 2022. AI-Assisted Food Intake Activity Recognition Using 3D mmWave Radars . In Proc.\ 7th Int.\ Workshop on Multimedia Assisted Dietary Management , 81--89

    2022

  17. [25]

    Yan, S.; Xiong, Y.; and Lin, D. 2018. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition . Proc.\ of the AAAI Conf.\ on AI , 32(1)

    2018

  18. [26]

    Zheng, J.; Papapanagiotou, P.; and Fleuriot, J. 2024. Alignment-Based Conformance Checking over Probabilistic Events. In Proc.\ of 57th Hawaii Int.\ Conf.\ on System Sciences

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.