REVIEW 3 major objections 6 minor 29 references
Scalable Inference of System-level Models from Component Logs
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read SCALER infers a system-level guarded finite-state machine from component logs using only high-level architecture dependencies and a partially known set of communication templates, running about 245 times faster than the baseline and…
desk verdict SCALER's divide-and-conquer scaling results are solid, but the accuracy claims rest on an unvalidated heuristic and a circular evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the leads-to relation among log entries, which connects a communication entry of a caller component to a sequence of entries in a callee component. It is computed by a minimum-timestamp-difference heuristic over communication templates and architectural dependency edges, and it determines where component models are spliced. On top of this relation, STITCH builds one model per execution by GRAFT, which slices each component model along the log, recursively grafts the models of dependent components into the transition that reads the triggering entry, and uses parallel composition and determinization through INSERT; the per-execution models are then merged by DFA union. This decomposition keeps every inference step at component scale, which is what avoids the cubic behavior of direct system-level inference.
What would settle it
Take a system with known ground-truth inter-component interactions and rerun SCALER on logs whose timestamps are artificially truncated to second granularity or shifted by a small random skew. If recall drops substantially below the reported 0.79 average while the baseline's specificity stays high, the accuracy gain is an artifact of timestamp quality rather than of the stitching method itself.
Extended reading notes
Core claim
The central claim is that a system-level guarded finite-state machine (gFSM) for a component-based system can be inferred from the individual component logs by first learning each component's model with an existing gFSM inference algorithm and then merging those models with a set of stitching operations. The merge is driven by a leads-to relation computed from architectural dependencies and communication templates: for each dependency edge, a communication entry in the callee is attributed to the most recent communication entry in the caller, selected by minimum timestamp difference. Log sequences following that entry are grouped into slices, grafted into the caller's model, and combined with parallel composition and determinization. The paper argues the resulting system model captures both individual component behaviors and inter-component interactions, and reports that on seven industrial datasets the approach is about 245 times faster on average than running the same component-level inferer on linearized system-level logs, with recall higher by 25 to 56 percentage points and specificity lower by at most 2 points.
Load-bearing premise
The whole accuracy argument rests on the heuristic that, for each architectural dependency, a communication entry in the callee is caused by the most recent communication entry in the caller, chosen by minimum timestamp difference and timestamp ordering; if timestamps are coarse, unsynchronized, or out of order, the inferred interleavings are wrong even though the stitching still runs quickly.
Editorial extensions
If this is right
- Engineers can obtain a system-level gFSM without channel definitions, so third-party black-box components with limited documentation are no longer a blocker for model-based testing or program comprehension.
- Logs that overwhelm a direct inferer (the paper shows a 35K-entry dataset where the baseline times out) become processable, and the speedup grows with log size, from 27x at 5K entries to 428x at 25K entries.
- The inferred models are nearly correct (specificity above 0.96) and largely complete (average recall 0.79), meaning downstream tasks can rely on the model rejecting invalid behavior while missing only a minority of valid behavior.
- Because the approach sees all system-level interleavings through the leads-to relation, it recovers valid behaviors that a single linearized system-level log hides from a monolithic inferer.
Reading between the lines
- The paper leaves implicit that the stitching stage is not tied to the particular component-level inferer; replacing it with any model learner that works at component scale should preserve most of the scalability gain, since the per-component logs stay small.
- Beyond that, the accuracy bottleneck is the timestamp-based dependency heuristic; a probabilistic alignment of communication entries across components, or use of vector clocks when available, would be a natural replacement and is left as future work in the paper.
- The observed trend that speedup increases with log size also suggests the gap is structural rather than incidental: the baseline's inference cost is cubic in model size while the stitching stage's cost is bounded by per-component logs plus a DFA union.
- A direct stress test with deliberately coarse timestamps or injected clock skew would show how much of the claimed recall depends on timestamp quality, since the paper itself flags coarse timestamps as a source of imprecision.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SCALER, a divide-and-conquer technique for inferring a system-level guarded finite state machine (gFSM) from component-level execution logs. SCALER first infers a gFSM for each component using MINT, then identifies dependencies between log entries of different components with a timestamp-based heuristic, and finally stitches the component models into a system model using the GRAFT, SLICE, and INSERT algorithms. The evaluation on seven industrial datasets reports that SCALER is on average about 245 times faster than MINT, processes logs that cause MINT to time out, and achieves higher recall (by 25 to 56 percentage points) with only a small specificity loss. The paper argues that this makes SCALER a practical alternative to monolithic system-level inference when only high-level architectural dependencies and a possibly incomplete list of communication templates are available.
Significance. If the accuracy claims were fully supported, SCALER would be a practically valuable contribution: it addresses a real scalability bottleneck in model inference for component-based systems and relaxes the strong assumption, made by tools such as CSight, that communication channels are known. The scalability contribution is credible and well demonstrated: the reported speedups are large, MINT times out on the largest dataset, and the divide-and-conquer design is clearly described in pseudocode. The paper also provides anonymized logs and evaluation results, which supports reproducibility. However, the accuracy half of the central claim is currently not established: the evaluation uses the same dependency heuristic to construct the positive test logs that SCALER's stitching encodes into the model, and the MINT baseline is given strictly less information than SCALER. The manuscript therefore needs additional validation before the 'more accurate models' claim can be accepted.
major comments (3)
- [Section V-C1 and Section IV-A] The accuracy evaluation is partly circular. Positive system-level test logs are derived by linearizing component logs 'taking into account the log entries dependencies' using the same heuristic defined in Section IV-A that SCALER's stitching stage encodes into the inferred model. Recall therefore measures how well the model reproduces the heuristic's own assumptions, not how faithfully it captures the true system behavior. No ground-truth dependencies are available in the evaluation, so the claim that SCALER 'yields more accurate models' is not validated independently of the heuristic. I recommend validating the dependency heuristic on at least a subset of executions with manually labeled dependencies, or using an independent source such as known message channels or instrumented traces.
- [Section V-B1 and Section V-C1] The comparison with MINT is not apples-to-apples. MINT is given only one linearized system-level log per execution, selected using the extracted dependencies, while SCALER receives all component logs plus the dependency information and thereby effectively accesses all interleavings consistent with the heuristic. The reported recall gains (+25 to +56 pp) and the much larger inferred models (4.6x states, 7x transitions) are consistent with SCALER being given more input information and with over-generalization, rather than with superior inference precision. To support the claim, the experiment should include a control in which MINT is given multiple linearizations (or all feasible ones) per execution, or in which SCALER is restricted to the same single linearization per execution as MINT.
- [Section IV-A and Section V-D] The log-entry dependency heuristic is the load-bearing component of the accuracy claim, yet it is unvalidated in the paper's own experimental setting. The authors acknowledge that the heuristic 'may introduce some imprecisions' with coarse-grained timestamps, and the benchmark logs use second-level timestamps (visible in Figure 2), which is precisely the regime where the minimum-timestamp-difference rule can misorder concurrent events. If the leads-to relation is wrong, GRAFT and INSERT (Algorithms 2 and 4) insert incorrect interleavings into the system-level model, over-generalizing it. No sensitivity analysis with respect to timestamp granularity or pair selection is provided. I recommend adding an evaluation of the heuristic against known dependencies on a labeled subset, or an ablation that perturbs timestamp granularity to measure the effect on recall and specificity.
minor comments (6)
- [Algorithm 4, line 8] The variable sx in the call duplicateAndRedirectTransitions(t, sx, Fy) is not defined in Algorithm 4; it should be the source state of the duplicated outgoing transition.
- [Section IV-B2, Algorithm 3] The text says the sliced gFSM msl 'accepts only lc', but because SLICE starts from the current state of mc, the returned automaton accepts a suffix of the component log rather than the complete log lc; this should be clarified.
- [Section V-B1 and Table II] The 'Avg' row in Table II is computed over the six datasets for which MINT produced a model, while the D35K dataset is excluded; the caption or text should state this explicitly to avoid confusion.
- [Section IV-A] The heuristic selects the communication entry with the minimum timestamp difference, but no tie-breaking rule is given for entries sharing the same timestamp; this is relevant because the benchmark logs have second-level granularity.
- [Section II] The log definition assumes timestamps are natural numbers, but the paper later relies on synchronized clocks across components; the synchronization assumption should be stated in the formal definitions.
- [Section I and Code Availability] The text says the implementation is publicly available but also notes that the open-source license is still being reviewed; the artifact availability statement should be updated to reflect the actual status.
Circularity Check
Accuracy evaluation is partially circular: held-out system-level logs used as ground truth are linearized using SCALER's own dependency heuristic, so the reported recall gain partly measures self-consistency rather than correctness.
-
self definitional
[Sections V-B1 and V-C1 (evaluation methodology); test-log construction; cf. Section IV-A dependency heuristic]
"We derived these system-level logs by linearizing the individual component logs in each execution, taking into account the log entries dependencies. To guarantee a fair comparison, these dependencies are the same as those extracted in the pre-processing stage of SCALER. ... for each execution in the test sets, we linearized the individual component logs to derive the system-level log."
SCALER's stitching stage encodes the log-entries dependencies extracted by the Section IV-A heuristic (ceX leads-to ceY if ceY is the most recent communication entry of cY after ceX). For evaluation, the ground-truth system-level logs are built by linearizing component logs 'taking into account the log entries dependencies' and repeating the same linearization for the test folds. Hence positive test logs are generated from the very dependency relation SCALER is designed to represent; accepting them measures self-consistency, not correctness of the inferred dependencies. The paper itself concedes the heuristic 'may introduce some imprecisions' with coarse-grained timestamps and no ground-truth dependencies are provided.
full rationale
The algorithmic derivation is largely self-contained: component-level gFSMs are inferred by the external MINT tool, and the GRAFT/SLICE/INSERT stitches are defined operations on those models and on the log-entries dependencies, with no fitted constants or self-citation used to force the model content. The only self-citation, MoLFI for template extraction, is not load-bearing for the central derivation because a domain expert refines the templates and MoLFI is externally evaluated. The single substantive circularity is in the accuracy evaluation: the held-out system-level logs used as ground truth for recall are linearized using the same log-entries dependency heuristic that SCALER computes in its pre-processing stage (Sections V-B1 and V-C1). Thus the +25 to +56 pp recall advantage is partly a measure of whether SCALER can reproduce its own dependency relation, not whether that relation matches reality. The paper's own Section IV-A limitation admits the heuristic 'may introduce some imprecisions,' yet no ground-truth dependencies are used to validate it. Specificity differences (-1 to -2 pp) and the scalability speedup (27x to 428x) are not forced by this construction, so the paper is not wholly circular; the score reflects the partial circularity of the central accuracy claim.
Assumptions & free parameters
free parameters (2)
- MINT state merging threshold k =
2
- MINT data classifier =
J48
assumptions (6)
- domain assumption System architecture is known as a dependency graph with a single root component cmain.
- domain assumption Component clocks are synchronized and timestamps are precise enough to order communication events.
- domain assumption Log message templates for communication events are available, possibly incomplete, and correctly identify communication events.
- domain assumption All log entries of a used component cY are ultimately consequences of log entries of cX, and the most recent prior communication entry identifies the dependency.
- domain assumption MINT infers component-level gFSMs with acceptable accuracy.
- standard math DFA union, parallel composition, and the determinization procedure behave as standard automata operations.
invented entities (1)
-
Log entry dependency (leads-to relation)
Cite this review
Pith. "Pith review of Scalable Inference of System-level Models from Component Logs." pith.science (2026). https://pith.science/paper/VNXJM6Y6
@misc{pith2026190802329,
author = {Pith},
title = {Pith review of: Scalable Inference of System-level Models from Component Logs},
year = {2026},
howpublished = {\url{https://pith.science/paper/VNXJM6Y6}},
note = {Machine review of arXiv:1908.02329}
}
read the original abstract
Behavioral software models play a key role in many software engineering tasks; unfortunately, these models either are not available during software development or, if available, they quickly become outdated as the implementations evolve. Model inference techniques have been proposed as a viable solution to extract finite-state models from execution logs. However, existing techniques do not scale well when processing very large logs, such as system-level logs obtained by combining component-level logs. Furthermore, in the case of component-based systems, existing techniques assume to know the definitions of communication channels between components. However, this information is usually not available in the case of systems integrating 3rd-party components with limited documentation. In this paper, we address the scalability problem of inferring the model of a component-based system from the individual component-level logs, when the only available information about the system are high-level architecture dependencies among components and a (possibly incomplete) list of log message templates denoting communication events between components. Our model inference technique, called SCALER, follows a divide and conquer approach. The idea is to first infer a model of each system component from the corresponding logs; then, the individual component models are merged together taking into account the dependencies among components, as reflected in the logs. We evaluated SCALER in terms of scalability and accuracy, using a dataset of logs from an industrial system; the results show that SCALER can process much larger logs than a state-of-the-art tool, while yielding more accurate models.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Discovering models of software processes from event-based data,
J. E. Cook and A. L. Wolf, “Discovering models of software processes from event-based data,”ACM Trans. Softw. Eng. Methodol., vol. 7, no. 3, pp. 215–249, 1998
work page 1998
-
[2]
Behaviourally adequate software testing,
G. Fraser and N. Walkinshaw, “Behaviourally adequate software testing,” in Proceedings of the 5th International Conference on Software Testing, Verification and Validation (ICST 2012). Piscataway, NJ, USA: IEEE, 2012, pp. 300–309
work page 2012
-
[3]
E. M. Clarke Jr, O. Grumberg, D. Kroening, D. Peled, and H. Veith, Model checking. MIT press, 2018
work page 2018
-
[4]
A framework for the competitive evaluation of model inference tech- niques,
N. Walkinshaw, K. Bogdanov, C. Damas, B. Lambeau, and P. Dupont, “A framework for the competitive evaluation of model inference tech- niques,” in Proceedings of the First International Workshop on Model Inference In Testing (MIIT 2010) . New York, NY , USA: ACM, 2010, pp. 1–9
work page 2010
-
[5]
On the synthesis of finite-state machines from samples of their behavior,
A. W. Biermann and J. A. Feldman, “On the synthesis of finite-state machines from samples of their behavior,” IEEE Trans. Comput. , vol. 100, no. 6, pp. 592–597, 1972
work page 1972
-
[6]
Leveraging existing instrumentation to automatically infer invariant- constrained models,
I. Beschastnikh, Y . Brun, S. Schneider, M. Sloan, and M. D. Ernst, “Leveraging existing instrumentation to automatically infer invariant- constrained models,” in Proceedings of the 19th ACM SIGSOFT Sym- posium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE 2011). New York, NY , USA: ACM, 2011, pp. 267–277
work page 2011
-
[7]
Inferring software behavioral models with mapreduce,
C. Luo, F. He, and C. Ghezzi, “Inferring software behavioral models with mapreduce,” Sci. Comput. Program. , vol. 145, pp. 13–36, 2017
work page 2017
-
[8]
Inferring extended finite state machine models from software executions,
N. Walkinshaw, R. Taylor, and J. Derrick, “Inferring extended finite state machine models from software executions,” Empir. Softw. Eng., vol. 21, no. 3, pp. 811–853, 2016
work page 2016
Show all 29 references
-
[9]
Gk-tail+ an efficient approach to learn software models,
L. Mariani, M. Pezz `e, and M. Santoro, “Gk-tail+ an efficient approach to learn software models,” IEEE Trans. Softw. Eng. , vol. 43, no. 8, pp. 715–738, 2017
2017
-
[10]
Inferring extended probabilistic finite-state automaton models from software executions,
S. S. Emam and J. Miller, “Inferring extended probabilistic finite-state automaton models from software executions,” ACM Trans. Softw. Eng. Methodol., vol. 27, no. 1, pp. 4:1–4:39, 2018
2018
-
[11]
Scalable parallelization of specification mining using distributed computing,
S. Wang, D. Lo, L. Jiang, S. Maoz, and A. Budi, “Scalable parallelization of specification mining using distributed computing,” in The Art and Science of Analyzing Software Data . Morgan Kaufmann, 2015, pp. 623–648
2015
-
[12]
Inferring models of concurrent systems from logs of their behavior with CSight,
I. Beschastnikh, Y . Brun, M. D. Ernst, and A. Krishnamurthy, “Inferring models of concurrent systems from logs of their behavior with CSight,” in Proceedings of the 36th International Conference on Software Engi- neering (ICSE 2014). New York, NY , USA: ACM, 2014, pp. 468–479
2014
-
[13]
A search-based approach for accurate identification of log message for- mats,
S. Messaoudi, A. Panichella, D. Bianculli, L. Briand, and R. Sasnauskas, “A search-based approach for accurate identification of log message for- mats,” in Proceedings of the 26th International Conference on Program Comprehension (ICPC 2018) . New York, NY , USA: ACM, 2018, pp. 167–177
2018
-
[14]
Results of the Abbadingo One DFA learning competition and a new evidence-driven state merging algorithm,
K. J. Lang, B. A. Pearlmutter, and R. A. Price, “Results of the Abbadingo One DFA learning competition and a new evidence-driven state merging algorithm,” in Proceedings of the 4th International Colloquium on Grammatical Inference (ICGI 1998) , ser. LNCS, vol. 1433. Berlin, He...
1998
-
[15]
Automatic extraction of object-oriented component interfaces,
J. Whaley, M. C. Martin, and M. S. Lam, “Automatic extraction of object-oriented component interfaces,” in Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002). New York, NY , USA: ACM, 2002, pp. 218–228
2002
-
[16]
Toward the reverse engineering of UML sequence diagrams for distributed Java software,
L. C. Briand, Y . Labiche, and J. Leduc, “Toward the reverse engineering of UML sequence diagrams for distributed Java software,” IEEE Trans. Softw. Eng., vol. 32, no. 9, pp. 642–663, 2006
2006
-
[17]
Internet time synchronization: The network time protocol,
D. L. Mills, “Internet time synchronization: The network time protocol,” Transactions on Communications, vol. 39, no. 10, pp. 1482–1493, 1991
1991
-
[18]
Generating annotated behavior models from end-user scenarios,
C. Damas, B. Lambeau, P. Dupont, and A. van Lamsweerde, “Generating annotated behavior models from end-user scenarios,” IEEE Trans. Softw. Eng., vol. 31, no. 12, pp. 1056–1073, 2005
2005
-
[19]
Performance metrics: Keeping the focus on runtime,
S. Sahni and V . Thanvantri, “Performance metrics: Keeping the focus on runtime,” IEEE Parallel Distributed Technology: Systems Applications , vol. 4, no. 1, pp. 43–56, 1996
1996
-
[20]
Using declarative specification to improve the understanding, extensibility, and comparison of model-inference algorithms,
I. Beschastnikh, Y . Brun, J. Abrahamson, M. D. Ernst, and A. Krishna- murthy, “Using declarative specification to improve the understanding, extensibility, and comparison of model-inference algorithms,” IEEE Trans. Softw. Eng., vol. 41, no. 4, pp. 408–428, 2015
2015
-
[21]
The daikon system for dynamic detection of likely invariants,
M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, “The daikon system for dynamic detection of likely invariants,” Sci. Comput. Program. , vol. 69, no. 1, pp. 35–45, 2007
2007
-
[22]
Automatic functional test generation using the extended finite state machine model,
K. Cheng and A. S. Krishnakumar, “Automatic functional test generation using the extended finite state machine model,” in Proceedings of the 30th Design Automation Conference (DAC 1993). New York, NY , USA: ACM, 1993, pp. 86–91
1993
-
[23]
T. M. Mitchell, Machine Learning. New York, NY , USA: McGraw-Hill, Inc., 1997
1997
-
[24]
Stamina: A competition to encourage the development and assessment of software model inference techniques,
N. Walkinshaw, B. Lambeau, C. Damas, K. Bogdanov, and P. Dupont, “Stamina: A competition to encourage the development and assessment of software model inference techniques,” Empir. Softw. Eng. , vol. 18, no. 4, pp. 791–824, Aug 2013
2013
-
[25]
Language identification in the limit,
E. M. Gold, “Language identification in the limit,” Information and Control, vol. 10, no. 5, pp. 447–474, 1967
1967
-
[26]
Mining message sequence graphs,
S. Kumar, S. Khoo, A. Roychoudhury, and D. Lo, “Mining message sequence graphs,” in Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011) . New York, NY , USA: ACM, 2011, pp. 91–100
2011
-
[27]
Inferring class level specifications for distributed systems,
S. Kumar, S.-C. Khoo, A. Roychoudhury, and D. Lo, “Inferring class level specifications for distributed systems,” in Proceedings of the 34th International Conference on Software Engineering (ICSE 2012) . Piscataway, NJ, USA: IEEE, 2012, pp. 914–924
2012
-
[28]
Compo- nent behavior discovery from software execution data,
C. Liu, B. van Dongen, N. Assy, and W. M. P. van der Aalst, “Compo- nent behavior discovery from software execution data,” in Proceedings of the Symposium Series on Computational Intelligence (SSCI 2016) . Piscataway, NJ, USA: IEEE, 2016, pp. 1–8
2016
-
[29]
Mapreduce: Simplified data processing on large clusters,
J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing on large clusters,” Commun. ACM, vol. 51, no. 1, pp. 107–113, 2008
2008
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.