{"id":"ceb4d183-747d-4339-a94b-ab1b079ffa54","arxiv_id":"1908.02329","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SCALER stitches per-component finite-state models inferred from logs into a system model using timestamp-based dependencies, scaling to larger logs than existing tools.","lead":"SCALER infers a system-level behavioral model of a component-based system by first learning a model for each component and then stitching the component models together using timestamps and architecture. It handles much larger logs than a state-of-the-art tool, but the evidence that it is more accurate is weakened by an unfair baseline setup.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dependency heuristic is the load-bearing assumption: if timestamp-based 'most recent' pairing is wrong, the stitched model encodes incorrect interleavings and the accuracy claim collapses; no ground-truth validation is provided.","rationale":"The reader's weakest_assumption — the timestamp-based 'most recent communication entry' heuristic — is indeed the load-bearing element of the central accuracy claim. The paper's stated goal is to infer system-level models when only high-level dependencies and a possibly incomplete template list are available; the heuristic is the only bridge from component-level logs to inter-component dependencies. Its fragility with coarse timestamps is admitted in the text, yet the evaluation dataset uses exactly second-level timestamps, and no ground-truth validation is provided. If the heuristic fails, the stitched model is not merely suboptimal: it encodes incorrect interleavings, which can simultaneously inflate recall (by accepting invalid sequences) and lower specificity. The observed specificity drop in Table II (1–2 pp) is consistent with this failure mode, but the metrics cannot distinguish over-generalization from genuinely correct concurrency modeling. I therefore agree with the reader's identification of the weakest assumption. I also note the secondary concern of the MINT baseline receiving only one linearized log per execution, which further weakens the accuracy comparison; however, the dependency heuristic is more fundamental because it is the paper's core algorithmic contribution. A concrete, feasible test — comparing inferred dependencies against ground truth on a synthetic or instrumented system, and perturbing timestamp granularity — would settle whether the concern lands. The scalability evidence is strong and independent, so the paper remains conditionally acceptable pending that validation. Thus the reader's CONDITIONAL verdict is unchanged.","tokens_in":18484,"tokens_out":5339,"duration_ms":63607,"concrete_test":"Re-run SCALER on a synthetic or instrumented system where the true causal communication dependencies are known (e.g., messages carry unique correlation IDs or the system is instrumented to record send/receive pairs). Compare the leads-to relation extracted by the Section IV-A heuristic with the ground-truth dependency relation, computing precision and recall over communication-entry pairs. Then coarsen the timestamps (e.g., from milliseconds to seconds) and repeat the comparison, also re-measuring recall and specificity on held-out logs. If the heuristic's dependency-identification precision is low, or if coarsening timestamps changes the inferred leads-to pairs and materially shifts recall/specificity, the accuracy claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The accuracy half of the central claim ('yielding more accurate models') rests entirely on the log entries dependency heuristic in Section IV-A. For each architectural dependency cX→cY, a communication entry ceY of cY is attributed to the most recent communication entry ceX of cX, selected by minimum timestamp difference. The paper itself notes (Section IV-A, threats in Section V-D) that this heuristic 'may introduce some imprecisions' when timestamp granularity is coarse, and the benchmark logs use second-level timestamps (visible in Figure 2) — exactly the problematic regime. No ground-truth dependencies are provided, so the correctness of the inferred leads-to relation is never validated. If the heuristic misassigns a single communication pairing, the GRAFT/SLICE/INSERT stitching (Algorithms 2–4) inserts the wrong interleaving into the system-level gFSM, over-generalizing it: the model accepts invalid behaviors (reducing specificity) and may inflate recall. This pattern is visible in Table II: SCALER systematically has lower specificity (by 1–2 pp) and higher recall than MINT, consistent with over-generalization from incorrect stitching. The scalability contribution (divide-and-conquer on component logs) would survive even if the heuristic is wrong, but the 'more accurate models' claim would not. Since this heuristic is the core novelty of the stitching stage and is unvalidated in the paper's own experimental setting, it is the single most load-bearing assumption. The reader identified the same point; the unfair MINT baseline (one linearized log per execution) is a related but secondary concern, because even a perfect dependency oracle would not fully justify comparing against a non-concurrency-aware baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCALER, a divide-and-conquer technique for inferring a system-level guarded finite state machine (gFSM) from component-level execution logs. SCALER first infers a gFSM for each component using MINT, then identifies dependencies between log entries of different components with a timestamp-based heuristic, and finally stitches the component models into a system model using the GRAFT, SLICE, and INSERT algorithms. The evaluation on seven industrial datasets reports that SCALER is on average about 245 times faster than MINT, processes logs that cause MINT to time out, and achieves higher recall (by 25 to 56 percentage points) with only a small specificity loss. The paper argues that this makes SCALER a practical alternative to monolithic system-level inference when only high-level architectural dependencies and a possibly incomplete list of communication templates are available.","tokens_in":18707,"tokens_out":4452,"duration_ms":49143,"significance":"If the accuracy claims were fully supported, SCALER would be a practically valuable contribution: it addresses a real scalability bottleneck in model inference for component-based systems and relaxes the strong assumption, made by tools such as CSight, that communication channels are known. The scalability contribution is credible and well demonstrated: the reported speedups are large, MINT times out on the largest dataset, and the divide-and-conquer design is clearly described in pseudocode. The paper also provides anonymized logs and evaluation results, which supports reproducibility. However, the accuracy half of the central claim is currently not established: the evaluation uses the same dependency heuristic to construct the positive test logs that SCALER's stitching encodes into the model, and the MINT baseline is given strictly less information than SCALER. The manuscript therefore needs additional validation before the 'more accurate models' claim can be accepted.","major_comments":[{"comment":"The accuracy evaluation is partly circular. Positive system-level test logs are derived by linearizing component logs 'taking into account the log entries dependencies' using the same heuristic defined in Section IV-A that SCALER's stitching stage encodes into the inferred model. Recall therefore measures how well the model reproduces the heuristic's own assumptions, not how faithfully it captures the true system behavior. No ground-truth dependencies are available in the evaluation, so the claim that SCALER 'yields more accurate models' is not validated independently of the heuristic. I recommend validating the dependency heuristic on at least a subset of executions with manually labeled dependencies, or using an independent source such as known message channels or instrumented traces.","section":"Section V-C1 and Section IV-A"},{"comment":"The comparison with MINT is not apples-to-apples. MINT is given only one linearized system-level log per execution, selected using the extracted dependencies, while SCALER receives all component logs plus the dependency information and thereby effectively accesses all interleavings consistent with the heuristic. The reported recall gains (+25 to +56 pp) and the much larger inferred models (4.6x states, 7x transitions) are consistent with SCALER being given more input information and with over-generalization, rather than with superior inference precision. To support the claim, the experiment should include a control in which MINT is given multiple linearizations (or all feasible ones) per execution, or in which SCALER is restricted to the same single linearization per execution as MINT.","section":"Section V-B1 and Section V-C1"},{"comment":"The log-entry dependency heuristic is the load-bearing component of the accuracy claim, yet it is unvalidated in the paper's own experimental setting. The authors acknowledge that the heuristic 'may introduce some imprecisions' with coarse-grained timestamps, and the benchmark logs use second-level timestamps (visible in Figure 2), which is precisely the regime where the minimum-timestamp-difference rule can misorder concurrent events. If the leads-to relation is wrong, GRAFT and INSERT (Algorithms 2 and 4) insert incorrect interleavings into the system-level model, over-generalizing it. No sensitivity analysis with respect to timestamp granularity or pair selection is provided. I recommend adding an evaluation of the heuristic against known dependencies on a labeled subset, or an ablation that perturbs timestamp granularity to measure the effect on recall and specificity.","section":"Section IV-A and Section V-D"}],"minor_comments":[{"comment":"The variable sx in the call duplicateAndRedirectTransitions(t, sx, Fy) is not defined in Algorithm 4; it should be the source state of the duplicated outgoing transition.","section":"Algorithm 4, line 8"},{"comment":"The text says the sliced gFSM msl 'accepts only lc', but because SLICE starts from the current state of mc, the returned automaton accepts a suffix of the component log rather than the complete log lc; this should be clarified.","section":"Section IV-B2, Algorithm 3"},{"comment":"The 'Avg' row in Table II is computed over the six datasets for which MINT produced a model, while the D35K dataset is excluded; the caption or text should state this explicitly to avoid confusion.","section":"Section V-B1 and Table II"},{"comment":"The heuristic selects the communication entry with the minimum timestamp difference, but no tie-breaking rule is given for entries sharing the same timestamp; this is relevant because the benchmark logs have second-level granularity.","section":"Section IV-A"},{"comment":"The log definition assumes timestamps are natural numbers, but the paper later relies on synchronized clocks across components; the synchronization assumption should be stated in the formal definitions.","section":"Section II"},{"comment":"The text says the implementation is publicly available but also notes that the open-source license is still being reviewed; the artifact availability statement should be updated to reflect the actual status.","section":"Section I and Code Availability"}],"recommendation":"major_revision","confidential_remarks":"The scalability result is solid and likely publishable, but the accuracy comparison needs substantial additional validation before the central claim is credible. The circularity concern is not just a threat-to-validity caveat: the positive test logs are built with the same heuristic whose correctness the experiment is supposed to demonstrate. I would not reject the paper, because the issue is fixable with additional experiments, but it is load-bearing for the 'more accurate models' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the divide-and-conquer scaling story is real. The paper shows SCALER runs 27–428x faster than MINT on seven industrial datasets and handles a 35K-entry log that makes MINT time out. The stitching algorithms (GRAFT, SLICE, INSERT) are clearly described, and the leads-to heuristic is a sensible use of the only information available. That part is worth building on.\n\nThe soft spots are in the accuracy half of the central claim. The dependency heuristic in Section IV-A is the load-bearing piece: if it pairs the wrong communication entries, the stitched model encodes wrong interleavings. The paper admits timestamps are second-granularity and the heuristic 'may introduce some imprecisions.' But then the evaluation constructs system-level test logs using the same heuristic, so the held-out logs are exactly what the heuristic assumes. That makes the recall numbers largely self-fulfilling. The MINT baseline also only sees one linearized log per execution, so the +25 to +56 pp recall gain is almost by construction. The lower specificity (1–2 pp) is consistent with over-generalization but small.\n\nNone of this kills the scalability contribution. The speedups are real, and the component-level inference guarantees a model exists where MINT times out. But 'more accurate models' is not established. To accept that, we'd need either ground-truth dependencies or an evaluation that doesn't build test inputs from the same heuristic.\n\nThis paper deserves a serious referee. It is well-written, the algorithms are novel, and the scalability evidence is strong. My recommendation is peer review with a request to revise the accuracy evaluation—ideally to validate the leads-to relation on a few executions manually, and to compare against a concurrency-aware baseline or against MINT with multiple linearizations.","headline":"SCALER's divide-and-conquer scaling results are solid, but the accuracy claims rest on an unvalidated heuristic and a circular evaluation.","tokens_in":19350,"tokens_out":2840,"would_cite":true,"duration_ms":29444,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCALER infers a system-level guarded finite-state machine from component logs using only high-level architecture dependencies and a partially known set of communication templates, running about 245 times faster than the baseline and…","keywords":["model inference","guarded finite state machine","component logs","divide and conquer","scalability","log entry dependency","system-level model","log message templates"],"falsifier":"Take a system with known ground-truth inter-component interactions and rerun SCALER on logs whose timestamps are artificially truncated to second granularity or shifted by a small random skew. If recall drops substantially below the reported 0.79 average while the baseline's specificity stays high, the accuracy gain is an artifact of timestamp quality rather than of the stitching method itself.","tokens_in":18243,"feed_emoji":"🧩","tokens_out":7516,"duration_ms":71728,"temperature":0.7,"pith_summary":"SCALER claims that system-level behavioral models can be inferred from component logs without source code, channel definitions, or complete documentation, as long as the architecture's component dependencies and a partial list of communication templates are known. The method divides the problem: it infers a guarded finite-state machine for each component separately, then stitches these models together along dependency edges detected in the logs. On industrial satellite ground-control logs, the paper reports an average 245-fold speedup over a state-of-the-art baseline, successful processing of logs where the baseline times out, recall gains of 25 to 56 percentage points, and a specificity penalty of no more than 2 points. A reader would take away that scalability and inter-component accuracy are not in conflict once inference is decomposed per component and recombined through log-entry dependencies.","feed_headline":"Divide-and-conquer infers system models from component logs 245x faster","feed_subtitle":"System-level guarded state machines stitched from component logs beat the baseline's recall by up to 56 points.","key_machinery":"The load-bearing mechanism is the leads-to relation among log entries, which connects a communication entry of a caller component to a sequence of entries in a callee component. It is computed by a minimum-timestamp-difference heuristic over communication templates and architectural dependency edges, and it determines where component models are spliced. On top of this relation, STITCH builds one model per execution by GRAFT, which slices each component model along the log, recursively grafts the models of dependent components into the transition that reads the triggering entry, and uses parallel composition and determinization through INSERT; the per-execution models are then merged by DFA union. This decomposition keeps every inference step at component scale, which is what avoids the cubic behavior of direct system-level inference.","core_discovery":"The central claim is that a system-level guarded finite-state machine (gFSM) for a component-based system can be inferred from the individual component logs by first learning each component's model with an existing gFSM inference algorithm and then merging those models with a set of stitching operations. The merge is driven by a leads-to relation computed from architectural dependencies and communication templates: for each dependency edge, a communication entry in the callee is attributed to the most recent communication entry in the caller, selected by minimum timestamp difference. Log sequences following that entry are grouped into slices, grafted into the caller's model, and combined with parallel composition and determinization. The paper argues the resulting system model captures both individual component behaviors and inter-component interactions, and reports that on seven industrial datasets the approach is about 245 times faster on average than running the same component-level inferer on linearized system-level logs, with recall higher by 25 to 56 percentage points and specificity lower by at most 2 points.","pith_inferences":["The paper leaves implicit that the stitching stage is not tied to the particular component-level inferer; replacing it with any model learner that works at component scale should preserve most of the scalability gain, since the per-component logs stay small.","Beyond that, the accuracy bottleneck is the timestamp-based dependency heuristic; a probabilistic alignment of communication entries across components, or use of vector clocks when available, would be a natural replacement and is left as future work in the paper.","The observed trend that speedup increases with log size also suggests the gap is structural rather than incidental: the baseline's inference cost is cubic in model size while the stitching stage's cost is bounded by per-component logs plus a DFA union.","A direct stress test with deliberately coarse timestamps or injected clock skew would show how much of the claimed recall depends on timestamp quality, since the paper itself flags coarse timestamps as a source of imprecision."],"forward_implications":["Engineers can obtain a system-level gFSM without channel definitions, so third-party black-box components with limited documentation are no longer a blocker for model-based testing or program comprehension.","Logs that overwhelm a direct inferer (the paper shows a 35K-entry dataset where the baseline times out) become processable, and the speedup grows with log size, from 27x at 5K entries to 428x at 25K entries.","The inferred models are nearly correct (specificity above 0.96) and largely complete (average recall 0.79), meaning downstream tasks can rely on the model rejecting invalid behavior while missing only a minority of valid behavior.","Because the approach sees all system-level interleavings through the leads-to relation, it recovers valid behaviors that a single linearized system-level log hides from a monolithic inferer."],"supporting_citations":[{"why":"Supplies the guarded-FSM inference tool used to learn each component model and serves as the baseline tool for the scalability and accuracy comparison.","marker":"[8]"},{"why":"Provides the log-message template miner that turns raw component logs into the event templates needed for parsing by the inference tool.","marker":"[13]"},{"why":"Represents the distributed-system inference approach that requires channel definitions, the limitation SCALER removes.","marker":"[12]"},{"why":"Supplies the determinization procedure used during INSERT to eliminate nondeterminism introduced by duplication and redirection.","marker":"[18]"},{"why":"Grounds the problem in the classical FSM inference setting and its computational complexity, motivating the need for scalable approximations.","marker":"[5]"},{"why":"A MapReduce-based scalable inference approach that cannot be applied in black-box settings because it requires application-specific key-value encodings; SCALER is positioned against it.","marker":"[7]"}],"fun_headline_variants":["Divide-and-conquer system model inference 245x faster","Stitch component logs into system models accurately","System-level models from logs without channel specs","Scaler merges component models to beat recall by 56","Infer system behavior from component logs at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole accuracy argument rests on the heuristic that, for each architectural dependency, a communication entry in the callee is caused by the most recent communication entry in the caller, chosen by minimum timestamp difference and timestamp ordering; if timestamps are coarse, unsynchronized, or out of order, the inferred interleavings are wrong even though the stitching still runs quickly.","fun_headline_variants_meta":{"raw":{"variants":["Divide-and-conquer system model inference 245x faster","Stitch component logs into system models accurately","System-level models from logs without channel specs","Scaler merges component models to beat recall by 56","Infer system behavior from component logs at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1421,"prompt_tokens":1006,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":622,"tokens_out":415,"duration_ms":4893,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:47:39.441483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a system with known ground-truth inter-component interactions and rerun SCALER on logs whose timestamps are artificially truncated to second granularity or shifted by a small random skew. If recall drops substantially below the reported 0.79 average while the baseline's specificity stays high, the accuracy gain is an artifact of timestamp quality rather than of the stitching method itself.","supporting_citations":[{"cited_title":"Inferring extended finite state machine models from software executions,","cited_arxiv_id":null,"evidence_quote":"Supplies the guarded-FSM inference tool used to learn each component model and serves as the baseline tool for the scalability and accuracy comparison."},{"cited_title":"A search-based approach for accurate identification of log message for- mats,","cited_arxiv_id":null,"evidence_quote":"Provides the log-message template miner that turns raw component logs into the event templates needed for parsing by the inference tool."},{"cited_title":"Inferring models of concurrent systems from logs of their behavior with CSight,","cited_arxiv_id":null,"evidence_quote":"Represents the distributed-system inference approach that requires channel definitions, the limitation SCALER removes."},{"cited_title":"Generating annotated behavior models from end-user scenarios,","cited_arxiv_id":null,"evidence_quote":"Supplies the determinization procedure used during INSERT to eliminate nondeterminism introduced by duplication and redirection."},{"cited_title":"On the synthesis of finite-state machines from samples of their behavior,","cited_arxiv_id":null,"evidence_quote":"Grounds the problem in the classical FSM inference setting and its computational complexity, motivating the need for scalable approximations."},{"cited_title":"Inferring software behavioral models with mapreduce,","cited_arxiv_id":null,"evidence_quote":"A MapReduce-based scalable inference approach that cannot be applied in black-box settings because it requires application-specific key-value encodings; SCALER is positioned against it."}],"review_version":1}