{"id":"26442c64-14dd-4dfe-9b99-eff0537c0679","arxiv_id":"2506.06448","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Palette generates representative microservice benchmark systems from distributed traces using a topology built from a directed graph, a probabilistic automaton, and a graphical causal model.","lead":"This paper introduces Palette, a system that turns distributed trace data from large cloud services into runnable microservice benchmark systems. If it works, researchers could test new network stacks, schedulers, and tracing tools on realistic, generated versions of production systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Palette's Table 2 equations fit conditional means only; without explicit residual/noise sampling the generated traces cannot reproduce observed variance or tail latencies, so 'representative' is unsupported and likely false.","rationale":"Read in good faith: the paper's goal is to enable intervention experiments on representative systems generated from traces. The enabling assumption is that a GCM built with Table 2 equations and fit to traces yields samples representative of the original data. This is the load-bearing claim because interventions are only meaningful if the unmodified generated system matches the original's behavior. The authors do not provide code, data, or evaluation, so the claim is empirically unverified. More importantly, the described model cannot deliver the claim as written. Table 2 defines deterministic functional relationships: once parent latencies and Bernoulli call indicators are known, the child latency is fixed (up to the constant C). Real conditional latency distributions are not point masses; they exhibit scheduler noise, queuing effects, and multi-modal service times. Omitting exogenous noise means regenerated traces will have too little variance and suppressed tails. Tail latency is one of the properties highlighted in Section 2.1 as necessary for network-stack and congestion-control research, so this is not a minor detail. A GCM can be representative if it includes noise distributions for every node and the graph is correct, but the paper does not describe such noise estimation. The PFA captures branching and execution order, not residual latency variability, and the 'live measurements' in Section 3.4 only substitute observed parent values into the same deterministic equations. The reader's weakest assumption identified the same area: the unvalidated representativeness of the simple GCM. I partially agree; I would sharpen it to the missing residual-distribution specification, which makes the problem structural rather than merely unmeasured. No independent support (machine-checked proofs, released code, experiments) exists in the manuscript to mitigate this. The verdict of REJECT is appropriate; a CONDITIONAL with a required residual-modeling and validation section would be the most favorable outcome if the authors add the missing specification and held-out comparison.","tokens_in":9251,"tokens_out":4555,"duration_ms":51239,"concrete_test":"Implement the described pipeline on a public trace set (e.g., Alibaba microservice traces or a TrainTicket deployment): build the PFA/causal graph, fit Table 2 equations by regressing each child latency on parent latencies and call indicators, then generate a new trace set by sampling from the fitted model exactly as specified (no added residual noise). Compare marginal latency CDFs and p50/p99 of generated vs held-out traces for each service. If the generated p99 is more than ~20% below the held-out p99, or conditional variance given parent latencies collapses toward zero, the representativeness claim fails. Also fit the same model with an explicit residual distribution and show the contrast; the paper's described equations omit that residual, which is the disputed point.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 2.3 and 3 claim that any new sample from the GCM is representative of the original system, and Section 3.4 says the runtime 'samples the GCM' at each API. The only generative equations given are in Table 2: latency is a weighted sum (or max) of parent latencies plus an intercept, with Bernoulli call indicators; Section 3.2 says coefficients are fit to observed traces. The paper never specifies how residual or exogenous noise is estimated or sampled. If C and the lambdas are constants, then conditioned on the parent values and call indicators every sample of a node's latency is a single point mass. In real traces, calls with equal parent conditions still exhibit spread, multi-modality, and heavy tails. A deterministic conditional-mean model therefore will systematically understate variance and miss tail quantiles such as p99, which are precisely the performance characteristics the paper says researchers need (Section 2.1). Leaf nodes are not given a generative distribution either, so the problem cannot be pushed to the leaves. Figure 1 is a strawman illustration, not a validation. Thus the central representativeness claim is not just unvalidated; the described model is structurally incapable of it unless an omitted noise term is added.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Palette, a design for generating runnable microservice benchmark systems from distributed traces. It introduces a \"system topology\" abstraction that combines a directed graph of services and APIs, a probabilistic finite automaton (PFA) modeling execution paths, and a graphical causal model (GCM) for performance properties such as latency. Section 3 describes a pipeline that processes traces, builds and fits GCM equations (Table 2), converts the topology into specifications, generates code with Blueprint, and uses a GCM-based runtime to sample latency values and propagate causal baggage. The central assertion, stated in Section 2.3, is that because a GCM is fit to observed data, any new sample generated by the GCM is representative of the original data. The paper presents no implementation, no experiments, and no comparisons; the only figure shows a failure of a simple statistical baseline rather than a result produced by Palette.","tokens_in":9508,"tokens_out":5088,"duration_ms":50347,"significance":"If the central representativeness claim were established, Palette would address a recognized need for representative, configurable microservice benchmarks for intervention experiments. The proposed decomposition into a call graph, a PFA, and a GCM is a reasonable and extensible framing, and the paper explicitly identifies the design dimensions that matter for benchmark use cases, such as topology, execution paths, and tail latencies. However, in its current form the paper provides no evidence for representativeness: there is no prototype, no validation on held-out traces, no comparison with existing generators, and no demonstration that generated systems preserve scale, topology, or latency distributions. The significance is therefore potential rather than demonstrated. I credit the authors for clearly formulating the problem and for grounding the design in existing systems such as Blueprint and causal root-cause analysis, but the paper offers no reproducible artifacts and no falsifiable predictions that have been tested.","major_comments":[{"comment":"The representativeness claim is asserted rather than tested. Section 2.3 states that \"any new sample generated by the GCM will be representative of the original data\" solely because the GCM internally uses inferred causal effects from the observed data. This is a fitting-then-resampling procedure, and it is circular without validation on data not used for fitting. The paper provides no experiment in which generated traces or a generated system are compared with the original traces on structural properties (call graph, fan-in/fan-out), execution paths, latency CDFs, or tail quantiles. This validation is load-bearing because the motivation in Section 2.1 explicitly requires preservation of scale, topology, and tail latency.","section":"Section 2.3 and Section 3.2"},{"comment":"The generative equations in Table 2 specify only conditional means: each latency is a weighted sum (or a max) of parent latencies plus an intercept C, with Bernoulli call indicators, and no noise or residual term is defined anywhere in the paper. Section 3.2 says only that coefficients are fit to observed trace data. Consequently, conditioned on parent values and call indicators, every generated sample of a node's latency is a single point mass; the model has no mechanism to reproduce observed variance, multi-modality, or heavy tails. Leaf nodes are not assigned any generative distribution, so the problem cannot be delegated to the leaves. Since Section 2.1 identifies tail latency (for example, p99) as a key property for research use cases, the described model is structurally incapable of the paper's central representativeness claim as written.","section":"Table 2 and Section 3.4"},{"comment":"The paper is design-only: no prototype, deployment, or measurement is reported. There is no end-to-end generated system, no comparison against existing open-source microservice benchmarks such as TrainTicket or DeathStarBench, no comparison against trace replay or other workload generators, and no quantification of scale, generation time, or runtime overhead. For a paper that introduces \"Palette, a system that generates representative flexible macrobenchmark microservice systems,\" the absence of any evaluation is a central omission rather than a presentation issue.","section":"Sections 3.1-3.4"},{"comment":"Figure 1 is presented as motivation, but it only demonstrates that a \"simple statistical approach\" fails to reproduce the latency distribution of a downstream service; it does not show a generated latency CDF from Palette. The text gives no dataset, no experimental setup, and no precise definition of the baseline. As the only figure in the paper, it cannot support the claim that GCM-based generation preserves representativeness; it only establishes that a strawman fails.","section":"Figure 1"},{"comment":"The model fitting procedure is underspecified. The paper states that coefficients (the various lambda_a) are fit to observed trace data, but it gives no algorithm for estimating the parameters of the nonlinear equations in Table 2, which include Bernoulli call indicators, weighted sums, and max operations. Identifiability of coefficients in the concurrent (max) case is not discussed, and no procedure is given for estimating the intercept C or for handling missing calls. Without this information, it is impossible to assess whether the claimed model can actually be estimated from real traces.","section":"Section 3.2"}],"minor_comments":[{"comment":"The phrase \"macrobenchmark microservice systems\" in the title and abstract is grammatically awkward; consider \"macrobenchmark microservice systems\" or \"macrobenchmarks for microservice systems.\"","section":"Abstract and Title"},{"comment":"There is a typo in the sentence \"GCMs help preserve the properties of the original system an can generate new samples\": \"an\" should be \"and.\"","section":"Section 2.3"},{"comment":"The manuscript uses the template header \"Conference'17, July 2017, Washington, DC, USA,\" which should be updated for the submission venue.","section":"Page header"},{"comment":"Figure 1 would benefit from a description of the dataset, the exact statistical model used as the strawman, and a statement of how many traces or requests were used to compute the CDFs.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The manuscript is a design proposal with no implementation and no evaluation. The central representativeness claim is asserted from the fitting procedure rather than validated on held-out traces, and the specified generative equations cannot reproduce distributional properties without an explicit noise model. These gaps are too large for a minor or even a major revision of the current text: the generative model itself would need to be extended and a full system would need to be built and measured. If the authors add a complete implementation, a noise-aware generative model, and validation against held-out traces and existing benchmarks, a resubmission could be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the Palette paper. It's a clean, well-written design for a system that turns distributed traces into runnable microservice benchmarks, motivated by a real problem: existing open-source systems like TrainTicket are too small and unrepresentative. The decomposition into a directed graph for structure, a PFA for execution paths, and a GCM for performance is sensible, and the intervention story is well thought out. If it worked, it would be useful.\n\nWhat's actually new is the specific pipeline and the use of GCMs as a runtime steering mechanism. The components (CausalSim, Latenseer, Blueprint) are all prior work, but the combination is not something I've seen before. The writing is clear and the related work is well covered.\n\nThe soft spots are substantial. There is no implementation, no evaluation, and no comparison with existing generators. The central claim—that samples from the fitted GCM are representative of the original traces—is asserted rather than demonstrated, and the model as specified in Table 2 is deterministic: each latency is a weighted sum of parent latencies plus an intercept, with no error term. That means conditioned on the parents, every sample is a point mass. Real traces have spread, multi-modality, and heavy tails, so a deterministic conditional mean cannot reproduce p99 or even the variance. The stress-test note is right: unless an explicit residual term is added and validated, the representativeness claim is not just unverified; the described model is structurally incapable of it. Figure 1 only shows a strawman failing, not Palette succeeding.\n\nThe paper also skips some important design details: how the PFA is inferred from traces, how state coarsening works, and what 'samples the GCM' means when the equations are deterministic. These might be fixed in a fuller version, but they're not addressed here.\n\nWho is this for? Readers interested in the design space of benchmark generation and causal modeling of traces. It deserves a serious referee, because the problem is important and the design is thoughtful, but the referee should require an explicit noise model and at least a small-scale validation on public traces before the representativeness claim can be taken seriously. As is, it's a position paper, not a research result.","headline":"A clean, well-written design for synthesizing microservice benchmarks from traces, but with no implementation and a deterministic latency model that cannot reproduce variance, the load-bearing representativeness claim is unsupported.","tokens_in":10012,"tokens_out":2809,"would_cite":false,"duration_ms":28740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Palette turns distributed traces into runnable, representative microservice benchmark systems by modeling the observed system as a graphical causal model.","keywords":["microservices","distributed tracing","graphical causal models","benchmark generation","macrobenchmarks","system topology","intervention experiments","trace-driven generation"],"falsifier":"Compare the per-service latency distributions, branching probabilities, and call-graph fan-out of a generated system, running the trace workload, against a held-out portion of the original trace data; if the regenerated tail latencies, multi-modal peaks, or dependency-call ratios differ beyond sampling error, the representativeness claim fails.","tokens_in":9038,"feed_emoji":"🕸️","tokens_out":5285,"duration_ms":44981,"temperature":0.7,"pith_summary":"The paper argues that distributed traces, which record every API call, its latency, and its caller-callee relationships, carry enough information to reconstruct a runnable stand-in for the original service. It proposes Palette, which digests a trace dataset into a 'system topology' — a directed graph of services and APIs, a probabilistic finite automaton of each API's execution paths, and a graphical causal model of how parent latencies and call decisions shape downstream behavior. From this topology it generates a deployable macrobenchmark whose runtime samples the causal model, conditioned on live measurements, to reproduce the original system's structure and performance. If the approach works, researchers who only have traces — not production access — can run intervention experiments on systems that match production scale, topology, and execution patterns.","feed_headline":"From traces, Palette builds runnable microservice systems","feed_subtitle":"Graphical causal models preserve call patterns and latency, enabling intervention tests without production access.","key_machinery":"The load-bearing object is the system topology, a three-layer abstraction: a directed graph of APIs and services; a probabilistic finite automaton per API encoding how dependency calls are ordered (sequential, concurrent, or chosen) and at what probabilities; and a graphical causal model (GCM) per API whose nodes are latency (or other property) variables plus Bernoulli call indicators, with causal equations that combine parent values according to the PFA's behavior type. The GCM is the mechanism that lets the generated system sample execution times and payloads conditioned on the state of the call chain instead of from marginal statistics, and its fitted coefficients carry the performance characteristics of the original system into the benchmark.","core_discovery":"The central claim is that a system topology, learned purely from distributed traces, is a faithful-enough encoding that a generated system will be representative of the original. Structure is captured as a directed graph whose vertices are APIs and whose edges are caller-callee relations, partitioned into services; execution behavior is captured per API by a probabilistic finite automaton that distinguishes sequential and concurrent dependency calls; performance is captured per API by a graphical causal model with Bernoulli 'called' nodes and linear causal equations (weighted sums of parent latencies plus an intercept). Palette fits the coefficients to the observed traces, converts the topology into code, and at runtime plugs measured latencies into the causal equations so sampled work and delays reflect the current call context. The paper asserts that because the GCM infers causal effects from observed data, any new sample it generates is representative of the original data (Section 2.3).","pith_inferences":["A natural next test is comparing the generated system's end-to-end latency and error distributions against held-out traces from the same production system, under identical workload; the paper does not report such validation.","The linear additive causal equations in Table 2 will likely struggle with strongly non-linear or threshold-like dependencies (e.g., queueing effects that kick in at high concurrency); those cases would need richer equations or latent states.","Palette's GCM models a single request's causal path and does not model cross-request contention or load-dependent queueing, so generated systems may be representative only up to the boundary of independent requests.","One could test the abstraction's portability by deriving a topology from a public trace set, generating a system, and measuring whether an intervention's relative effect (e.g., adding a slow dependency) matches the effect predicted by the causal model."],"forward_implications":["If representative, researchers can run interventions (modify topology, specs, instantiation) while holding everything else fixed, because the model isolates each causal factor.","Generated systems inherit production-scale fan-in and fan-out and multi-modal latency from traces rather than the limited design space of hand-built open-source benchmarks.","Trace datasets already collected by large operators become reusable experimental infrastructure, not just post-mortem artifacts.","The same pipeline extends to other performance properties such as payload sizes by adding nodes and equations; live measurements plus baggage propagation let the runtime steer behavior back toward the observed joint distribution when hardware changes cause divergence."],"supporting_citations":[{"why":"Defines graphical causal models, the formal foundation of the topology's performance model.","marker":"[9]"},{"why":"The toolchain the paper builds on to turn topology specifications into a deployable implementation.","marker":"[2]"},{"why":"Provides the baggage propagation mechanism Palette uses to carry causal measurements downstream.","marker":"[24]"},{"why":"Prior work showing GCMs trained on traces can generate representative samples for simulation.","marker":"[1]"},{"why":"Related causal modeling of end-to-end latency from traces that motivates the GCM runtime.","marker":"[38]"},{"why":"Prior work generating microservice workloads and graphs from traces, the context Palette extends.","marker":"[8]"}],"fun_headline_variants":["Palette builds representative microservices from traces","From traces to runnable microservice systems with Palette","Trace-driven microservice generation with Palette","Palette: turning traces into macrobenchmark systems","Generate representative microservice systems from traces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a simple causal model fit to observed traces — weighted sums of parent latencies plus an intercept, with Bernoulli call indicators — will generate new samples, including tail latencies and multi-modal behavior, that are representative of the original system, a claim the paper states but does not validate on held-out data.","fun_headline_variants_meta":{"raw":{"variants":["Palette builds representative microservices from traces","From traces to runnable microservice systems with Palette","Trace-driven microservice generation with Palette","Palette: turning traces into macrobenchmark systems","Generate representative microservice systems from traces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1509,"prompt_tokens":883,"completion_tokens":626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":499,"tokens_out":626,"duration_ms":5981,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:55:43.763390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the per-service latency distributions, branching probabilities, and call-graph fan-out of a generated system, running the trace workload, against a held-out portion of the original trace data; if the regenerated tail latencies, multi-modal peaks, or dependency-call ratios differ beyond sampling error, the representativeness claim fails.","supporting_citations":[{"cited_title":"Graphical causal models","cited_arxiv_id":null,"evidence_quote":"Defines graphical causal models, the formal foundation of the topology's performance model."},{"cited_title":"Blueprint: A toolchain for highly-reconfigurable microservice appli- cations","cited_arxiv_id":null,"evidence_quote":"The toolchain the paper builds on to turn topology specifications into a deployable implementation."},{"cited_title":"Universal context propagation for distributed system instrumentation","cited_arxiv_id":null,"evidence_quote":"Provides the baggage propagation mechanism Palette uses to carry causal measurements downstream."},{"cited_title":"In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1115–1147, 2023","cited_arxiv_id":null,"evidence_quote":"Prior work showing GCMs trained on traces can generate representative samples for simulation."},{"cited_title":"Latenseer: Causal modeling of end-to-end latency distributions by harnessing distributed tracing","cited_arxiv_id":null,"evidence_quote":"Related causal modeling of end-to-end latency from traces that motivates the GCM runtime."},{"cited_title":"A mi- croservice graph generator with production characteristics, 2024","cited_arxiv_id":null,"evidence_quote":"Prior work generating microservice workloads and graphs from traces, the context Palette extends."}],"review_version":1}