{"id":"c529c1e4-b8f7-4d01-97f0-8400d98620db","arxiv_id":"2501.08479","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Skyrise is a fully serverless SQL engine running on AWS Lambda and S3 that matches the performance and cost of server-based and prior serverless systems on scan-heavy TPC-H queries.","lead":"This paper introduces Skyrise, a SQL query processor that runs entirely on serverless cloud functions and storage, with no persistent servers. It shows that for three analytical TPC-H queries at terabyte scale, Skyrise's speed and cost are competitive with commercial cloud databases and prior serverless prototypes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central TPC-H competitiveness claim rests on only three scan-heavy queries (Q1, Q6, Q12), which the authors explicitly chose to avoid shuffle-dominated workloads; with no join-heavy TPC-H query evaluated, the abstract's claim of competitiveness for 'terabyte-scale queries of the analytical…","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the broad TPC-H claim is supported only by three scan-heavy queries. My stress-test confirms this is the single most consequential issue. The paper's own Section 4.2 explicitly limits the benchmark to queries 'not dominated by shuffling operators,' which directly contradicts the abstract's unqualified phrasing. A concrete test on shuffle-heavy TPC-H queries would either substantiate the broad claim or force a narrower framing. I considered other concerns—Lambada's different dataset, missing error bars, and the precise accounting of S3 request costs—but these affect only specific comparisons or secondary claims; the query-selection gap strikes at the central claim itself. The open-source artifact, the clearly described architecture, and the internally consistent evaluation of scan-heavy workloads all count in the paper's favor, so no correctness defect is alleged. The verdict of CONDITIONAL remains appropriate: conditional on broader TPC-H evaluation, which the authors have not provided.","tokens_in":13595,"tokens_out":3381,"duration_ms":36109,"concrete_test":"Run a set of join- and shuffle-heavy TPC-H queries (e.g., Q5, Q7, Q9, Q21, and possibly Q3/Q10) at the largest feasible scale factor, ideally SF1000, against the same S3-resident Parquet data used in Section 4, comparing Skyrise to Athena (and Snowflake external tables) on median latency and cost. If Skyrise's latency or cost is more than a modest multiple (e.g., 3-5x) of Athena's on any such query, or if any of these queries cannot be executed by Skyrise, the abstract's general TPC-H competitiveness claim should be formally narrowed to scan-heavy workloads.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Skyrise's headline claim ('both Skyrise's performance and cost are competitive to other cloud data systems for terabyte-scale queries of the analytical TPC-H benchmark') is far broader than what the evaluation demonstrates. Section 4.2 reports only TPC-H queries 1, 6, and 12, and the authors state these are 'well-suited for serverless execution as they are not dominated by shuffling operators.' This is an explicit, self-acknowledged limitation, yet the abstract and introduction still claim general TPC-H competitiveness. The three measured queries are essentially scans with local aggregation (Q12 includes a foreign-key join, but it is not shuffle-dominated); every TPC-H query with multi-way joins, large group-by/order-by, or substantial data movement is untested. Because Skyrise's architecture introduces a tiered shuffle mechanism and the physical optimizer chooses between repartition and broadcast joins, the join/shuffle path is a central and non-trivial part of the system. Without any evaluation of join-heavy queries, the observed scan performance cannot support the broad claim; the claim should be narrowed to scan-heavy analytical queries unless shuffle-heavy TPC-H queries are shown to be competitive. The artifact is open-source, and the paper is otherwise internally consistent, so this is an external-validity gap rather than a correctness defect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Skyrise, a query processor built entirely on serverless AWS infrastructure: the coordinator and workers run as Lambda functions, all persistent and intermediate data resides on serverless object storage, and messaging uses SQS. The system compiles SQL through a rule-based logical optimizer and a physical optimizer that introduces pipeline breakers and chooses shuffle strategies, then executes stage-wise with adaptive retriggering of straggling workers and an intermediate-result cache. The evaluation compares Skyrise against Lambada, Athena, and several Snowflake configurations on TPC-H at scale factor 1000, measuring runtime and cost for the three scan-heavy queries Q1, Q6, and Q12, and reports an elasticity experiment from SF 1 to SF 10000. The central claims are that Skyrise is the first fully serverless query processor with no server-based component and that its performance and cost are competitive to other cloud data systems for terabyte-scale TPC-H queries.","tokens_in":13849,"tokens_out":2921,"duration_ms":29762,"significance":"If the architecture and measurements hold, the paper makes a notable contribution: it is one of the first open-source systems to demonstrate that a complete SQL query processor, including coordinator, workers, and shuffle path, can run end-to-end on FaaS plus object storage while scaling from zero workers to thousands. The adaptive worker retriggering and the serverless intermediate-result cache are clearly described and are reasonable engineering contributions. The elasticity experiment, showing query latency within one order of magnitude across five orders of magnitude of data size, is an informative and rare measurement for this class of systems. The main weakness is that the empirical support for the headline TPC-H competitiveness claim is limited to three deliberately scan-heavy queries, and one of the comparison baselines (Lambada) is not measured on the same dataset.","major_comments":[{"comment":"The abstract and Section 1 claim that 'both Skyrise's performance and cost are competitive to other cloud data systems for terabyte-scale queries of the analytical TPC-H benchmark.' The evaluation in §4.2 supports this only for the three scan-heavy queries Q1, Q6, and Q12, which the authors themselves describe as 'well-suited for serverless execution as they are not dominated by shuffling operators.' No TPC-H query with multi-way joins, substantial grouping and sorting, or large data movement is tested, even though the tiered shuffle mechanism and the physical optimizer's repartition/broadcast join decisions are central components of the architecture. The observed scan performance therefore does not establish the broad TPC-H competitiveness claim; the claim should be narrowed to scan-heavy analytical workloads unless join- or shuffle-heavy TPC-H queries are evaluated.","section":"Abstract; §4.2"},{"comment":"The Lambada comparison uses runtimes and costs published in [22] rather than measurements on the same dataset. The paper itself notes in §4.2.1 that the Lambada dataset replaces string columns with integers and sorts lineitem on l_shipdate, which enables partition pruning; both differences directly reduce I/O and parsing costs for the selected queries. Consequently, the statement in §4.2.2 that Skyrise achieves latencies and costs 'comparable' to Lambada is not supported by a like-for-like comparison. Please re-run Lambada on the identical dataset and Parquet files, or clearly present the comparison as indicative and quantify the expected effect of the dataset differences.","section":"§4.2.1, Fig. 5"},{"comment":"All comparative latency and cost figures report only the median of five executions, with no measure of spread. Given that the paper's own elasticity experiment (Fig. 7) shows considerable variance at larger scale factors and that serverless functions are described as straggler-prone, the median alone is insufficient to determine whether observed differences between Skyrise and the comparison systems are meaningful. Please report per-execution values or at least min/max ranges for each configuration, or justify why the median is sufficient.","section":"§4.2, Figs. 5–6"}],"minor_comments":[{"comment":"The abbreviation 'OOM' is used for 'order of magnitude' without being defined; the phrase 'less than an order of magnitude (OOM) difference' is also redundant and should be simplified.","section":"§4.3"},{"comment":"The phrase 'competitive to other cloud data systems' should be 'competitive with other cloud data systems'.","section":"Abstract"},{"comment":"The sentence 'Athena being 7 × cheaper on average' is ambiguous because it does not state which configuration it is compared with; clarify the comparison and whether the ratio is per query or based on a different normalization.","section":"§4.2.2, Fig. 6"},{"comment":"The header 'T ail Latency' contains a typo ('T ail'), and the entry '>1k' for S3 Standard read tail latency should specify units and the percentile measured.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The core architectural claim is sound and the paper is internally consistent, so this is an external-validity gap rather than a correctness defect. The broad TPC-H competitiveness claim can likely be fixed within the manuscript's scope by either adding one or two shuffle/join-heavy TPC-H queries (e.g., Q3, Q5, or Q9 at SF 1000) or by explicitly narrowing the claim to scan-heavy analytical workloads. Re-running Lambada on the same dataset would substantially strengthen the comparison, though the authors may not have access to the original Lambada artifact; if not, the dataset differences should be discussed more prominently. The paper is well within scope for a database systems venue and the open-source artifact is a clear positive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Skyrise is a real contribution: to my knowledge it is the first fully serverless SQL query processor, with the coordinator and workers all running as Lambda functions, and the code is open source. The 'first' claim is credible. The system is a solid engineering effort—about 27K lines of C++, built on Hyrise's frontend, with a two-level invocation scheme, tiered shuffle to S3 Express, progress-tracking straggler retriggering, and an intermediate-result registry for caching. On three scan-heavy TPC-H queries at scale factor 1000, Skyrise is within about 30% of Lambada's runtime and cost, and it beats Athena and Snowflake on external tables. The elasticity test is also meaningful: aggregate latency for Q1 and Q6 grows by less than an order of magnitude while the data grows by five orders of magnitude, with no provisioning in advance.\n\nThe soft spot is the performance claim, not the architecture. Only Q1, Q6, and Q12 are measured, and the authors write in Section 4.2 that these are 'well-suited for serverless execution as they are not dominated by shuffling operators.' That is an explicit limitation, but the abstract and introduction still claim competitiveness for 'terabyte-scale queries of the analytical TPC-H benchmark.' That generalization is not backed by the experiments. The join/shuffle path is a core part of the system—tiered shuffle and repartition/broadcast join selection—and it is never tested end-to-end. The paper should either narrow the claim to scan-heavy analytical workloads or add shuffle-heavy TPC-H queries. The Lambada comparison is also weaker than it appears: the numbers come from the original Lambada paper and use a different dataset (integer strings, sorted lineitem). The authors acknowledge this, but they still present it as a head-to-head. No error bars or variance are reported, so the median of five runs tells us little about stability.\n\nNone of this undermines the central architectural claim. The system exists, it works, and the artifact is open. This paper is for researchers in serverless data processing and cloud database architecture. I'd bring it to a reading group and cite it as a reference point. It deserves a serious referee: the evaluation needs to match the claim, either by adding join-heavy queries or by scoping the claim down, and the Lambada comparison should be rerun on the same dataset. Send it to review.","headline":"Skyrise delivers the first fully serverless SQL engine, but the abstract's TPC-H competitiveness claim is broader than the three scan-heavy queries it actually evaluates.","tokens_in":14381,"tokens_out":3815,"would_cite":true,"duration_ms":35372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Skyrise shows a complete SQL engine can run on serverless functions alone and match cloud systems on TPC-H.","keywords":["serverless computing","function as a service","query processing","cloud data analytics","TPC-H","elasticity","object storage","AWS Lambda"],"falsifier":"Run the join- and shuffle-heavy TPC-H queries, for example Q5, Q8, or Q9, at scale factor 1,000 on Skyrise and compare latency and cost against Snowflake or Athena. If those queries are several times slower or more expensive than the scan-heavy set, the paper's general claim that Skyrise is competitive on the analytical TPC-H benchmark would be refuted.","tokens_in":13399,"feed_emoji":"⚡","tokens_out":6060,"duration_ms":51307,"temperature":0.7,"pith_summary":"This paper claims that a complete SQL query processor, including its coordinator and execution workers, can run entirely on serverless cloud infrastructure — function-as-a-service compute plus object storage — with no server-based component and no persistent cluster. The system, Skyrise, scales down to zero under no load and up to thousands of cloud functions for a single query. The authors argue that, for terabyte-scale scan-heavy TPC-H queries, Skyrise's runtime and monetary cost are competitive with commercial cloud data systems such as Athena and Snowflake. If true, this removes the standing assumption that an analytical database needs at least one always-on server, and replaces it with a pay-per-use model that matches query size automatically.","feed_headline":"Fully serverless SQL engine matches cloud systems on TPC-H","feed_subtitle":"A complete SQL engine on cloud functions scales to zero and stays cost-competitive on terabyte scans.","key_machinery":"The load-bearing mechanism is a per-query coordinator running as a serverless function. It compiles SQL through a rule-based logical optimizer and a physical optimizer, breaks the plan into data-parallel pipelines, and for each pipeline chooses a worker count from the input size and the network burst capacity of a function. Workers are invoked via a two-level fan-out, fetch only the relevant columnar slices from object storage in parallel, and push vectorized batches to the final operator, which writes a single output object. Because every worker is stateless, the coordinator can re-trigger stragglers mid-query without invalidating results — an idempotent worker recomputes and overwrites the same deterministic file. The second mechanism is an intermediate-result registry: the coordinator checks a hash of the logically optimized plan before scheduling a pipeline and skips it if a matching result already exists.","core_discovery":"Skyrise is the first end-to-end SQL query processor built entirely on serverless infrastructure: the query coordinator and all workers run as cloud functions, data is read from and written to object storage, and no server is involved at any point. To make this viable, Skyrise compiles SQL into staged pipelines, sizes the number of workers per pipeline from input size and network capacity, invokes workers through a two-level fan-out to speed up cluster startup, and aggressively re-triggers straggling workers or storage requests. Workers are stateless and idempotent, writing a single deterministic output file, so a re-triggered worker can overwrite a racing copy safely and aborted queries can resume from stage checkpoints. A cache of intermediate results, keyed by a hash of the logically optimized plan, lets recurring queries skip pipelines. In the evaluation, Skyrise's runtime and cost on TPC-H Q1, Q6, and Q12 at scale factor 1,000 are on par with the research baseline and competitive with the commercial systems, and query latency grows by less than an order of magnitude as input size grows from about 1 GB to about 10 TB.","pith_inferences":["The paper leaves join- and shuffle-heavy TPC-H queries untested; if those also stay competitive, the result would extend serverless processing from scan-dominated analytics to general decision-support workloads, but that remains an open bet.","The same architecture could plausibly be ported to other function-as-a-service providers, since the design relies only on functions, object storage, a queue, and a metadata store, though the paper only evaluates one cloud environment.","The result cache keyed on a logical-plan hash implies that semantically equivalent queries with different physical plans can share intermediate results, a form of cross-query reuse the paper does not explore.","Because the bottleneck at 10 TB is object-store request limits and function-invocation stragglers, the next limit to test is whether hotter storage tiers remove the remaining latency gap."],"forward_implications":["A full SQL engine can operate with zero idle infrastructure: no coordinator, no worker pool, and no shuffle service are kept running between queries.","For scan-heavy analytical workloads, serverless execution is cost-competitive with provisioned warehouses and with query-as-a-service offerings.","Elasticity becomes automatic: worker count and query cost track input size, with latency staying within one order of magnitude across five orders of magnitude of data.","Recurring or overlapping queries become cheaper because intermediate results are reused via the serverless storage cache.","Stragglers, a known weakness of function-as-a-service platforms, can be handled by re-triggering idempotent workers without compromising correctness."],"supporting_citations":[{"why":"Lambada is the FaaS-based baseline; Skyrise compares against its published numbers and adopts its two-level function invocation procedure.","marker":"[22]"},{"why":"Supplies the SQL frontend, expression and type systems reused by Skyrise.","marker":"[37]"},{"why":"Prior empirical study of serverless infrastructure that sets the per-pipeline worker count from input size and network burst capacity.","marker":"[42]"},{"why":"S3 Express One Zone provides the hot tier for shuffles and intermediate results.","marker":"[43]"},{"why":"TPC-H supplies the queries and the scale-factor-1,000 dataset used in the evaluation.","marker":"[51]"},{"why":"Athena is the commercial serverless baseline compared for latency and cost.","marker":"[24]"},{"why":"Snowflake is the commercial warehouse baseline, tested in internal, external, Iceberg, and serverless-task setups.","marker":"[50]"}],"fun_headline_variants":["First fully serverless SQL engine rivals cloud systems","Skyrise: elastic SQL on cloud functions, no servers","Serverless SQL scales to zero, matches TPC-H costs","Skyrise: end-to-end serverless query processing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on three scan-heavy TPC-H queries (Q1, Q6, Q12) that the authors chose because they are not dominated by shuffling; the paper's broad claim of competitive performance and cost for terabyte-scale TPC-H would fail if join- and shuffle-heavy queries perform much worse.","fun_headline_variants_meta":{"raw":{"variants":["First fully serverless SQL engine rivals cloud systems","Skyrise: elastic SQL on cloud functions, no servers","Serverless SQL scales to zero, matches TPC-H costs","Skyrise: end-to-end serverless query processing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1858,"prompt_tokens":954,"completion_tokens":904,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":839}},"tokens_in":570,"tokens_out":904,"duration_ms":8643,"temperature":1.0,"reasoning_tokens":839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:24:49.105308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the join- and shuffle-heavy TPC-H queries, for example Q5, Q8, or Q9, at scale factor 1,000 on Skyrise and compare latency and cost against Snowflake or Athena. If those queries are several times slower or more expensive than the scan-heavy set, the paper's general claim that Skyrise is competitive on the analytical TPC-H benchmark would be refuted.","supporting_citations":[{"cited_title":"In: ACM SIGMOD, pp","cited_arxiv_id":null,"evidence_quote":"Lambada is the FaaS-based baseline; Skyrise compares against its published numbers and adopts its two-level function invocation procedure."},{"cited_title":"In: EDBT, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the SQL frontend, expression and type systems reused by Skyrise."},{"cited_title":"An Empirical Evaluation of Serverless Cloud Infrastructure for Large-Scale Data Processing","cited_arxiv_id":"2501.07771","evidence_quote":"Prior empirical study of serverless infrastructure that sets the per-pipeline worker count from input size and network burst capacity."},{"cited_title":"https://aws .amazon.com/s3/ storage-classes/express-one-zone/","cited_arxiv_id":null,"evidence_quote":"S3 Express One Zone provides the hot tier for shuffles and intermediate results."},{"cited_title":"https://www.tpc.org/tpch/","cited_arxiv_id":null,"evidence_quote":"TPC-H supplies the queries and the scale-factor-1,000 dataset used in the evaluation."},{"cited_title":"https://aws .amazon.com/ athena/","cited_arxiv_id":null,"evidence_quote":"Athena is the commercial serverless baseline compared for latency and cost."},{"cited_title":"https://www.snowflake.com/de/data-cloud/ platform/","cited_arxiv_id":null,"evidence_quote":"Snowflake is the commercial warehouse baseline, tested in internal, external, Iceberg, and serverless-task setups."}],"review_version":1}