{"id":"081d8bb6-ab94-46af-bce8-6a98459d5312","arxiv_id":"1909.00084","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A Microsoft team argues that enterprise ML will converge with database management, with training in the cloud, scoring inside DBMSs, and governance everywhere.","lead":"This paper predicts that enterprise machine learning will run on database systems, with models trained in the cloud and scored inside the database engine. It lays out a research agenda and early results for in-database inference, provenance, and governance of ML models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The in-DB scoring thesis rests on a uniform-IR premise the paper only hedges; end-to-end featurization coverage and fair performance comparisons are unvalidated.","rationale":"The reader's weakest assumption identifies exactly the point where the central claim is most exposed: the uniform IR/ONNX coverage and the implied performance parity. The manuscript itself flags the relevant evidence as preliminary ('full papers under preparation', 'early results'), and the quantitative claims in Figure 4 are not accompanied by the methodology needed to assess fairness of the comparison. A conditional verdict is therefore appropriate, and my read does not move it: the paper is a vision/roadmap with useful prototypes and honest hedging, not a falsifiable systems claim that collapses on its own equations. The governance part of the vision is independently plausible and supported by the enterprise conversations and provenance coverage numbers, so it should not be dismissed. The remaining risk is that the 'score in the DBMS' prediction is overgeneralized from a narrow set of ONNX-representable models; a coverage study on the authors' own corpus would settle whether that risk is real. I agree with the reader's conditional verdict rather than recommending rejection, because the paper's value is as a research agenda and the weak spot is an unproven premise, not a demonstrated falsehood.","tokens_in":11938,"tokens_out":4255,"duration_ms":41129,"concrete_test":"Using the authors' own corpus of >4M GitHub notebooks (referenced as [48]), classify each notebook's scoring pipeline into model type and featurization operations, then determine what fraction of complete pipelines can be exported to ONNX or to the paper's claimed IR without manual rewriting. If fewer than 70% of the most common pipeline patterns are representable, the uniform-representation premise fails for the intended 'default' role. A complementary check is to rerun the Figure 4 benchmark with the full pipeline (featurization plus model) on identical hardware against a standalone serving system that performs the same featurization in-process; if the 5.5x/24x speedups do not reproduce or are only for model scoring while featurization is excluded, the performance premise is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that models 'must be stored and scored in managed environments such as a DBMS' depends on two premises: (a) common end-to-end ML pipelines, including featurization, can be represented uniformly (e.g., ONNX) and compiled into the query runtime, and (b) doing so is competitive with dedicated serving systems. Premise (a) is supported only by 'It appears likely' in Section 1, with a citation to MLflow/ONNX that is not a coverage study; Figure 2's coverage plot lacks a defined metric and is not tied to ONNX exportability. Premise (b) is supported by Figure 4's 5.5x and 24x speedups, but the figure omits methodology, error bars, and a clear statement of whether the comparison includes the full featurization pipeline or only model scoring. The paper's own acknowledgements and Section 4.2 state that full papers on these systems are 'under preparation' and 'ongoing', so the evidence is not public or independently checkable. If common featurizers or proprietary models cannot be captured in the IR, the prediction narrows to a small subset of tabular models and the 'DBMS as default execution layer' claim is substantially weakened. This is not an internal inconsistency, but it is the least secure load-bearing point in the vision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This CIDR-style vision paper predicts that over the next ten years enterprise-grade machine learning will converge on three themes: model training and development in the cloud, model scoring inside managed data platforms such as DBMSs, and pervasive governance and provenance across the ML lifecycle. The paper introduces the Flock reference architecture, grounds the prediction in the authors' industrial experience, customer conversations, a GitHub analysis of over four million notebooks, and a survey of the competitive landscape. It also presents early technical results: an ONNX Runtime integration into SQL Server with cross-optimizations for in-DBMS inference, and SQL/Python provenance capture modules with initial evaluation on TPC-H, TPC-C, Kaggle, and internal Microsoft scripts.","tokens_in":12163,"tokens_out":4429,"duration_ms":40276,"significance":"If the prediction is correct, the database community's role in ML shifts from peripheral to central: DBMSs become the default execution and governance layer for enterprise model scoring. The paper is valuable as a position piece because it names concrete open problems (provenance modeling, model versioning, inference-as-query-processing) and offers a falsifiable research agenda. Its strengths include grounding in a large-scale notebook analysis, early prototypes with reported speedups, and an explicit call to action. The main limitation is that the load-bearing evidence for the strongest claim (in-DBMS scoring) is preliminary and not fully specified, so the paper is better read as a roadmap than as a validated result. It should be publishable after the evidence for that claim is either made reproducible or explicitly scoped as speculative.","major_comments":[{"comment":"The benchmark evidence for the central 'score in the DBMS' claim lacks methodology and internally inconsistent numbers. The figure reports speedups of 17x and 24x, while the text says 'up to 5.5x' for SQL Server/ORT integration and 'up to 24x' for combined optimizations in §4.1, and '5x to 24x' in §2. The figure does not specify the models, datasets, hardware, run counts, error bars, or whether the comparison includes the full featurization pipeline or only model scoring. Since this is the main empirical support for the paper's second core prediction, the authors should provide a detailed experimental setup, reference a public artifact, or clearly label the figure as an illustrative early result with the specific conditions under which it was obtained.","section":"§4.1, Figure 4"},{"comment":"The premise that the most widely used model families can be uniformly represented and compiled into efficient in-database code is asserted with hedging language ('It appears likely') and cited to MLflow and ONNX, which are platforms/format specifications rather than coverage studies. Figure 2 measures notebook package coverage, not model or featurization representability. This assumption is load-bearing for the 'score in the DBMS' prediction: if common featurization pipelines or proprietary model types cannot be captured in the IR, the prediction narrows to a small subset of tabular models. The paper should either provide a coverage analysis over the model/featurizer space or explicitly state the scope of the claim.","section":"§1 and §4.1"},{"comment":"The provenance results are presented as validation, but they are preliminary and not independently checkable. The SQL provenance table reports latency and graph size on TPC-H and TPC-C without a baseline, accuracy metric, or end-to-end correctness check; the Python provenance table covers only 49 Kaggle and 37 Microsoft scripts. The text and acknowledgements state that full papers are 'under preparation' and 'ongoing', meaning the described systems are not publicly available. This is acceptable for a vision paper, but the claims should be framed as early feasibility checks rather than demonstrated solutions, and the authors should make clear what fraction of the reported provenance capture is automated versus manually inspected.","section":"§4.2"}],"minor_comments":[{"comment":"The figure contains typographical issues: 'deploymen t' and 'policiesLive Data' appear without proper spacing, and the label 'Model deploymen t' is split across a line break. Please fix these in the final version.","section":"§2, Figure 1"},{"comment":"The caption 'Total:3x more packagesTop10: 5% morecoverage' is malformed and lacks spacing. The x-axis labels are missing; while the text defines 'coverage' as the fraction of notebooks fully supported, the figure should be self-contained with axis labels and a short definition of the metric in the caption.","section":"§3, Figure 2"},{"comment":"The comparison table is based on 'a few weeks of analysis of marketing material, code skimming, and light experimentation' as stated in footnote 5. Since the systems and their features are dated, the figure should include an explicit 'as of' date and a caveat in the main text that the comparison is subjective and may already be outdated.","section":"§3, Figure 3"},{"comment":"The sentence introducing the provenance tables contains a repetition: 'The above table table shows...' Please remove the duplicated word.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a position paper from a large industrial lab, and its value is in articulating a research agenda rather than in delivering fully validated results. The main risk is that the 'score in the DBMS' claim is supported by unpublished or under-specified benchmarks; the authors should be encouraged to make the evidence reproducible or to hedge the claim more explicitly. The reliance on companion papers ([40], [48], [30]) is reasonable for a vision paper, but the editor may want to confirm that these companion papers are publicly available before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on the Cloudy with DBMS paper. It's a solid, articulate industrial vision paper. The main claim—enterprise ML will be governed and scored inside DBMSs—is presented as a ten-year prediction, supported by firsthand experience, customer conversations, a large-scale GitHub notebook analysis, and early benchmarks. The paper is honest that the details live in companion papers.\n\nWhat's genuinely new: the Flock reference architecture, which treats models as derived data and lays out a data-management-centric lifecycle; the in-database cross-optimizer that does predicate-based model pruning, projection pushdown, model inlining, etc.; and the SQL and Python provenance modules that track lineage across heterogeneous systems. The early results (5.5x to 24x speedups) are encouraging, and the provenance coverage numbers on TPC-H/TPC-C and Kaggle scripts are a reasonable first step.\n\nThe soft spots are real but not fatal for a vision paper. The central premise—that widely used model families and featurization pipelines can be uniformly represented (e.g., ONNX) and compiled into the query runtime—is backed only by 'it appears likely' in the intro and a few benchmark teasers. Figure 2's coverage plot is about package popularity, not ONNX exportability. Figure 4 omits methodology, error bars, hardware, and whether the comparison includes end-to-end featurization or just model scoring. The paper even acknowledges the full systems papers are under preparation, so the evidence isn't independently checkable yet. If common featurizers or proprietary models don't fit the IR, the 'DBMS as default scoring layer' claim narrows considerably.\n\nI think these are legitimate concerns, but they don't sink the paper. It's a provocation and a research agenda, and it does a good job of laying out the open problems for the database community. The self-citation pattern is reasonable here because the cited works are the actual companions where the details will appear.\n\nWho should read it: anyone working at the intersection of database systems and ML, especially those interested in model serving, provenance, and governance. It would be a good reading-group starting point. I'd recommend a serious referee engage with it—it deserves peer review, probably at a venue that welcomes vision papers, with the expectation that the companion papers deliver the goods.","headline":"A credible industrial vision paper for in-DB ML scoring and governance, with early proof points—but the uniform-IR premise that holds the thesis together is still a hope, not a demonstrated result.","tokens_in":12794,"tokens_out":2368,"would_cite":true,"duration_ms":20459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper predicts that within ten years, enterprise machine learning will be trained in the cloud, scored inside database engines, and governed through continuous provenance tracking, making the database engine the default platform for…","keywords":["enterprise machine learning","in-database inference","model scoring","model governance","provenance","SQL/ML optimization","cloud machine learning","Flock architecture"],"falsifier":"Run a broad benchmark suite spanning deep networks, tree ensembles, text featurizers, and proprietary models, comparing compiled in-database inference against standalone serving on the same hardware and latency target; if any common model family either cannot be represented in the intermediate form or loses to standalone serving on realistic workloads, the 'score in the DBMS' prediction fails for that family.","tokens_in":11739,"feed_emoji":"☁️","tokens_out":8000,"duration_ms":66358,"temperature":0.7,"pith_summary":"Enterprise-grade machine learning, the paper argues, will be reshaped by two forces at once: ML's spread beyond a few high-value web applications, and tightening regulatory pressure on data handling. The paper's prediction is that the next decade brings 'cloudy with a high chance of DBMS, and governance throughout': model training and development move to private or public clouds, trained models are stored and scored inside managed database environments rather than standalone serving containers, and provenance is tracked end to end from training data to the decisions made from predictions. The claim matters because if it holds, the database engine becomes the default execution and governance layer for enterprise ML inference, not just a repository. Early experimental results support the feasibility: integrating an ML runtime into a database engine and applying cross-optimizations such as predicate-based model pruning and model inlining yields up to 5.5x over standalone serving and up to 24x with combined optimizations.","feed_headline":"Enterprise ML's future: train in cloud, score in the database","feed_subtitle":"Early results show in-database scoring up to 24x faster than standalone serving.","key_machinery":"The load-bearing mechanism is Flock, a reference architecture for the canonical data-science lifecycle, together with an intermediate representation for inference pipelines that can be compiled into optimized database execution plans. The key technical move is to treat scoring as a foundational extension of relational algebra: given a uniform representation of a model, the system compiles the full featurization-plus-model pipeline into relational operators and lets the SQL optimizer apply predicate-based model pruning, model-projection pushdown, model clustering, model inlining, and physical operator selection across the SQL/ML boundary. For governance, the machinery is a provenance catalog that combines coarse-grained SQL provenance with static analysis of Python scripts, connecting database columns to trained models and enabling applications like model linting, compliance checks, and impact analysis.","core_discovery":"The central claim is a design point: an ML model should be thought of as software derived from data, and therefore as both a program to be engineered and a dataset to be governed. From that lens the paper derives three predictions: (1) training and model development will happen in the cloud, where centralized data, elastic resources, and latest hardware are available; (2) models must be stored, versioned, and scored in managed environments such as a DBMS, with inference expressed as an extension of relational query processing so data never has to be exfiltrated; and (3) provenance must be collected across all phases, connecting the data that trained a model to the decisions the model later influences. The paper supports these claims with a reference architecture (Flock), an analysis of over four million public notebooks, conversations with enterprises, and early benchmarks of in-database inference.","pith_inferences":["Beyond the paper: if the uniform-intermediate-representation premise holds beyond the benchmarked models, the same compiled inference plans could be pushed to edge databases, making the database the governance boundary for on-device scoring.","Beyond the paper: the 24x speedup likely reflects workloads where predicate pruning applies; on dense scoring over all rows the advantage would narrow to parallelization and column pushdown, so a reproducible benchmark across model families would sharpen the decade-long prediction.","Beyond the paper: the provenance design implies row- and column-granular data versioning rather than file-granular versioning, since files are not the atomic unit of training data—a storage research agenda the paper only sketches.","Beyond the paper: the notebook analysis suggests the Python data-science ecosystem is consolidating around a core of packages; if that consolidation continues, the compiler-based IR approach becomes progressively more tractable."],"forward_implications":["In-database model scoring becomes the default for batch and many latency-sensitive enterprise predictions, with transactional updates across multiple deployed models.","Database query optimizers gain a new class of ML-aware rewrites, so the same inference pipeline runs faster as data volumes grow rather than requiring separate serving infrastructure.","Model management inherits enterprise data features: access control, versioning, auditing, and high availability apply to models as first-class DBMS data types.","Automated provenance turns compliance checks into queries: detecting label leakage, PII usage, or the impact of a dropped column becomes a programmatic analysis of the captured lineage.","The division between application-level business policies and raw model predictions becomes explicit and auditable, with policies applied transactionally around model outputs."],"supporting_citations":[{"why":"Presents the in-database cross-optimizer and the experimental setup behind the 5.5x and 24x speedup claims.","marker":"[40]"},{"why":"The inference runtime integrated into the database engine to enable in-process model scoring.","marker":"[15]"},{"why":"The uniform model-exchange format the paper relies on for representing diverse model families.","marker":"[47]"},{"why":"Shows how model descriptions can be compiled into optimized code for different execution environments and hardware, supporting the compilation argument.","marker":"[25]"},{"why":"The more than four million notebook analysis behind the coverage and adoption estimates for ML pipelines.","marker":"[48]"},{"why":"Acts as the central catalog that stores versioned provenance and bridges the SQL and Python provenance modules.","marker":"[22]"},{"why":"Existing experiment-tracking platform cited as an industry effort that must be expanded to full lineage capture.","marker":"[43]"}],"fun_headline_variants":["Cloud trains, database scores, provenance tracks","Train in cloud, score in DB, govern with provenance","10-year ML forecast: cloud training, DB scoring","ML's future: training in cloud, inference in database","In-database scoring: the DBMS as ML runtime"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the most widely used model families and their featurization pipelines can be captured in a uniform intermediate representation and compiled into efficient in-database code without data exfiltration, so in-database scoring matches or beats dedicated serving systems; the paper offers early benchmark evidence but not broad coverage.","fun_headline_variants_meta":{"raw":{"variants":["Cloud trains, database scores, provenance tracks","Train in cloud, score in DB, govern with provenance","10-year ML forecast: cloud training, DB scoring","ML's future: training in cloud, inference in database","In-database scoring: the DBMS as ML runtime"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001111,"raw_usage":{"total_tokens":4596,"prompt_tokens":881,"completion_tokens":3715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":3638}},"tokens_in":497,"tokens_out":3715,"duration_ms":23761,"temperature":1.0,"reasoning_tokens":3638,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:01:38.312463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a broad benchmark suite spanning deep networks, tree ensembles, text featurizers, and proprietary models, comparing compiled in-database inference against standalone serving on the same hardware and latency target; if any common model family either cannot be represented in the intermediate form or loses to standalone serving on realistic workloads, the 'score in the DBMS' prediction fails for that family.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the in-database cross-optimizer and the experimental setup behind the 5.5x and 24x speedup claims."},{"cited_title":"https://engineering.linkedin.com/blog/2019/01/ scaling-machine-learning-productivity-at-linkedin, 2018","cited_arxiv_id":null,"evidence_quote":"The inference runtime integrated into the database engine to enable in-process model scoring."},{"cited_title":"https://www.kaggle.com/datasets, 2019","cited_arxiv_id":null,"evidence_quote":"The uniform model-exchange format the paper relies on for representing diverse model families."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how model descriptions can be compiled into optimized code for different execution environments and hardware, supporting the compilation argument."},{"cited_title":"Karanasos, M","cited_arxiv_id":null,"evidence_quote":"The more than four million notebook analysis behind the coverage and adoption estimates for ML pipelines."},{"cited_title":"https://github.com/Microsoft/onnxruntime, 2019","cited_arxiv_id":null,"evidence_quote":"Acts as the central catalog that stores versioned provenance and bridges the SQL and Python provenance modules."},{"cited_title":"https://mlflow.org, 2019","cited_arxiv_id":null,"evidence_quote":"Existing experiment-tracking platform cited as an industry effort that must be expanded to full lineage capture."}],"review_version":1}