{"id":"03cfc7b9-c15b-46bb-9c81-d0d46d8b4ee2","arxiv_id":"2411.14788","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Jovis visualizes PostgreSQL's standard and genetic query optimizers and lets users guide optimization through hints, parameters, and prior results.","lead":"Jovis is a new tool that shows the inside of PostgreSQL's query optimizer, letting users watch how the database builds execution plans. It turns hidden optimizer choices into interactive diagrams and adds controls for hints, parameters, and reusable plans.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of faithfully visualizing the optimizer's explored search space depends on the logging patch capturing paths before pruning; the paper never shows this, so the visualizations may omit rejected candidates.","rationale":"The reader's weakest assumption correctly identifies the logging patch as the load-bearing foundation: if the patch is incomplete or perturbing, the visualizations do not faithfully represent PostgreSQL's optimizer. My stress-test sharpens this into a concrete, checkable gap: the paper does not establish whether pruned paths and rejected genes are logged and displayed. This matters because PostgreSQL's add_path() deliberately discards dominated candidates, and GEQO's population replacement discards low-fitness genes; a tool that only shows survivors is showing the optimizer's decided results, not its exploration. The paper also provides no independent validation (e.g., comparison with an instrumented counter) that the log matches the actual optimizer activity. This concern reinforces the reader's CONDITIONAL verdict: the architecture and demonstration are plausible, but the central fidelity claim needs verification before the tool can be recommended as a faithful window into optimizer internals. I do not see a basis to change the verdict; the concern is addressable and the reader already conditioned acceptance on such validation. The availability of source code makes the proposed counter-probe test straightforward.","tokens_in":7781,"tokens_out":7806,"duration_ms":84226,"concrete_test":"Install the public Jovis patch on a clean PostgreSQL 16.2 and run a query known to generate a slightly more expensive path that is pruned (e.g., force a nested-loop path that loses to a hash join by a small margin). Independently instrument PostgreSQL's add_path() (or add_partial_path()) to emit one log record for every path object at the moment it is considered, before pruning. Compare the number and identity of paths in this independent trace against the nodes in Jovis's DAG for the same query. Additionally, for GEQO, instrument geqo_eval() to log every gene cost evaluation and compare the count and coordinates against the heatmap cells in the GEQO view. If any independently logged path or gene is absent from Jovis's visualization, the central fidelity claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that Jovis reveals the optimizer's internal decision process, including 'all paths considered during optimization' (Section 3.1.1). This requires the patch described in Section 3.2 to log every candidate path that PostgreSQL actually evaluates. But PostgreSQL's dynamic programming uses add_path() to discard dominated paths immediately; the optimizer does not retain them in the final path list. The paper does not state whether the patch records paths before or after add_path() pruning. If the log captures only the surviving path lists, then the DAG visualizes the optimizer's final candidates, not the full explored search space. Similarly, for GEQO, crossover operations generate offspring that are immediately evaluated and may be discarded; the paper says the patch 'records each step of join sequence generation and crossover operations', but the parser extracts 'generation information, join sequences, and associated costs' without confirming that rejected genes appear in the heatmap. No validation in the paper checks that the logged entries match an independent trace of the optimizer's actual activity. If a path or gene is generated and then rejected, and Jovis omits it, the tool misrepresents the very decision space it claims to expose.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Jovis, an interactive visualization tool for PostgreSQL's query optimizer. The system consists of a patched PostgreSQL 16.2 that logs optimizer-internal state, a backend that parses those logs into JSON, and a React/D3 frontend that renders the results. For the standard dynamic-programming optimizer, Jovis shows a DAG of base-relation access paths, join orders, physical operators, and cost estimates; for GEQO, it shows generation heatmaps, cost trend charts, crossover edges, and per-gene join-order trees. The tool also supports user-guided optimization through pg_hint_plan hints and a 'guided GEQO' feature for seeding the population and tuning geqo_seed and selection_bias. The paper's central claim is that Jovis exposes the optimizer's internal decision process, including paths that were considered and rejected, and that this transparency helps users understand and improve query performance; the reported quantitative evidence is a single JOB Query 10c example showing a 7% estimated-cost reduction.","tokens_in":1464,"tokens_out":1581,"duration_ms":58155,"significance":"If the central claim holds, Jovis addresses a real gap: existing tools such as PEV2 and MySQL Workbench visualize only the final EXPLAIN plan, while Jovis targets the search process itself, and it appears to be the first dedicated GEQO visualization for PostgreSQL. The engineering contributions are concrete and reproducible in spirit: a logging patch, structured JSON extraction, DAG and heatmap visualizations, cost-formula drill-down, and integration of pg_hint_plan and guided GEQO. The source code is publicly linked. The main weakness is that the paper does not yet establish faithfulness of the logged search space or generalizability of the reported improvement: there is no validation of the log against an independent trace, no systematic experiment, and no user study. These are fixable within the scope of a demonstration paper, but they are load-bearing for the strongest claims.","major_comments":[{"comment":"The claim that Jovis lets users 'visually grasp all paths considered during optimization' is not supported by the described logging mechanism. PostgreSQL's dynamic-programming optimizer routinely discards dominated paths inside add_path(), and the paper does not state whether the patch records paths at generation time, at the time they are inserted into the path list, or only after pruning. Similarly, for GEQO the paper says the patch 'records each step of join sequence generation and crossover operations,' but the parser extracts only 'generation information, join sequences, and associated costs,' without confirming that rejected genes and discarded offspring are retained. If the log captures only surviving candidates, the DAG and heatmap misrepresent the search space. Please specify the exact logging points and provide a validation that the logged entries match an independent trace of optimizer activity for at least a few representative queries.","section":"§3.1.1, §3.2"},{"comment":"The patch is described as 'non-intrusive,' but no evidence is given that logging does not perturb the optimizer. If the logging code allocates memory, performs I/O, or consumes random-number state during GEQO, it could change plan choices or estimated costs. The authors should either state explicitly that the patch has no side effects on optimizer state, or provide a before/after comparison of EXPLAIN output and GEQO results with logging enabled and disabled on the same queries.","section":"§3, §3.2"},{"comment":"The only quantitative evidence of practical benefit is a single JOB Query 10c example reporting a 7% cost reduction. This is a hand-picked instance, and the paper does not report how often guided GEQO improves plans, by how much, or whether the improvement is stable across different random seeds. Since the abstract and introduction claim that Jovis helps users 'improve query performance,' a small systematic experiment over several JOB or TPC-H queries with multiple seeds, plus a comparison against at least one existing tool, would make the claim proportionate to the evidence.","section":"§3.3.2, Example 3"},{"comment":"The paper frames Jovis as both an educational and a practical resource, but it contains no evaluation of whether users actually understand the optimization process better or can diagnose suboptimal plans more effectively. Although a demonstration paper may not require a full user study, the stated educational value is currently an assertion rather than a demonstrated result; at minimum, the authors should calibrate the abstract and introduction to the evidence presented.","section":"§1, Abstract"}],"minor_comments":[{"comment":"The phrase 'Application Protocol Interface service' should read 'Application Programming Interface (API) service.'","section":"§3.2"},{"comment":"The phrase 'the fiver lowest-cost genes' is a typo; it should be 'five lowest-cost genes.'","section":"§3.3.2, Example 3"},{"comment":"The phrase 'for each row inlineitem' should be 'for each row in lineitem.'","section":"§3.3.1, Example 2"},{"comment":"The artifact URL is a GitHub organization page rather than a direct repository URL; the authors should provide a working, specific repository link for the reviewed version.","section":"Abstract"},{"comment":"The prose refers to views 'C' and 'D' but also to subcomponents C1, C2, D1, etc.; labeling the figure panels consistently would reduce ambiguity for readers.","section":"§3.1, Figure 2"},{"comment":"The related-work discussion would benefit from a brief comparison of Jovis with QO-Insight and MOCHA in terms of what each visualizes (execution traces vs. search space), since the novelty claim depends on that distinction.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"For a demonstration paper, the bar for evaluation is lower than for a full research paper, and I would not reject this work solely for the absence of a user study. The main issue is that the core transparency claim depends on the logging patch capturing rejected candidates, and the manuscript does not currently establish that. A targeted validation section or an explicit statement of the logging points, plus a slightly broader GEQO experiment, would probably be sufficient. I recommend major_revision rather than reject or minor_revision because the faithfulness concern directly affects the headline contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jovis is a real, working visualization tool with a genuinely new angle: it displays the internals of PostgreSQL's optimizer, including the genetic search in GEQO, rather than just the final plan. The GEQO heatmap, crossover arcs, and guided pool initialization are firsts, as far as I can tell. The source is available, the architecture is straightforward, and the cost drill-down with formulas is a nice touch for teaching.\n\nThe main thing to check is whether the visualization actually shows what the paper claims. Section 3.1.1 says users can see 'all paths considered during optimization,' and the backend (Section 3.2) logs 'path lists, costs, and the cheapest paths.' PostgreSQL's dynamic programming calls add_path() and discards dominated paths immediately; those discarded candidates are not retained in the final path list. The paper never states whether the patch records paths before or after pruning. If it logs only the surviving lists, the DAG shows the final candidates, not the considered alternatives. Same concern for GEQO: the heatmap may only contain genes that survived a generation, not every offspring generated and rejected. That is a load-bearing assumption for the paper's main value proposition, and it is not validated. A quick way to address this is to compare the log against a known exhaustive enumeration for small queries.\n\nThe evaluation is thin, even for a demo. There is no user study, no comparison with PEV2 or QO-Insight, and the only performance claim is a 7% cost reduction on one JOB query with manually chosen parameters. That example is fine as an illustration but not evidence of general effectiveness.\n\nThese are fixable issues. The tool is genuinely new and the authors have built something that database instructors and query-tuning practitioners will want to poke at. With a validation section on logging fidelity and a broader set of examples, it would be a solid demonstration paper.\n\nI'd send it to a serious referee on the demo track, with the logging question as the first thing to resolve. Worth putting on a reading group list.","headline":"Jovis is a genuinely useful demo with a new GEQO visualization angle, but its core transparency claim depends on unvalidated assumptions about what the logging patch captures.","tokens_in":8513,"tokens_out":2270,"would_cite":false,"duration_ms":22046,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jovis opens up PostgreSQL's query optimizer, exposing every plan it considered and why it chose the winner.","keywords":["query optimization","visualization","PostgreSQL","GEQO","genetic algorithm","execution plans","cost model","interactive tool"],"falsifier":"Instrument PostgreSQL's planner to emit every candidate path at the point of insertion, run a small multi-join query and a query with more tables than the GEQO threshold through Jovis, and compare the complete lists; if the DAG omits any emitted path, any crossover differs from the recorded parent pair, or any displayed cost disagrees with the instrumented value, the visualization is not faithful to the optimizer's actual search.","tokens_in":7558,"feed_emoji":"📊","tokens_out":9544,"duration_ms":90035,"temperature":0.7,"pith_summary":"Jovis is an interactive visualization tool that sets out to make the internal decision-making of PostgreSQL's query optimizer visible to users. Existing tools such as EXPLAIN show only the final execution plan, while Jovis logs and renders every access path, join order, physical operator, and cost estimate that the optimizer explored, with separate views for the dynamic-programming optimizer and for GEQO, the genetic query optimizer that handles very large joins. The paper further claims that Jovis is the first tool with dedicated visualization for GEQO. If that claim stands, database professionals can diagnose suboptimal plans by seeing which alternatives were rejected and why, and learners can watch the optimization process unfold instead of treating the optimizer as a black box.","feed_headline":"See every plan Postgres' optimizer considered","feed_subtitle":"Jovis turns hidden optimizer logs into interactive views, then lets users steer the search.","key_machinery":"The load-bearing mechanism is the optimizer-log pipeline: a lightweight patch to PostgreSQL writes the planner's internal state to a log, a backend parser converts the unstructured text into structured JSON, and the browser interface renders the JSON in two visual encodings. The standard-optimizer view is a directed acyclic graph whose circular nodes are relations or join sequences and whose rectangular nodes are physical operators, with edges showing which operator was applied to which inputs and an animated play mode tracing how the cheapest path is selected. The GEQO view is a grid heatmap of genes across generations colored by estimated cost, with a line chart of best, worst, median, and average costs and crossover diagrams that show how parent join sequences produce offspring. The same logged data drives interactive features: cost formulas on operator nodes, range sliders over generations, and panels for hints and guided pool initialization.","core_discovery":"The paper's central claim is that the PostgreSQL optimizer's search can be made fully inspectable without changing how it plans. By adding logging to PostgreSQL 16.2, Jovis captures the path list and costs generated during dynamic-programming enumeration and the generations, genes, and crossovers generated by GEQO; the backend parses these logs into JSON and the frontend renders them as an animated directed acyclic graph for the standard optimizer and as a grid heatmap with cost charts for GEQO. Jovis also supports user-guided optimization, letting users supply hints, seed the GEQO pool with high-performing genes from prior runs, and adjust genetic-algorithm parameters. The reported demonstration with JOB Query 10c shows a 7% cost reduction from guided GEQO, which the paper presents as evidence that the visualization improves both understanding and outcome.","pith_inferences":["A natural extension the paper leaves implicit is using the captured logs to diff optimizer behavior across PostgreSQL versions or configuration changes, turning Jovis into a regression-analysis tool for planner updates.","The guided-GEQO interaction suggests a closed loop in which a system mines past runs to propose seeding genes and parameter values automatically, rather than requiring a human to pick them.","If the logging patch is adopted upstream, the same JSON pipeline could make Jovis-style transparency a standard PostgreSQL diagnostic, potentially changing how optimizer defects are reported and reproduced.","Although the paper describes only PostgreSQL, its DAG representation of paths and costs should transfer to other cost-based optimizers that enumerate join orders with dynamic programming, since the underlying objects are conceptually the same."],"forward_implications":["PostgreSQL users can diagnose suboptimal plans by inspecting which join orders and physical operators were rejected and comparing their costs directly, rather than reasoning from the final EXPLAIN output alone.","Educators can use Jovis to show students how bottom-up dynamic-programming planning and the genetic algorithm's selection, crossover, and replacement actually work on real queries.","Practitioners can apply hints and configuration changes in the interface and immediately see how the explored search space and the chosen plan shift, making optimizer tuning an experimental process.","For repeatedly run analytical queries, guided GEQO can reuse the best join sequences from a previous run as pool seeds, and the paper's JOB Query 10c example reports a 7% estimated-cost reduction from doing so."],"supporting_citations":[{"why":"previous steered-optimizer trace visualization that Jovis positions against and extends.","marker":"[1]"},{"why":"supplies the data-driven visualization library used to build the interactive interface.","marker":"[2]"},{"why":"source of the heatmap-based genetic-algorithm visualization that Jovis adapts for GEQO.","marker":"[3]"},{"why":"EXPLAIN-output visualizer representing the baseline of final-plan-only tools.","marker":"[4]"},{"why":"PostgreSQL hint extension that Jovis patches with logging to support user-guided optimization.","marker":"[5]"},{"why":"documents GEQO's algorithm, the behavior Jovis's dedicated visualization must capture.","marker":"[6]"},{"why":"earlier query optimizer visualizer in the related-work landscape.","marker":"[8]"},{"why":"Join Order Benchmark, source of the preset workload and the guided-GEQO demonstration query.","marker":"[10]"},{"why":"establishes the dynamic-programming enumeration approach that the standard optimizer visualization depicts.","marker":"[12]"}],"fun_headline_variants":["Jovis: see inside Postgres' optimizer","Steer Postgres' planner with interactive logs","Visualize every plan Postgres' optimizer weighs","Jovis turns optimizer logs into a game board","Interactive tool lifts the lid on Postgres' planner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole tool rests on the assumption that the logging patch captures the optimizer's real internal decisions without changing them, so the DAG and heatmap faithfully show what the planner actually did rather than a plausible reconstruction.","fun_headline_variants_meta":{"raw":{"variants":["Jovis: see inside Postgres' optimizer","Steer Postgres' planner with interactive logs","Visualize every plan Postgres' optimizer weighs","Jovis turns optimizer logs into a game board","Interactive tool lifts the lid on Postgres' planner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1574,"prompt_tokens":882,"completion_tokens":692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":620}},"tokens_in":498,"tokens_out":692,"duration_ms":6942,"temperature":1.0,"reasoning_tokens":620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:52:54.730228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument PostgreSQL's planner to emit every candidate path at the point of insertion, run a small multi-join query and a query with more tables than the GEQO threshold through Jovis, and compare the complete lists; if the DAG omits any emitted path, any crossover differs from the recorded parent pair, or any displayed cost disagrees with the instrumented value, the visualization is not faithful to the optimizer's actual search.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"previous steered-optimizer trace visualization that Jovis positions against and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the data-driven visualization library used to build the interactive interface."},{"cited_title":"2014.Visualization for Genetic Algorithms","cited_arxiv_id":null,"evidence_quote":"source of the heatmap-based genetic-algorithm visualization that Jovis adapts for GEQO."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EXPLAIN-output visualizer representing the baseline of final-plan-only tools."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PostgreSQL hint extension that Jovis patches with logging to support user-guided optimization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents GEQO's algorithm, the behavior Jovis's dedicated visualization must capture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"earlier query optimizer visualizer in the related-work landscape."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"establishes the dynamic-programming enumeration approach that the standard optimizer visualization depicts."}],"review_version":1}