{"id":"65902c78-3fce-499d-84d2-e92f539175c6","arxiv_id":"2507.21810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"ChartMark is a structured, language-agnostic JSON grammar for chart annotations, with a toolkit that converts it to Vega-Lite.","lead":"This paper introduces ChartMark, a JSON-based grammar that describes chart annotations separately from the code that draws the chart. It also provides a Python toolkit that converts ChartMark descriptions into working Vega-Lite charts, so the same annotation can be reused across different visualization platforms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 lists 'compare' and 'explain' as task goals, but the formal task enumeration in Eq. 6 omits both; the grammar cannot encode the paper's own taxonomy, so the comprehensive/decoupling claim is unsupported.","rationale":"The reader's conditional verdict is reasonable, and I agree with the weakest assumption while sharpening it. The strongest claim, 'comprehensive and extensible framework that decouples semantics from implementation,' is load-bearing because the grammar itself is the paper's main contribution. The mismatch between Table 1 and Eq. 6 shows that exhaustiveness is not merely unproven; it is contradicted by the paper's own examples. This is a correctness risk, not a stylistic preference. The open-source toolkit and Vega-Lite demonstrations are genuine positive evidence, but they do not test taxonomy completeness or cross-platform semantic preservation. I recommend keeping the conditional verdict: the authors should align the formal grammar with their stated taxonomy and provide a corpus-based comprehensiveness check before the comprehensive/decoupling claim can be accepted as stated.","tokens_in":7881,"tokens_out":5285,"duration_ms":64113,"concrete_test":"Attempt to encode a 'compare' annotation, such as comparing Q1 versus Q2 sales, using only the grammar rules in Eq. 1-17. Because 'compare' is absent from Eq. 6, no faithful ChartMark document exists; any encoding must change the task to description/reference or add a type outside the grammar. Alternatively, run a small corpus audit: take annotations from ChartAccent [19] and VizBelle [9], map their task labels to Eq. 6, and count unmappable cases. If the count is nonzero or the internal Table 1 mismatch is confirmed, DC1 and the decoupling claim fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is DC1: ChartMark covers the annotation design space and decouples semantics from implementations. That requires every intended annotation task to be representable with the grammar's own vocabulary. The paper's Table 1 defines the Task dimension with examples 'highlight, compare, explain', but Eq. 5-6 in Section 4.2.2 fix type := reference | highlight | description | summary | trend | encoding, with no extension mechanism. 'compare' and 'explain' are therefore not expressible as task types using the formal grammar. A 'compare A and B' annotation cannot be written without either mislabeling it as 'description' or 'reference' or inventing a non-standard type outside Eq. 6, which violates the claimed comprehensiveness and the portability claim because the intended semantics are altered. The extensibility promise does not solve this: Section 5.2 describes adding custom nodes to the AST and implementing converter logic, which is a code-level extension, not a grammar-level slot. Thus the paper is internally inconsistent: its own taxonomy contains task types that the formal grammar cannot represent, and the escape hatch is implementation-dependent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ChartMark, a JSON-based structured grammar for chart annotations intended to separate annotation semantics from visualization implementations. It defines six annotation dimensions, a formal grammar with production rules for tasks, data, operations, targets, and markers, and a Python toolkit that converts ChartMark specifications into Vega-Lite visualizations. The central claims are comprehensiveness and extensibility (DC1), language independence (DC2), modularity (DC3), multi-level semantics (DC4), and atomic element design (DC5).","tokens_in":8146,"tokens_out":5641,"duration_ms":59146,"significance":"If its central claims are substantiated, ChartMark would provide a valuable abstraction layer for porting chart annotations across visualization platforms. The paper's decoupling idea is timely, and the open-source toolkit makes the approach concrete and reproducible. However, the comprehensiveness claim is currently asserted rather than demonstrated, and the formal grammar is incomplete in places, so the contribution is promising but not yet fully supported.","major_comments":[{"comment":"The formal grammar is internally inconsistent with the paper's own task taxonomy. Table 1 lists 'highlight, compare, explain' as example task goals, but Eq. (6) restricts task type to reference | highlight | description | summary | trend | encoding, omitting compare and explain. There is no subtype mechanism that recovers these goals: the subType field in Eq. (5) is only defined for particular types (e.g., summary with max/min) and Eq. (6) has no extension slot. Section 5.2's extensibility mechanism requires registering a new node in the AST and writing converter logic, which is an implementation-level extension rather than a grammar-level construct. Consequently, a 'compare A and B' annotation cannot be expressed without mislabeling it or modifying the grammar, directly undermining the DC1 comprehensiveness claim and the portability guarantee for such annotations.","section":"Section 4.2.2, Eq. (6); Table 1"},{"comment":"The claim that the enumerated dimensions and task types are comprehensive is not validated. The paper does not report a user study, a systematic survey, a corpus analysis, or a formal argument for exhaustiveness. The task type set in Eq. (6) is presented as an axiom, and the dimensions in Table 1 are attributed to prior taxonomies without showing that the union is complete. Since DC1 is the central contribution, the authors should provide evidence, such as a mapping of existing annotation taxonomies onto ChartMark or an evaluation over a corpus of annotated charts, or alternatively soften the claim to coverage of the surveyed design space.","section":"Section 3 (DC1), Section 4.2.2"},{"comment":"The formal grammar is incomplete as specified. Eq. (2) uses ellipses after x_name, y_name; Eq. (13) leaves target parameters unspecified after type; Eq. (15) and Eq. (17) define marker and text with ellipses and an undefined production (text := . . .). Because the grammar is the paper's main formal artifact, these omissions make it difficult to assess which annotations are actually representable and whether the portability claim holds. A complete BNF/EBNF or a precise statement that the grammar is an abbreviated sketch is needed.","section":"Section 4.2, Eqs. (2), (13), (15)-(17)"},{"comment":"The language-independence claim (DC2) is only partially supported. The toolkit converts ChartMark to Vega-Lite, but Figure 2 shows converters for ECharts and D3.js as well, and no implementation or evaluation for these is provided. The portability claim would be considerably strengthened by demonstrating at least one additional converter, or by explicitly framing the current toolkit as a Vega-Lite proof-of-concept.","section":"Section 5.1, Figure 2"}],"minor_comments":[{"comment":"'CharMark' appears to be a typo for 'ChartMark' in the sentence 'Please see the homepage of CharMark (chartmark.github.io).'","section":"Section 4"},{"comment":"The figure implies full converter support for Vega-Lite, ECharts, and D3.js, but only the Vega-Lite converter is described in Section 5; please clarify in the caption.","section":"Figure 2"},{"comment":"The subtype semantics would benefit from a complete list of valid subtype values for each task type; the summary/max-min example is helpful but not sufficient for reproducible grammar use.","section":"Section 4.2.2, Eq. (5)"},{"comment":"The capitalization of 'Annotation list' in the bullet points is inconsistent with the lowercase 'annotations' used in the JSON examples; please align terminology.","section":"Section 4.1"},{"comment":"Reference [17] appears to duplicate the Draco paper title with inconsistent capitalization ('draco' vs. 'Draco'); please verify the citation formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a visualization venue and the decoupling idea is worth pursuing. My main concern is that the formal claims (comprehensiveness, language independence) are not matched by the current evidence, and the internal inconsistency between Table 1 and Eq. (6) needs to be resolved before the paper can be accepted. The authors should also be encouraged to either provide a complete grammar or explicitly scope the claims to the implemented subset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChartMark is a plausible, useful step toward a portable annotation grammar, and the open-source converter to Vega-Lite is a nice artifact. But the paper's central claim of comprehensiveness (DC1) is damaged by an internal inconsistency: Table 1 lists 'compare' and 'explain' as example task goals, yet the formal task grammar in Eq. 6 only allows reference, highlight, description, summary, trend, and encoding. A 'compare A and B' annotation cannot be expressed without either mislabeling it or inventing a non-standard type outside the grammar. The extensibility story in Sec. 5.2 is about adding custom AST nodes and converter logic, not grammar-level vocabulary, so it doesn't fix the gap. This isn't a minor typo; it undercuts the decoupling claim that annotations can be moved across platforms without changing meaning.\n\nWhat the paper does well: the synthesis of prior taxonomies (Rahman et al., VizBelle, ChartAccent) into a single hierarchical JSON structure is genuinely new. The modular architecture, with annotations as pluggable units separate from the base chart, and the atomic element design are sensible. The examples in Figures 1–4 are clear and match the specs. The open-source Python package is a concrete, reproducible contribution.\n\nThe soft spots beyond the inconsistency: the comprehensiveness claim is not validated by a user study or systematic survey; extensibility is code-level, not grammar-level; the formal grammar uses ellipses (e.g., marker := (line?, text?, ...)) that leave the definition open-ended; only Vega-Lite conversion is demonstrated, though Figure 2 promises ECharts and D3; and there's no baseline comparison against existing annotation tools.\n\nNone of these are fatal to the overall idea, but together they mean the paper is a strong proposal, not a verified system. With a more modest claim or a fixed task type enumeration, it would be much stronger.\n\nI'd send this to a serious referee. It's a legitimate contribution with a real artifact, and the internal inconsistency is exactly what a good reviewer would catch. The authors should be asked to either expand the grammar to match their own taxonomy, add an explicit grammar-level extension mechanism, or soften DC1. I'd accept it conditionally.","headline":"A useful annotation grammar with a real internal inconsistency: the taxonomy in Table 1 includes task types the formal grammar cannot encode.","tokens_in":8618,"tokens_out":2905,"would_cite":true,"duration_ms":32165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChartMark claims that chart annotations reduce to five semantic dimensions plus a hierarchical grammar, so the same annotation can render on any platform.","keywords":["chart annotation","annotation grammar","visualization grammar","semantic annotation representation","cross-platform reuse","Vega-Lite","data storytelling","JSON specification"],"falsifier":"Take a sample of annotated charts from published sources and attempt to encode every annotation as (id, task, data, operations) using only the six task types reference, highlight, description, summary, trend, and encoding; any annotation that cannot be classified, or that changes meaning when round-tripped through the Vega-Lite converter, would falsify the exhaustiveness claim.","tokens_in":7739,"feed_emoji":"📊","tokens_out":7879,"duration_ms":83129,"temperature":0.7,"pith_summary":"ChartMark is a proposed structured grammar for chart annotations, covering labels, highlights, reference lines, and similar additions, that separates what an annotation means from how a particular charting library draws it. The paper argues that today's annotations are fragmented: visualization grammars and tools tie annotation effects to their own rendering logic, so an annotation created for one platform cannot be reused elsewhere. ChartMark's answer is a hierarchical JSON format organized around five dimensions (Data, Task, Marker, Target, Operation) plus Chart Context, with a small set of task types and atomic visual operations. To show this works, the authors provide a toolkit that converts ChartMark specifications into Vega-Lite charts. If the grammar is as comprehensive as claimed, annotations could be authored once and shared across visualization platforms without rewriting their meaning.","feed_headline":"One grammar carries chart annotations across visualization tools","feed_subtitle":"It separates an annotation's meaning from its rendering, so labels and highlights can move between chart libraries.","key_machinery":"The load-bearing object is the ChartMark grammar itself, expressed as a hierarchical JSON structure. Its outer layer separates the base chart from a pluggable list of annotations; its middle layer groups each annotation's Task, Data, and Operation lists; its inner layer pins each operation to a Target and the Marker parameters it sets. The grammar's power comes from two design choices: a taxonomy of annotation dimensions synthesized from prior design-space work, and atomic Marker and Operation components (such as color, opacity, and dashed lines) that can be reused and combined into composite annotations. Custom annotations are added by registering new nodes in an abstract syntax tree and writing transformation logic to a target language.","core_discovery":"The central claim is that the space of chart annotations is finite enough to be captured by a structured grammar. Every annotated chart is written as annotatedChart := (chart, annotations), where the base chart is separated from a list of annotation units, and each annotation is a tuple (id, task, data, operations). Task takes one of six types, reference, highlight, description, summary, trend, and encoding, with optional subtypes; Data specifies where the annotation's information comes from, external, derived, internal, or none; each Operation names a behavior, a Target (data items, coordinate, chart element, or annotation), and the Marker visual elements it adds or modifies. Because the representation is hierarchical and modular, high-level intent and low-level visual detail live in the same specification yet remain separable. The paper's demonstration converts ChartMark JSON into working Vega-Lite charts, including highlight, label, mean-line, reference-region, and grid-line annotations.","pith_inferences":["If the exhaustiveness claim holds, ChartMark could serve as a target schema for natural-language annotation generators, letting models output semantics instead of tool-specific code.","The same decoupling would make annotations shareable as data objects, enabling libraries of reusable annotation templates and cross-paper comparison of annotation practices.","A natural test is to take a diverse corpus of annotated charts from real publications and check whether every annotation maps cleanly onto the grammar; the paper itself does not report such a coverage study."],"forward_implications":["An annotation specified once in ChartMark can be ported to another charting system by swapping only the converter, not the annotation's meaning.","Because annotations are separate modules outside the base chart, adding or deleting one annotation leaves the chart structure unchanged.","The enumerated task types give downstream tools a stable vocabulary for searching, filtering, or automatically generating annotations by intent.","New annotation types can be assembled from existing atomic components plus a custom node, reducing the work of extending the grammar."],"supporting_citations":[{"why":"Supplies the annotation taxonomy and design space that ChartMark's five dimensions are synthesized from.","marker":"[18]"},{"why":"The Vega-Lite grammar is both the comparison baseline for coupled annotation and the target language of the ChartMark conversion toolkit.","marker":"[21]"},{"why":"ChartAccent is the main existing annotation tool whose rich techniques are bound to a single platform, motivating the portability claim.","marker":"[19]"},{"why":"Charticulator is cited as a tool whose annotation mechanisms are tied to its bespoke chart layouts.","marker":"[20]"},{"why":"Provides the idea of formalizing visualization design knowledge as constraints, grounding ChartMark's formal grammar approach.","marker":"[17]"},{"why":"Defines a design space of embellishments that feeds the Marker and Operation dimension definitions.","marker":"[9]"},{"why":"Establishes that graphical overlays aid chart reading, the functional motivation for the task component.","marker":"[14]"},{"why":"Represents domain-specific automatic annotation generation whose limited expression ChartMark aims to generalize.","marker":"[12]"}],"fun_headline_variants":["Grammar for chart annotations that travels across tools","ChartMark: one grammar to annotate any chart","Structured grammar unifies chart annotation specs","Chart annotations get a portable grammar","Separate annotation meaning from rendering with ChartMark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The grammar is only as comprehensive as its five dimensions and its six task types; if a real chart annotation does not fit one of those task categories, the central claim that ChartMark covers the annotation space starts to give way.","fun_headline_variants_meta":{"raw":{"variants":["Grammar for chart annotations that travels across tools","ChartMark: one grammar to annotate any chart","Structured grammar unifies chart annotation specs","Chart annotations get a portable grammar","Separate annotation meaning from rendering with ChartMark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1243,"prompt_tokens":784,"completion_tokens":459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":400,"tokens_out":459,"duration_ms":5012,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:19:33.328311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of annotated charts from published sources and attempt to encode every annotation as (id, task, data, operations) using only the six task types reference, highlight, description, summary, trend, and encoding; any annotation that cannot be classified, or that changes meaning when round-tripped through the Vega-Lite converter, would falsify the exhaustiveness claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the annotation taxonomy and design space that ChartMark's five dimensions are synthesized from."},{"cited_title":"Satyanarayan, D","cited_arxiv_id":null,"evidence_quote":"The Vega-Lite grammar is both the comparison baseline for coupled annotation and the target language of the ChartMark conversion toolkit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChartAccent is the main existing annotation tool whose rich techniques are bound to a single platform, motivating the portability claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Charticulator is cited as a tool whose annotation mechanisms are tied to its bespoke chart layouts."},{"cited_title":"Moritz, C","cited_arxiv_id":null,"evidence_quote":"Provides the idea of formalizing visualization design knowledge as constraints, grounding ChartMark's formal grammar approach."},{"cited_title":"VizBelle: A Design Space of Embellishments for Data Visualization","cited_arxiv_id":"2209.03642","evidence_quote":"Defines a design space of embellishments that feeds the Marker and Operation dimension definitions."},{"cited_title":"Kong and M","cited_arxiv_id":null,"evidence_quote":"Establishes that graphical overlays aid chart reading, the functional motivation for the task component."},{"cited_title":"Hullman, N","cited_arxiv_id":null,"evidence_quote":"Represents domain-specific automatic annotation generation whose limited expression ChartMark aims to generalize."}],"review_version":1}