{"id":"a147f270-5f79-478a-a50e-d6ad243b7bd9","arxiv_id":"2501.14520","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OpenSC transmits structured scene graphs produced by a scene-graph generator, maps their text tokens to QAM symbols via BERT token IDs, and uses an LLM with retrieval-augmented generation to answer visual questions.","lead":"This paper builds OpenSC, a semantic communication system that transmits scene graphs instead of raw images, then uses a large language model to answer visual questions about the scene. It reports large bandwidth savings and better question-answering recall than several text- and image-based baselines on an aerial urban image dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uncoded QAM mapping plus self-referential RAG cannot substantiate the claimed robustness: the reported VQA gains need a noiseless upper bound and token-error-rate check.","rationale":"Reader's weakest_assumption and my concern align: the robustness of the uncoded token-to-QAM mapping plus the self-referential RAG is the load-bearing joint. I agree with the CONDITIONAL verdict; no change is needed beyond making the condition explicit. The paper is not internally inconsistent on its face, but the empirical support for noise robustness is incomplete: no code/data, no error bars, no bit/token-error statistics, and an unexplained invariance to modulation order. The factor-of-100 efficiency-table error is real but presentation-level and fixable; it is not the deepest issue. The proposed three-arm test directly targets whether LLM correction is actually correcting channel errors or merely rephrasing received content, and whether the reported recall is already saturated by a noiseless scene graph. If the clean upper bound is close to the reported numbers, the claimed bandwidth savings would remain but the 'significant improvements' in semantic understanding over noisy channels would need to be re-scoped. If the raw corrupted text already matches the full pipeline, the RAG/LLM module is not adding value. The test is concrete, inexpensive (same dataset, same model calls), and would settle the central ambiguity.","tokens_in":15587,"tokens_out":7034,"duration_ms":68928,"concrete_test":"On the AUG test set, run the Section 4.2 transmitter/receiver at SNRs 0, 3, 6, 9, 12, 15, 18 dB under AWGN and Rayleigh, and before any LLM enhancement compute the WordPiece token error rate between the transmitted and demodulated token IDs. Then compare VQA recall/F1 for three decoders: (1) raw corrupted scene graph fed directly to VQALLM without RAG, (2) full OpenSC pipeline, (3) clean uncorrupted scene graph upper bound. If (3) matches the reported OpenSC numbers and (1) already matches (2), then the LLM/RAG correction and the 'open channel coding' contribution are not load-bearing; if (2) fails to approach (3) at low SNR, the no-FEC design is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency/robustness claim depends on the scene-graph text surviving the channel well enough for VQA. Section 4.2 maps BERT token IDs directly to QAM symbols (Eq. 11) with no forward error correction, so every bit error changes a WordPiece token; there is no channel code to recover it. The receiver's error-correction path is Section 4.3.5, which builds a vector database from chunks of the received information itself and retrieves the four nearest chunks. Because that database is derived from the same corrupted stream, the LLM is not given any independent reference (no original image, no clean knowledge base) from which to reconstruct what was actually lost. It can only rephrase or plausibly 'fill in' based on language priors, and when a key attribute (e.g., a color or relationship) is flipped to another valid word, the retrieved chunks will reinforce the wrong value. The paper reports only downstream recall/F1 and never reports token error rates, bit error rates, or a noiseless scene-graph upper bound; Figures 5-6 even show nearly identical performance across BPSK/4QAM/16QAM, which is unexplained if symbol errors are actually present at the tested SNRs. Without these measurements, the 4160-symbol advantage over JPEG-LDPC cannot be separated from an effectively noiseless or metric-insensitive evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OpenSC, a semantic communication system for visual question answering (VQA). At the transmitter, a scene graph is extracted from the input image, the graph text is tokenized with WordPiece, and the BERT token IDs are mapped directly to QAM symbols without forward error correction (Eq. (11)). The receiver demodulates the symbols, reconstructs token IDs and text, and then uses an LLM with retrieval-augmented generation to answer questions about category, quantity, location, and relationship. The RAG database is built from the received chunks themselves. Experiments on the AUG dataset compare OpenSC with DeepSC, MJCMSC, LLM-SC, and traditional schemes (5-bit+RS, Huffman+RS, JPEG+LDPC) over AWGN and Rayleigh channels, reporting recall/F1 curves and symbol counts. The headline efficiency result is that OpenSC transmits 4,160 symbols per item versus 16,242,744 for JPEG-LDPC while achieving higher recall and F1.","tokens_in":15723,"tokens_out":6249,"duration_ms":56937,"significance":"If substantiated, the paper would show a practically important result: a scene-graph description of an image can be communicated with orders-of-magnitude fewer symbols than JPEG-compressed image transmission while supporting VQA more accurately. The integration of scene graph generation, BERT tokenization, QAM modulation, and LLM-based semantic decoding is a plausible and timely system design. The paper also contains useful ablations over LLM choices and comparisons against several baselines, and it is clearly written in its high-level architecture. However, the central robustness and efficiency claims currently rest on an unspecified evaluation protocol, a self-referential RAG mechanism, and the absence of any channel-error measurement. The paper does not provide code, data, or machine-checked proofs, so independent verification is not possible from the manuscript alone.","major_comments":[{"comment":"The transmitter maps BERT token IDs directly to QAM symbols with no forward error correction, so a single bit error changes a WordPiece token and there is no channel code to recover it. The paper nevertheless claims robustness across AWGN and Rayleigh channels, but it reports no token error rate or bit error rate and no noiseless scene-graph upper bound. Moreover, Figs. 5–6 show nearly identical recall/F1 for BPSK, 4QAM, and 16QAM across SNR, which is unexplained if symbol errors are actually present at the tested SNRs; at a fixed symbol SNR these modulations have very different symbol error rates. Please report token error rates, add a noiseless upper bound, and explain the modulation invariance (or show that the SNR range is effectively error-free).","section":"Sec. 4.2, Eq. (11), and Figs. 5–6"},{"comment":"The RAG module builds its vector database from chunks of the received information itself and retrieves the four closest chunks to seed the LLM. Because the database is derived from the same corrupted stream, it provides no independent evidence about the original image; if a transmitted attribute is corrupted to another plausible word, retrieval will reinforce the wrong value. The claimed 'semantic enhancement' is therefore not separable from the LLM's language priors. Please evaluate the system with a clean knowledge base built from ground-truth scene graphs, and also with retrieval disabled, to quantify what the RAG step actually contributes.","section":"Sec. 4.3.5"},{"comment":"The evaluation protocol for recall and F1 is not specified: the paper does not state the test-set size, the number of questions per type, or the matching procedure between open-ended LLM answers and ground-truth objects, quantities, locations, and relationships. Figures 3–8 report point estimates only, with no error bars, confidence intervals, or significance tests. The word 'significant' in the abstract and conclusion is therefore not statistically supported. Please describe the exact evaluation protocol and report variability measures.","section":"Sec. 5.1.2 and Sec. 5.3"},{"comment":"The system model in Eqs. (2)–(3) describes trainable channel encoder and decoder parameters γ and θ, but Section 4.2 and Algorithm 1 state that channel encoding and decoding use a fixed LLM with constellation tokens and require no additional training. This mismatch means that what the paper calls 'open channel coding' is in fact uncoded QAM modulation plus LLM-based post-processing, not a trainable channel code. Please either instantiate the trainable channel code described in the system model or revise the system model to reflect the actual uncoded scheme.","section":"Sec. 3 vs. Sec. 4.2 and Algorithm 1"},{"comment":"The headline efficiency ratio is misstated. 4,160 / 16,242,744 ≈ 2.56 × 10^-4 = 0.0256%, not '2.6 × 10^-4%' as claimed in the text, and Table 3 labels the column 'Ratio (%)' but enters '2.6 × 10^-4' as if it were a dimensionless ratio. The qualitative conclusion is unchanged, but the numbers should be corrected. In addition, please state the SNR/channel conditions assumed for the symbol-count comparison and confirm that the baselines achieve comparable VQA performance at that operating point.","section":"Sec. 5.5 and Table 3"},{"comment":"The ablation rows in Table 2 are hard to interpret: the first row has SSC absent but OCC present, even though OCC is described as transmitting scene-graph tokens, and the third row reports no metrics at all. Please define what each ablation actually transmits and reports, and provide complete results for all rows, including the missing values.","section":"Sec. 5.4.1 and Table 2"}],"minor_comments":[{"comment":"The section title contains a typo: 'Open Channel Condings' should be 'Open Channel Coding'.","section":"Sec. 4.2 title"},{"comment":"Algorithm 1 has 'taining' instead of 'training', and Section 5.1.3 contains the incomplete sentence 'Learning rate decay occurs sets the 90,000 and 120,000 iterations.'","section":"Algorithm 1 and Sec. 5.1.3"},{"comment":"The column label 'BLUE (1-gram)' should be 'BLEU (1-gram)'.","section":"Table 1"},{"comment":"Figures 3–8 refer to 'A WGN' with a space; this should be 'AWGN'.","section":"Figure captions"},{"comment":"The paper alternates between 'LDPC' and 'LPDC'; the latter is a misspelling that should be corrected throughout.","section":"Sec. 5.5 and Table 3 caption"},{"comment":"Equation (5) is a verbatim repeat of Eq. (4); one of them should be removed or the two should be differentiated.","section":"Sec. 3.2.2"},{"comment":"The ACM template metadata still contains the placeholder conference name, 2018 date, and DOI; this should be updated before submission.","section":"Title page and metadata"},{"comment":"The sentence 'Similarly, proposed a semantic communication framework (LAM-SC)...' is missing the author names before 'proposed'.","section":"Sec. 2.1"},{"comment":"The visualization section claims the proposed method produces 'only three minor errors', but no protocol for counting errors is given.","section":"Sec. 5.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is an arXiv preprint using an ACM template with stale metadata, and the novelty claim of 'open channel coding' is overstated relative to the actual uncoded QAM plus LLM post-processing. The core idea is worth pursuing, but the current evaluation cannot support the robustness and efficiency claims without token-error measurements, a noiseless upper bound, and a non-circular RAG validation. I recommend major revision rather than rejection, since these issues are addressable with additional experiments and rewriting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: OpenSC is a plausible system with a genuinely new combination—scene-graph extraction, BERT-token-to-QAM mapping, and LLM-based retrieval-augmented answering—but the evaluation as written doesn't substantiate the headline efficiency and robustness claims. I'd send it to a competent referee, but it needs real work.\n\nWhat's new and good: The idea of transmitting a compact scene graph instead of pixels for VQA is sensible, and the reported symbol counts (4,160 vs. 16,242,744 for JPEG-LDPC) are the kind of number that makes you sit up. The system is coherent end-to-end, and the ablation study (Table 2) at least shows that removing either the structured semantic encoder or the open channel coder hurts performance. That's real evidence the components matter.\n\nThe soft spots are equally real. First, the evaluation is under-specified: point estimates only, no error bars, no test-set sizes, no significance tests, and no released code or data. Second, there's a concrete math error in Table 3: the ratio 4,160/16,242,744 is 2.6e-4 as a fraction, or about 0.026%, but the text calls it '2.6e-4%'—off by a factor of 100. Third, the robustness claim is undercut by the fact that BPSK, 4QAM, and 16QAM give nearly identical recall/F1 across all SNRs (Figures 5-6). That's suspicious: if symbol errors matter, higher-order modulation should degrade; if they don't, the SNR range or metric is not stress-testing the system. The paper never reports bit or token error rates, nor a noiseless upper bound.\n\nThe bigger conceptual issue is the self-referential RAG. Section 4.3.5 builds the vector database from the received scene graph itself, retrieves the four nearest chunks, and feeds them to the LLM. So the LLM cannot recover information that was actually lost on the channel—it can only rephrase or guess from language priors. When a key token flips to another valid word, the retrieved chunks will reinforce the wrong answer. That doesn't kill the system (the LLM's priors may help for common cases), but it means the 'semantic enhancement' is not error correction in any independent sense.\n\nBottom line: for someone working on task-oriented semantic communication, this is worth a look. The efficiency story is interesting and the system is coherent. But the evidence is not yet at the level of the claims. Deserves a serious peer review, with the expectation of major revision: released code, statistical rigor, token-error measurements, a noiseless upper bound, and a design or argument that gets around the RAG circularity.","headline":"Plausible scene-graph transmission system with a dramatic efficiency claim that the current evaluation doesn't yet back up.","tokens_in":16408,"tokens_out":3389,"would_cite":false,"duration_ms":28623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A scene graph transmitted as text tokens over QAM symbols and repaired by an LLM at the receiver can answer visual questions with roughly 4,160 symbols instead of the 16 million needed by image compression.","keywords":["semantic communication","scene graph encoding","open channel coding","visual question answering","large language models","retrieval-augmented generation","QAM modulation","6G networks"],"falsifier":"Disable the receiver-side LLM and retrieval-augmented repair, feed the demodulated token stream directly into a text-only question answering model, and measure recall on the four VQA question types; if recall collapses toward the ablation levels reported when the open coder is removed, then the LLM, not the open channel coding, is what makes the system work.","tokens_in":15244,"feed_emoji":"📡","tokens_out":12080,"duration_ms":98093,"temperature":0.7,"pith_summary":"The paper proposes OpenSC, a semantic communication system that turns an image into a scene graph and transmits that graph's text rather than the image pixels. On the transmit side, scene graph triples are tokenized with a pretrained transformer tokenizer, mapped to token IDs, and converted to QAM constellation symbols; no learned channel code is trained. On the receive side, the symbols are demodulated back to tokens, and a large language model with retrieval-augmented generation repairs the corrupted text and answers visual questions. The paper reports that on a 400-image aerial urban dataset, this pipeline answers category, quantity, location, and relationship questions with higher recall and F1 than conventional image compression and prior semantic communication baselines, while sending about 4,160 symbols per item instead of the 16,242,744 symbols used by JPEG with LDPC coding. If the claim holds, visual semantic communication can shift from fixed task-specific codebooks to shared language-model knowledge.","feed_headline":"Scene-graph text beats image compression: 4,160 symbols vs 16 million","feed_subtitle":"OpenSC turns an image into scene graph tokens; an LLM repairs the noise and answers visual questions.","key_machinery":"The load-bearing object is the scene-graph-to-text pipeline used as an open channel code. An image is first parsed into subject-predicate-object triples by a prototype-based scene graph generator; the triples are tokenized into subword units and mapped through the token vocabulary into binary strings, and those strings are segmented into $m$-bit chunks that become QAM constellation symbols. The receiver estimates the channel (zero-forcing linear MMSE with channel state information, or minimum-Euclidean-distance demodulation without it), converts symbols back to token IDs, and reconstructs the text. The paper's named innovation, open channel coding, is the replacement of a trained joint source-channel code with this token-to-constellation mapping plus receiver-side semantic repair: the LLM is prompted with the reconstructed text augmented by retrieved chunks from a vector database built from the received scene graph information, and it outputs a JSON summary (number, location, relationship) that supports final question answering. This is what lets the system claim knowledge-agnostic, training-free channel coding.","core_discovery":"OpenSC's central claim is that a scene graph, written as subject-predicate-object triples, is a sufficient semantic representation for visual question answering over wireless channels, and that the text itself can be protected not by error-correcting codes but by a large language model at the receiver. The transmitter converts the graph text into token IDs and then into QAM symbols; the receiver reverses this mapping and feeds the reconstructed text, together with structured JSON fields for number, location, and relationship, into a retrieval-augmented large language model that fills gaps and answers the question without ever seeing the original image. In the reported experiments on the aerial urban dataset, this yields recall values of 0.6401 for category, 0.3255 for quantity, 0.3227 for location, and 0.1398 for relationship at 18 dB SNR with the Qwen-plus receiver model, and it outperforms the baselines across most SNR points in both AWGN and Rayleigh channels. The paper also reports that changing the modulation scheme (BPSK, 4QAM, 16QAM) barely changes performance, and that swapping the receiver LLM causes only minor variation, which it reads as evidence that the semantic code, not the channel code or the answer model, carries the robustness.","pith_inferences":["Editorial inference: the 4,160-symbol figure is meaningful only relative to the scene graph's token count; for images with richer scenes or questions about attributes outside the number/location/relationship JSON slots, the symbol budget and the efficiency ratio would both grow.","Editorial inference: a direct stress test would rebuild the retrieval vector database from the corrupted received text instead of from clean scene graph chunks; if performance drops sharply, the reported robustness is partly an artifact of clean-side retrieval.","Editorial inference: the approach transfers to other visual domains only when a scene graph generator with matching object and predicate categories is available; domain shift in the graph vocabulary would silently drop the semantics needed by downstream questions.","Editorial inference: a natural extension is to compare OpenSC against a learned joint source-channel code that transmits the same scene graph text, isolating whether the LLM's repair, rather than the scene graph format itself, causes the accuracy gain."],"forward_implications":["If correct, the transmitter does not need to send image pixels at all for visual question answering: the scene graph text carries the task-relevant semantics, which implies large bandwidth savings for visual tasks on limited-resource links.","Because the channel coding stage has no learned parameters, the same OpenSC transmitter can in principle serve different downstream tasks and different receiver LLMs without retraining, as long as the scene graph vocabulary covers the task.","The near-invariance to modulation order means the system can trade spectral efficiency for robustness by choosing BPSK, 4QAM, or 16QAM after measuring the channel, rather than fixing one codebook.","The reported symbol count implies that structured text semantics can be more bandwidth-efficient than compressed image transmission by several orders of magnitude in this aerial-image setting."],"supporting_citations":[{"why":"Supplies the 400-image aerial urban dataset with object and relationship annotations on which all training and evaluation are run.","marker":"[38]"},{"why":"Supplies the scene graph generator that turns images into subject-predicate-object triples for the semantic encoder.","marker":"[37]"},{"why":"Supplies the detection backbone that produces the bounding boxes and feature maps used by the scene graph generator.","marker":"[39]"},{"why":"Supplies the tokenizer and token-ID vocabulary that converts scene graph text into binary strings and back into text.","marker":"[9]"},{"why":"Provides the receiver-side visual question answering LLM used in the main experiments.","marker":"[48]"},{"why":"Provides the retrieval-augmented generation mechanism that retrieves the nearest chunks and builds the enhancement prompt for the LLM.","marker":"[49]"},{"why":"Provides an LLM-based semantic communication baseline that transmits scene graph text and questions for comparison.","marker":"[44]"},{"why":"Provides the joint coding-modulation baseline that is extended into a multimodal system for comparison.","marker":"[43]"},{"why":"Provides the learned text semantic communication baseline and the ablation alternative used when the open coder is removed.","marker":"[5]"}],"fun_headline_variants":["Scene graph text tops raw image over wireless links","No error correction needed: LLM repairs scene-graph text","OpenSC sends scene graph as text, LLM answers VQA","Scene graphs + LLM beat image coding in noisy channels","4,160 symbols for scene graph, LLM fixes the rest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire efficiency gain rests on the premise that the receiver's large language model can repair a corrupted scene graph text well enough to answer visual questions, even though the symbols carry no error-correcting code and the original image is never transmitted.","fun_headline_variants_meta":{"raw":{"variants":["Scene graph text tops raw image over wireless links","No error correction needed: LLM repairs scene-graph text","OpenSC sends scene graph as text, LLM answers VQA","Scene graphs + LLM beat image coding in noisy channels","4,160 symbols for scene graph, LLM fixes the rest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1489,"prompt_tokens":1006,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":622,"tokens_out":483,"duration_ms":5481,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:04:04.862829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Disable the receiver-side LLM and retrieval-augmented repair, feed the demodulated token stream directly into a text-only question answering model, and measure recall on the four VQA question types; if recall collapses toward the ablation levels reported when the open coder is removed, then the LLM, not the open channel coding, is what makes the system work.","supporting_citations":[{"cited_title":"Prototype-based embedding network for scene graph generation","cited_arxiv_id":null,"evidence_quote":"Supplies the scene graph generator that turns images into subject-predicate-object triples for the semantic encoder."},{"cited_title":"BERT: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the tokenizer and token-ID vocabulary that converts scene graph text into binary strings and back into text."},{"cited_title":"Karagiannidis","cited_arxiv_id":null,"evidence_quote":"Provides the joint coding-modulation baseline that is extended into a multimodal system for comparison."},{"cited_title":"Deep learning enabled semantic communication systems","cited_arxiv_id":null,"evidence_quote":"Provides the learned text semantic communication baseline and the ablation alternative used when the open coder is removed."}],"review_version":1}