{"id":"2d9b94a9-4bf2-411d-944b-51ebaddd5f27","arxiv_id":"2504.13995","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLaNA processes NeRF network weights directly with a frozen meta-encoder and a LLaMA 2 backbone, beating image- and point-cloud-based baselines on NeRF captioning and Q&A, and is trained on a new 280K-object ObjaNeRF-Text dataset.","lead":"Researchers trained a multimodal AI that reads the internal weights of a NeRF, a neural network that stores a 3D object's shape and appearance, to caption it and answer questions about it. It outperforms AI models that first render images or point clouds from the NeRF on captioning and Q&A benchmarks, though not on all tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The direct-weight superiority claim is not yet isolated from an information-asymmetry artifact: image baselines get at most 3 views, point-cloud extraction is unspecified, and LLaNA's encoder is pre-trained on the target NeRF distribution.","rationale":"The central claim is an empirical comparison. The weakest link is not the model's ability (the captioning and Q&A results are strong and survive a from-scratch retraining protocol) but the validity of the comparison: no experiment shows that the baselines' representations are informationally comparable to the NeRF weights. The paper's own discussion of viewpoint and resolution sensitivity makes the confound explicit, and the point-cloud extraction is not described, so a reviewer cannot tell whether PointLLM or GPT4Point receives a fair sample. The from-scratch ablation is an important positive, since it rules out a simple pretraining-data advantage, but it leaves the encoder-adaptation asymmetry: nf2vec was pre-trained on the exact NeRF training distribution, whereas baseline encoders are frozen. Consequently the appropriate verdict is CONDITIONAL: accept only if a fidelity sweep shows the margin is robust, or if the claim is narrowed to a fixed, limited baseline budget. I agree with the reader's weakest assumption; the concrete test above would settle whether the concern actually lands.","tokens_in":25712,"tokens_out":8901,"duration_ms":82034,"concrete_test":"Run a representation-fidelity sweep on the ShapeNeRF-Text benchmark: (a) for LLaVA-13b/BLIP-2, render 1, 3, 6, 12, and 24 views and report S-BERT/SimCSE and classification accuracy; (b) for PointLLM/GPT4Point, extract point clouds from the same NeRFs using a specified method (e.g., density-thresholded marching cubes at 128^3) at 1K, 8K, 64K, and 256K points and re-evaluate. If baseline scores plateau below LLaNA at the highest fidelity, the direct-weight advantage is supported; if they approach or cross LLaNA, the headline claim reduces to an information-fairness artifact. This directly tests whether the full NeRF-weight information content, rather than weight-space processing per se, drives the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the claim that processing NeRF weights is intrinsically better than processing 2D or 3D renderings, the baseline representations must carry a comparable amount of object information. The paper never establishes this. Image baselines are limited to N=3 rendered views (Section V), and the point-cloud baselines are fed an extraction whose method and resolution are never specified; the paper instead argues that low resolution and bad viewpoints hurt image/point-cloud models (Section I), which is precisely the confound. The from-scratch ablation in Section V-B equalizes training annotations but not encoder adaptation: LLaNA's nf2vec meta-encoder is pre-trained on the train-set NeRFs with an image-reconstruction objective, while LLaVA/PointLLM/GPT4Point keep their modality encoders frozen per official protocols. Thus LLaNA's encoder is domain-adapted to the exact input distribution; the measured gap may reflect this adaptation plus information asymmetry rather than a fundamental advantage of weight-space processing. A related internal inconsistency is that the abstract's blanket claim is contradicted by Table VII, where LLaVA-13b multi-view (73.45) beats LLaNA-13b (69.27) on zero-shot classification, showing the advantage is task-dependent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LLaNA, a multimodal large language model that ingests NeRF MLP weights through an nf2vec meta-encoder and projects the resulting global embedding into a LLaMA-2 LLM. It introduces ObjaNeRF-Text, a 280K-object NeRF-language dataset built from Objaverse, with text annotations inherited from PointLLM and GPT4Point after replacing point-cloud vocabulary with 'NeRF' terms. The authors evaluate LLaNA-7b/13b against image-based (LLaVA, BLIP-2), point-cloud (PointLLM, GPT4Point), and mesh/multi-view (3D-LLM) baselines on brief and detailed captioning, single-round Q&A, and zero-shot classification, including a from-scratch ablation on ShapeNeRF-Text. The main claim is that directly processing NeRF weights leads to better performance on NeRF-language tasks than approaches that use 2D or 3D representations derived from NeRFs.","tokens_in":26122,"tokens_out":7479,"duration_ms":63523,"significance":"If fully established, the central claim would be a notable contribution: it would demonstrate that an MLLM can reason about object semantics directly from NeRF parameters, without rendering or materializing explicit geometry, and ObjaNeRF-Text would be a large-scale resource for future work. The paper has concrete strengths: it retrains several baselines from scratch on the same training split, it includes a language-only control showing that the benchmark cannot be solved by the LLM's priors alone, and it provides a scaling analysis across 7B and 13B models. However, the comparison supporting the headline claim is not yet fully controlled, and the abstract overstates a claim that the paper's own Table VII contradicts.","major_comments":[{"comment":"The abstract states that directly processing NeRF weights leads to better performance on NeRF-Language tasks than approaches relying on 2D or 3D representations derived from NeRFs, but Table VII and the accompanying text in Section V-A.d report that LLaVA-13b multi-view (73.45) outperforms LLaNA-13b (69.27) on zero-shot NeRF classification, and the text explicitly identifies LLaVA-13b as the best model for that task. The blanket claim is therefore contradicted by the paper's own results; it should be qualified to the captioning and Q&A tasks, or revised to state that the advantage holds on most, but not all, evaluated tasks.","section":"Abstract / Table VII"},{"comment":"The main comparison is confounded by information asymmetry. Image baselines receive at most N=3 rendered views (Section V-A), and the paper argues in Section I that low resolution and unfavorable viewpoints disadvantage image- and point-cloud-based models, which is exactly the confound: the derived representations are given less information than the full NeRF weights. The point-cloud extraction method, density, and resolution are never specified. To support the claim that weight-space processing is intrinsically superior, the authors should specify the point-cloud extraction protocol and run ablations that equalize the information budget, for example by rendering many more views at high resolution for image models and by using dense point clouds of a controlled, representative size.","section":"Section V-A / V-B"},{"comment":"A second confound is encoder adaptation. The nf2vec meta-encoder is pre-trained on the NeRFs of ShapeNeRF-Text and ObjaNeRF-Text with an image-reconstruction objective (Section III), while the official protocols for LLaVA, PointLLM, and GPT4Point keep their modality encoders frozen (Section V-B). LLaNA's encoder is therefore domain-adapted to the exact test distribution, and the performance gap may reflect this adaptation rather than an inherent advantage of processing weights. The from-scratch ablation in Section V-B equalizes training annotations but not encoder pre-training; a cleaner comparison would pre-tune the baseline encoders on analogous derived representations from the same training NeRFs, or use an nf2vec pre-trained on a disjoint distribution.","section":"Section III / Section V-B"}],"minor_comments":[{"comment":"Table IV lists two rows labeled 'LLaVA-vicuna-7b Image (RV)' with different scores, and Table XII lists two rows labeled 'LLaVA-vicuna-13b Image (FV)'; one of each is presumably a different condition such as multi-view, and the labels should be corrected.","section":"Table IV / Table XII"},{"comment":"The ObjaNeRF-Text annotations are original point-cloud captions with words like 'point cloud' converted to 'NeRF'; the semantic validity of the modified captions for the NeRF modality is not assessed, and the paper should report whether any human validation or a quantitative analysis of the replacement's impact was performed.","section":"Section IV-A"},{"comment":"All results are single point estimates with no error bars, significance tests, or multiple runs, and some reported margins are small (e.g., Table III S-BERT 42.08 vs 41.01; Table IV 44.26 vs 44.15). Adding uncertainty estimates would clarify which differences are meaningful.","section":"Tables I-XIII"},{"comment":"The statement 'we followed their protocol, which, for all methods, keeps the modality-specific encoder frozen and trains an adaptor' is ambiguous for LLaNA, since its nf2vec encoder is already pre-trained and then frozen; please clarify whether any LLaNA variant was trained from a randomly initialized meta-encoder.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct extension of the authors' earlier LLaNA preprint and relies heavily on the nf2vec meta-encoder from their group; the editor may wish to weigh the incremental contribution relative to those prior works. Releasing the benchmark and code would also strengthen the manuscript's reproducibility and impact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper delivers a genuinely new resource: ObjaNeRF-Text, 280K NeRFs with text annotations, plus a clean benchmark split. That alone makes the paper worth engaging with. Second, the central claim—that an MLLM can reason about an object's NeRF weights directly, without rendering—holds for captioning and Q&A, and the evidence is substantial. But the abstract's blanket superiority claim does not survive Table VII, where LLaVA-13b multi-view beats LLaNA on zero-shot classification, and the weight-versus-derivation comparison has a confound the authors do not fully address.\n\nCredit where due: the dataset construction is careful, the from-scratch ablation (Tables VIII–XII) retrains baselines on the same training data and LLaNA still wins, and the language-only baselines in Section V-C show the tasks actually require object information. The scaling analysis (7B vs 13B) is honest: larger LLMs give marginal gains, which is a useful negative result. The qualitative examples are compelling.\n\nSoft spots, in order. The biggest is the information-asymmetry problem. Image baselines get at most 3 rendered views; the point-cloud extraction is never specified—method, resolution, sample count. LLaNA's nf2vec encoder is pre-trained on the exact train distribution, while LLaVA/PointLLM/GPT4Point use frozen encoders. So part of the measured gap is domain adaptation and representation fidelity, not a fundamental advantage of weight space. The paper argues that bad viewpoints and low resolution hurt baselines, but that is precisely the confound. The result is still practically meaningful—if you have a NeRF, LLaNA is the better assistant—but the causal claim that weights are intrinsically better is not isolated. Second, the abstract says \"better performance on NeRF-Language tasks\" without noting the classification reversal. That should be fixed. Third, minor: no error bars or significance tests, and the point-cloud extraction details need to be in the paper. The self-citation to nf2vec and prior LLaNA is disclosed and legitimate; the new material is clearly separated.\n\nBottom line: this is a solid empirical paper with a useful benchmark. It deserves a serious referee, with requested revisions: narrow the claim, specify the point-cloud extraction, and discuss the pre-training confound. I'd cite the dataset.","headline":"Solid scaling paper with a genuinely useful new dataset; the weight-vs-rendering claim is real for captioning and Q&A but the abstract overstates it and the comparison is not cleanly isolated.","tokens_in":26555,"tokens_out":2488,"would_cite":true,"duration_ms":23081,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an MLLM, LLaNA, can directly read the weights of a NeRF's MLP and, without rendering images or extracting point clouds, caption the object, answer questions about it, and classify it, outperforming baselines that…","keywords":["Neural Radiance Fields","Multimodal Large Language Models","weight-space learning","meta-networks","3D captioning","3D question answering","zero-shot classification","Objaverse"],"falsifier":"Render all 40 training views (or a large random sample) for the image baselines and extract a dense, high-resolution point cloud or mesh for the 3D baselines, then re-run the same benchmark; if LLaNA's margin disappears or becomes negligible, the claim that weight-space input is intrinsically superior is refuted. A complementary check is to have human annotators judge whether the 'point cloud' to 'NeRF' rewritten captions remain valid descriptions of the NeRF, since systematic invalidity would undermine the benchmark.","tokens_in":25516,"feed_emoji":"🤖","tokens_out":9958,"duration_ms":81053,"temperature":0.7,"pith_summary":"This paper asks whether a multimodal language model can understand a 3D object by reading the raw parameters of a Neural Radiance Field (NeRF) — the network that stores an object's geometry and appearance — instead of rendering images or extracting point clouds from it. The authors build LLaNA, which uses a meta-network to compress the NeRF's MLP weights into a single embedding, projects that embedding into a LLaMA language model, and fine-tunes the LLM to caption, answer questions about, and classify the object. To train and test this idea, they assemble ObjaNeRF-Text, 280K NeRFs from Objaverse with paired text, plus the existing ShapeNeRF-Text. On their benchmark, LLaNA-13b scores 75.09 versus 61.00 for the best image-based baseline on brief captioning, and 81.05 versus 74.65 for the best point-cloud model on single-round Q&A. The paper concludes that the weights themselves carry enough semantic information for language tasks, and that for this setting a larger LLM matters less than the encoder design and instruction tuning.","feed_headline":"Reading NeRF weights outperforms rendered views and point clouds","feed_subtitle":"LLaNA-13b reads MLP weights directly, beating image and point-cloud baselines on captioning and Q&A.","key_machinery":"The load-bearing component is the nf2vec meta-encoder, a per-row MLP with batch normalization that processes each row of a matrix formed by stacking the NeRF's weight matrices and biases, then max-pools the row tokens into one 1024-dimensional global embedding. A three-layer GeLU projector maps that embedding into LLaMA-2's token embedding space, bracketed by learned <n_start> and <n_end> tokens, and the LLM is instruction-tuned on paired captions and dialogues. Because the NeRFs all share a fixed architecture (3 hidden layers, 64 units, 24-frequency positional encoding), row-stacking is consistent, and the paper argues the resulting global vector distills the object's geometry and appearance from weights alone, sidestepping the viewpoint and resolution choices that hurt image- and point-cloud-based baselines.","core_discovery":"The central discovery is that an MLLM can extract object-level semantic information from the parameters of a NeRF's MLP alone. LLaNA never renders a pixel and never materializes a point cloud, yet it outperforms LLaVA, BLIP-2, PointLLM, GPT4Point, and 3D-LLM on NeRF captioning and Q&A across ShapeNeRF-Text and ObjaNeRF-Text. The paper attributes this to the completeness of weight-space information: rendered images depend on a chosen viewpoint, and point clouds depend on an extraction resolution, whereas the weights encode the whole object at once. It also reports that instruction fine-tuning of the LLM is a major contributor, since the best-performing methods are exactly those that fine-tune the LLM, and that scaling the LLM from 7B to 13B yields only marginal improvements.","pith_inferences":["If weight-space reading truly beats rendered views at equal information, then rendering a NeRF before asking a language model to describe it is not just inefficient but lossy; a 20-view or full-360-degree baseline would likely narrow but perhaps not close the gap.","The paper's reuse of point-cloud captions (replacing 'point cloud' with 'NeRF') could inflate LLaNA's scores if the original captions describe cloud-specific artifacts; a human validity study on the rewritten captions would settle this.","The 'LLM size doesn't matter' finding is conditional on a small, fixed NeRF MLP; with larger or hybrid NeRF architectures, the encoder may need more capacity, re-opening the scaling question.","A natural next test is zero-shot generalization across NeRF architecture families: if the meta-encoder is retrained on hash-grid or tri-plane NeRFs, one can check whether the weight-space advantage persists beyond MLP-only NeRFs."],"forward_implications":["A NeRF assistant can be built with no rendering or surface-extraction stage: understanding costs one forward pass of the meta-encoder, so captioning a NeRF is cheaper than generating even a single image.","Scaling the LLM from 7B to 13B improves S-BERT by only fractions of a point, so for NeRF-language tasks the encoder/projector and LLM fine-tuning dominate; compute is better invested there than in a larger backbone.","Because no viewpoint is chosen, the method is invariant to object orientation and camera position, a property that matters for 3D datasets like Objaverse where front and back views are not standardized.","The ObjaNeRF-Text benchmark, with human-written test captions from the PointLLM and GPT4Point splits, gives the community a way to measure NeRF-language understanding against both 2D and 3D baselines.","For detailed captioning the gap is largest (75.51 vs 60.21 S-BERT), suggesting that a handful of views, however chosen, loses details that the weights retain."],"supporting_citations":[{"why":"Defines NeRF, the input modality whose MLP weights LLaNA reads; supplies the MLP architecture and volumetric rendering background.","marker":"[13]"},{"why":"Provides nf2vec, the meta-encoder that maps NeRF weight matrices to a global embedding; LLaNA adopts it as its encoder.","marker":"[15]"},{"why":"Provides LLaMA-2, the LLM backbone whose embedding space the NeRF token is projected into and which is instruction-tuned.","marker":"[4]"},{"why":"PointLLM: source of the ObjaNeRF-Text textual annotations and one test split, and the strongest point-cloud baseline on Q&A.","marker":"[10]"},{"why":"GPT4Point: supplies the other ObjaNeRF-Text test split and a point-cloud baseline for captioning and classification.","marker":"[11]"},{"why":"LLaVA: used in the automated annotation pipeline for ShapeNeRF-Text and as the image baseline, including multi-view variants.","marker":"[7]"},{"why":"Introduces the original LLaNA and the ShapeNeRF-Text dataset of 40K NeRFs with generated annotations; this paper scales that line of work.","marker":"[17]"},{"why":"Objaverse: the 3D object repository from which the 280K ObjaNeRF-Text NeRFs are trained.","marker":"[19]"}],"fun_headline_variants":["LLaNA reads NeRF weights, beats images and point clouds","NeRF weights alone beat renders and point clouds in MLLM","LLaNA: MLLM that reads NeRF weights, no rendering needed","Direct NeRF weight processing tops 2D/3D baselines","LLaNA: NeRF captioning without rendering or point clouds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the image and point-cloud baselines get a fair look at the NeRF: image models see at most three rendered views and the point-cloud extraction method is unspecified, so part of the measured advantage may come from the baselines being under-informed rather than from weight-space processing being intrinsically better.","fun_headline_variants_meta":{"raw":{"variants":["LLaNA reads NeRF weights, beats images and point clouds","NeRF weights alone beat renders and point clouds in MLLM","LLaNA: MLLM that reads NeRF weights, no rendering needed","Direct NeRF weight processing tops 2D/3D baselines","LLaNA: NeRF captioning without rendering or point clouds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2950,"prompt_tokens":976,"completion_tokens":1974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1893}},"tokens_in":592,"tokens_out":1974,"duration_ms":12224,"temperature":1.0,"reasoning_tokens":1893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:58:15.575984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render all 40 training views (or a large random sample) for the image baselines and extract a dense, high-resolution point cloud or mesh for the 3D baselines, then re-run the same benchmark; if LLaNA's margin disappears or becomes negligible, the claim that weight-space input is intrinsically superior is refuted. A complementary check is to have human annotators judge whether the 'point cloud' to 'NeRF' rewritten captions remain valid descriptions of the NeRF, since systematic invalidity would undermine the benchmark.","supporting_citations":[{"cited_title":"Nerf: Representing scenes as neural radiance fields for view synthesis,","cited_arxiv_id":null,"evidence_quote":"Defines NeRF, the input modality whose MLP weights LLaNA reads; supplies the MLP architecture and volumetric rendering background."},{"cited_title":"Deep Learning on Object-centric 3D Neural Fields","cited_arxiv_id":"2312.13277","evidence_quote":"Provides nf2vec, the meta-encoder that maps NeRF weight matrices to a global embedding; LLaNA adopts it as its encoder."},{"cited_title":"Gpt4point: A unified framework for point-language understanding and generation,","cited_arxiv_id":null,"evidence_quote":"GPT4Point: supplies the other ObjaNeRF-Text test split and a point-cloud baseline for captioning and classification."}],"review_version":1}