Pith. sign in

REVIEW 3 major objections 4 minor 85 references

Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that an MLLM, LLaNA, can directly read the weights of a NeRF's MLP and, without rendering images or extracting point clouds, caption the object, answer questions about it, and classify it, outperforming baselines that…

desk verdict Solid scaling paper with a genuinely useful new dataset; the weight-vs-rendering claim is real for captioning and Q&A but the abstract overstates it and the comparison is not cleanly isolated. read the letter →

arxiv 2504.13995 v1 pith:R22W77W6 submitted 2025-04-18 cs.CV

Andrea Amaduzzi , Pierluigi Zama Ramirez , Giuseppe Lisanti , Samuele Salti , Luigi Di Stefano This is my paper · ORCID
classification cs.CV
keywords NeuralRadianceFieldsMultimodalLargeLanguageModelsweight-spacelearningmeta-networks3Dcaptioningquestionansweringzero-shotclassificationObjaverse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a multimodal language model can understand a 3D object by reading the raw parameters of a Neural Radiance Field (NeRF) — the network that stores an object's geometry and appearance — instead of rendering images or extracting point clouds from it. The authors build LLaNA, which uses a meta-network to compress the NeRF's MLP weights into a single embedding, projects that embedding into a LLaMA language model, and fine-tunes the LLM to caption, answer questions about, and classify the object. To train and test this idea, they assemble ObjaNeRF-Text, 280K NeRFs from Objaverse with paired text, plus the existing ShapeNeRF-Text. On their benchmark, LLaNA-13b scores 75.09 versus 61.00 for the best image-based baseline on brief captioning, and 81.05 versus 74.65 for the best point-cloud model on single-round Q&A. The paper concludes that the weights themselves carry enough semantic information for language tasks, and that for this setting a larger LLM matters less than the encoder design and instruction tuning.

What carries the argument

The load-bearing component is the nf2vec meta-encoder, a per-row MLP with batch normalization that processes each row of a matrix formed by stacking the NeRF's weight matrices and biases, then max-pools the row tokens into one 1024-dimensional global embedding. A three-layer GeLU projector maps that embedding into LLaMA-2's token embedding space, bracketed by learned <n_start> and <n_end> tokens, and the LLM is instruction-tuned on paired captions and dialogues. Because the NeRFs all share a fixed architecture (3 hidden layers, 64 units, 24-frequency positional encoding), row-stacking is consistent, and the paper argues the resulting global vector distills the object's geometry and appearance from weights alone, sidestepping the viewpoint and resolution choices that hurt image- and point-cloud-based baselines.

What would settle it

Render all 40 training views (or a large random sample) for the image baselines and extract a dense, high-resolution point cloud or mesh for the 3D baselines, then re-run the same benchmark; if LLaNA's margin disappears or becomes negligible, the claim that weight-space input is intrinsically superior is refuted. A complementary check is to have human annotators judge whether the 'point cloud' to 'NeRF' rewritten captions remain valid descriptions of the NeRF, since systematic invalidity would undermine the benchmark.

Watch

Extended reading notes

Core claim

The central discovery is that an MLLM can extract object-level semantic information from the parameters of a NeRF's MLP alone. LLaNA never renders a pixel and never materializes a point cloud, yet it outperforms LLaVA, BLIP-2, PointLLM, GPT4Point, and 3D-LLM on NeRF captioning and Q&A across ShapeNeRF-Text and ObjaNeRF-Text. The paper attributes this to the completeness of weight-space information: rendered images depend on a chosen viewpoint, and point clouds depend on an extraction resolution, whereas the weights encode the whole object at once. It also reports that instruction fine-tuning of the LLM is a major contributor, since the best-performing methods are exactly those that fine-tune the LLM, and that scaling the LLM from 7B to 13B yields only marginal improvements.

Load-bearing premise

The load-bearing assumption is that the image and point-cloud baselines get a fair look at the NeRF: image models see at most three rendered views and the point-cloud extraction method is unspecified, so part of the measured advantage may come from the baselines being under-informed rather than from weight-space processing being intrinsically better.

Editorial extensions

If this is right

  • A NeRF assistant can be built with no rendering or surface-extraction stage: understanding costs one forward pass of the meta-encoder, so captioning a NeRF is cheaper than generating even a single image.
  • Scaling the LLM from 7B to 13B improves S-BERT by only fractions of a point, so for NeRF-language tasks the encoder/projector and LLM fine-tuning dominate; compute is better invested there than in a larger backbone.
  • Because no viewpoint is chosen, the method is invariant to object orientation and camera position, a property that matters for 3D datasets like Objaverse where front and back views are not standardized.
  • The ObjaNeRF-Text benchmark, with human-written test captions from the PointLLM and GPT4Point splits, gives the community a way to measure NeRF-language understanding against both 2D and 3D baselines.
  • For detailed captioning the gap is largest (75.51 vs 60.21 S-BERT), suggesting that a handful of views, however chosen, loses details that the weights retain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If weight-space reading truly beats rendered views at equal information, then rendering a NeRF before asking a language model to describe it is not just inefficient but lossy; a 20-view or full-360-degree baseline would likely narrow but perhaps not close the gap.
  • The paper's reuse of point-cloud captions (replacing 'point cloud' with 'NeRF') could inflate LLaNA's scores if the original captions describe cloud-specific artifacts; a human validity study on the rewritten captions would settle this.
  • The 'LLM size doesn't matter' finding is conditional on a small, fixed NeRF MLP; with larger or hybrid NeRF architectures, the encoder may need more capacity, re-opening the scaling question.
  • A natural next test is zero-shot generalization across NeRF architecture families: if the meta-encoder is retrained on hash-grid or tri-plane NeRFs, one can check whether the weight-space advantage persists beyond MLP-only NeRFs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents LLaNA, a multimodal large language model that ingests NeRF MLP weights through an nf2vec meta-encoder and projects the resulting global embedding into a LLaMA-2 LLM. It introduces ObjaNeRF-Text, a 280K-object NeRF-language dataset built from Objaverse, with text annotations inherited from PointLLM and GPT4Point after replacing point-cloud vocabulary with 'NeRF' terms. The authors evaluate LLaNA-7b/13b against image-based (LLaVA, BLIP-2), point-cloud (PointLLM, GPT4Point), and mesh/multi-view (3D-LLM) baselines on brief and detailed captioning, single-round Q&A, and zero-shot classification, including a from-scratch ablation on ShapeNeRF-Text. The main claim is that directly processing NeRF weights leads to better performance on NeRF-language tasks than approaches that use 2D or 3D representations derived from NeRFs.

Significance. If fully established, the central claim would be a notable contribution: it would demonstrate that an MLLM can reason about object semantics directly from NeRF parameters, without rendering or materializing explicit geometry, and ObjaNeRF-Text would be a large-scale resource for future work. The paper has concrete strengths: it retrains several baselines from scratch on the same training split, it includes a language-only control showing that the benchmark cannot be solved by the LLM's priors alone, and it provides a scaling analysis across 7B and 13B models. However, the comparison supporting the headline claim is not yet fully controlled, and the abstract overstates a claim that the paper's own Table VII contradicts.

major comments (3)
  1. [Abstract / Table VII] The abstract states that directly processing NeRF weights leads to better performance on NeRF-Language tasks than approaches relying on 2D or 3D representations derived from NeRFs, but Table VII and the accompanying text in Section V-A.d report that LLaVA-13b multi-view (73.45) outperforms LLaNA-13b (69.27) on zero-shot NeRF classification, and the text explicitly identifies LLaVA-13b as the best model for that task. The blanket claim is therefore contradicted by the paper's own results; it should be qualified to the captioning and Q&A tasks, or revised to state that the advantage holds on most, but not all, evaluated tasks.
  2. [Section V-A / V-B] The main comparison is confounded by information asymmetry. Image baselines receive at most N=3 rendered views (Section V-A), and the paper argues in Section I that low resolution and unfavorable viewpoints disadvantage image- and point-cloud-based models, which is exactly the confound: the derived representations are given less information than the full NeRF weights. The point-cloud extraction method, density, and resolution are never specified. To support the claim that weight-space processing is intrinsically superior, the authors should specify the point-cloud extraction protocol and run ablations that equalize the information budget, for example by rendering many more views at high resolution for image models and by using dense point clouds of a controlled, representative size.
  3. [Section III / Section V-B] A second confound is encoder adaptation. The nf2vec meta-encoder is pre-trained on the NeRFs of ShapeNeRF-Text and ObjaNeRF-Text with an image-reconstruction objective (Section III), while the official protocols for LLaVA, PointLLM, and GPT4Point keep their modality encoders frozen (Section V-B). LLaNA's encoder is therefore domain-adapted to the exact test distribution, and the performance gap may reflect this adaptation rather than an inherent advantage of processing weights. The from-scratch ablation in Section V-B equalizes training annotations but not encoder pre-training; a cleaner comparison would pre-tune the baseline encoders on analogous derived representations from the same training NeRFs, or use an nf2vec pre-trained on a disjoint distribution.
minor comments (4)
  1. [Table IV / Table XII] Table IV lists two rows labeled 'LLaVA-vicuna-7b Image (RV)' with different scores, and Table XII lists two rows labeled 'LLaVA-vicuna-13b Image (FV)'; one of each is presumably a different condition such as multi-view, and the labels should be corrected.
  2. [Section IV-A] The ObjaNeRF-Text annotations are original point-cloud captions with words like 'point cloud' converted to 'NeRF'; the semantic validity of the modified captions for the NeRF modality is not assessed, and the paper should report whether any human validation or a quantitative analysis of the replacement's impact was performed.
  3. [Tables I-XIII] All results are single point estimates with no error bars, significance tests, or multiple runs, and some reported margins are small (e.g., Table III S-BERT 42.08 vs 41.01; Table IV 44.26 vs 44.15). Adding uncertainty estimates would clarify which differences are meaningful.
  4. [Section V-B] The statement 'we followed their protocol, which, for all methods, keeps the modality-specific encoder frozen and trains an adaptor' is ambiguous for LLaNA, since its nf2vec encoder is already pre-trained and then frozen; please clarify whether any LLaNA variant was trained from a randomly initialized meta-encoder.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical benchmark comparison, not a derivation from fitted inputs or from a self-citation chain.

full rationale

The paper does not contain a derivation chain in which an output is equivalent to an input by construction. The main claim, that directly processing NeRF weights outperforms image- or point-cloud-based baselines on NeRF-language tasks, is supported by supervised training on held-out test splits with metrics computed against external or externally sourced text annotations. The meta-encoder nf2vec and the earlier LLaNA paper are self-citations, but they are used as disclosed building blocks and training components, not as authority for the empirical outcome; the evaluation itself is independent of those citations. The dataset annotations in ObjaNeRF-Text are reused from PointLLM and GPT4Point with 'point cloud' wording replaced by 'NeRF', which is a benchmark-construction validity concern but not a circular reduction of the paper's result to its inputs. Similarly, the information-asymmetry caveats (image baselines receive at most three rendered views, point-cloud extraction is unspecified, and LLaNA's encoder is pre-trained on the target NeRF distribution) concern experimental fairness and confound control, not circularity. The paper even runs a language-only control showing that its benchmarks require at least some object information, and it transparently reports Table VII where LLaVA-13b multi-view beats LLaNA on zero-shot classification, which undercuts the abstract's blanket superiority statement but again is an overclaim rather than a circular step. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation; therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claim does not rest on a fitted physical parameter; it rests on the nf2vec encoder, the dataset construction, and the fairness of the baseline comparison. Listed are the hand-chosen design constants and domain assumptions that the comparison hinges on.

free parameters (4)
  • Meta-encoder embedding dimension G = 1024
    The global NeRF embedding length is set to 1024 (Section III). This is a hand-chosen bottleneck between the NeRF and the LLM; performance could depend on it.
  • Multi-view baseline count N = 3
    Image baselines receive N=3 rendered views because the model cannot process more (Section V). This choice caps the information available to image baselines and is part of the comparison.
  • NeRF architecture L and H = L=3 hidden layers, H=64 neurons
    The MLP-only NeRF architecture is inherited from nf2vec (Section III). The central claim of direct weight processing is tested only on this fixed architecture.
  • Frequency encoding count = 24
    Input coordinates use a 24-frequency positional encoding before the first layer (Section III), a design choice from the original NeRF formulation.
assumptions (3)
  • domain assumption nf2vec embeddings preserve object shape and appearance information needed for language tasks
    The frozen meta-encoder is pre-trained only for image reconstruction (self-training protocol of nf2vec), yet is assumed to transfer to captioning and Q&A. Invoked in Section III (Meta-encoder).
  • ad hoc to paper Text annotations collected for point clouds remain valid for NeRFs after string replacement
    Section IV-A: 'the words of the original text annotations referring to the point cloud data structure... have been modified into NeRF'. This assumes the captions and Q&A are representation-independent.
  • standard math Standard LLM and NeRF training procedures behave as expected
    The paper relies on the usual volumetric rendering equation and transformer auto-regressive training without proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training." pith.science (2026). https://pith.science/paper/R22W77W6

@misc{pith2026250413995,
  author       = {Pith},
  title        = {Pith review of: Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R22W77W6}},
  note         = {Machine review of arXiv:2504.13995}
}
read the original abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities in understanding both images and 3D data, yet these modalities face inherent limitations in comprehensively representing object geometry and appearance. Neural Radiance Fields (NeRFs) have emerged as a promising alternative, encoding both geometric and photorealistic properties within the weights of a simple Multi-Layer Perceptron (MLP). This work investigates the feasibility and effectiveness of ingesting NeRFs into an MLLM. We introduce LLaNA, the first MLLM able to perform new tasks such as NeRF captioning and Q\&A, by directly processing the weights of a NeRF's MLP. Notably, LLaNA is able to extract information about the represented objects without the need to render images or materialize 3D data structures. In addition, we build the first large-scale NeRF-language dataset, composed by more than 300K NeRFs trained on ShapeNet and Objaverse, with paired textual annotations that enable various NeRF-language tasks. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that directly processing NeRF weights leads to better performance on NeRF-Language tasks compared to approaches that rely on either 2D or 3D representations derived from NeRFs.

Figures

Figures reproduced from arXiv: 2504.13995 by the authors.

Figure 1
Figure 1. LLaNA. A new Multimodal Large Language Model that understands and reasons on an input NeRF. Notably, our framework processes directly the NeRF weights and performs tasks such as captioning, Q&A, and zero-shot classification of NeRFs. traditional approaches that process NeRFs by first converting them to explicit data representations – either rendered images or 3D point clouds – and then using existing MLLMs designed … view at source ↗
Figure 2
Figure 2. Framework overview. Example of NeRF captioning. RGB color values and density estimates. These values are then integrated along camera rays using volumetric rendering techniques [13], to produce the final image. Each NeRF is trained for approximately 2000 steps, until it achieves good reconstruction quality as measured by the Peak Signal-to￾Noise Ratio (PSNR). Meta-encoder In this work, we investigate how to design a… view at source ↗
Figure 4
Figure 4. Automatic annotation pipeline. Given a 3D model, N views are rendered and processed by a VLM (LLaVA) to generate view-specific captions. These are aggregated by an LLM (LLaMA) for final descriptions and Q&A. and appearance. Detailed descriptions are longer sentences that describe all the details of the object. The single-round Q&As consist of a question about the object and the corresponding ground-truth answer, whi… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Qualitative results on ShapeNeRF–Text brief descriptions. PointLLM-13b 3D-LLM LLaVA-13b BLIP-2 FlanT5-xxl How would you summarize this NeRF? 3D model of a cartoon cow with wings. How would you summarize this point cloud? This is the representation of a cuddly teddy bea…
Figure 6
Figure 6. Figure 6: Qualitative results on ObjaNeRF–Text brief descriptions (PointLLM test set). and ObjaNeRF-Text, we employ the official code and pre￾trained models released by the respective authors1 . A. Experiments on ObjaNeRF-Text and ShapeNeRF-Text Tables I, II, V, VI and VII show …
Figure 7
Figure 7. Figure 7: Qualitative results on ObjaNeRF–Text brief descriptions (GPT4Point test set). TABLE II: NeRF brief captioning on the HST dataset. Best results are in bold, runner-up is underlined. (FV: front-view, BV: back-view, MV: multi-view) Model Modality S-BERT SimCSE BLEU-1 ROUG…
Figure 8
Figure 8. Figure 8: Qualitative results on ShapeNeRF–Text detailed descriptions. From top to bottom: brief and detailed descriptions, single-round Q&A TABLE V: NeRF detailed captioning on ShapeNeRF–Text. Best results are in bold, runner-up is underlined. (FV: front-view, BV: back-view, MV…
Figure 9
Figure 9. Figure 9: Qualitative results on ShapeNeRF–Text single-round Q&A. NeRF This model can be used for videogame nature-themed graphics. What could be a potential use for the model, given its detailed design? The model features its outstretched arms and short legs. What distinct phys…
Figure 10
Figure 10. Figure 10: NeRF multi-round Q&A example from ObjaNeRF–Text. adapt its pre-trained language understanding capabilities to the specific characteristics and vocabulary of NeRF-based object descriptions. c) single-round Q&A: In the single-round Q&A experi￾ment, we test the ability o…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 46 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2

    2019

  2. [2]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  3. [3]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  4. [4]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  5. [5]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” in International Conference on Machine Learning. PMLR, 2023, pp. 8469–8488

    2023

  6. [6]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention,

    R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y . Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” arXiv preprint arXiv:2303.16199 , 2023

    arXiv 2023

  7. [7]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

    2024

  8. [8]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” Advances in Neural Informa- tion Processing Systems , vol. 36, 2024

    2024

Show all 85 references
  1. [9]

    Videollm: Modeling video sequence with large language models,

    G. Chen, Y .-D. Zheng, J. Wang, J. Xu, Y . Huang, J. Pan, Y . Wang, Y . Wang, Y . Qiao, T. Lu et al. , “Videollm: Modeling video sequence with large language models,” arXiv preprint arXiv:2305.13292 , 2023

    2023 arXiv

  2. [10]

    Pointllm: Empowering large language models to understand point clouds,

    R. Xu, X. Wang, T. Wang, Y . Chen, J. Pang, and D. Lin, “Pointllm: Empowering large language models to understand point clouds,” arXiv preprint arXiv:2308.16911, 2023

    2023 arXiv

  3. [11]

    Gpt4point: A unified framework for point-language understanding and generation,

    Z. Qi, Y . Fang, Z. Sun, X. Wu, T. Wu, J. Wang, D. Lin, and H. Zhao, “Gpt4point: A unified framework for point-language understanding and generation,” in CVPR, 2024

    2024

  4. [12]

    3d-LLM: Injecting the 3d world into large language models,

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan, “3d-LLM: Injecting the 3d world into large language models,” in Thirty- seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=YQA28p7qNz 13

    2023

  5. [13]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European conference on computer vision. Springer, 2020, pp. 405–421

    2020

  6. [14]

    Nerf-rpn: A general framework for object detection in nerfs,

    B. Hu, J. Huang, Y . Liu, Y .-W. Tai, and C.-K. Tang, “Nerf-rpn: A general framework for object detection in nerfs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 23 528–23 538

    2023

  7. [15]

    Deep learning on 3D neural fields,

    P. Zama Ramirez, L. De Luigi, D. Sirocchi, A. Cardace, R. Spezialetti, F. Ballerini, S. Salti, and L. Di Stefano, “Deep learning on 3D neural fields,” arXiv preprint arXiv:2312.13277 , 2023

    2023 arXiv

  8. [16]

    Graph metanetworks for processing diverse neural architectures,

    D. Lim, H. Maron, M. T. Law, J. Lorraine, and J. Lucas, “Graph metanetworks for processing diverse neural architectures,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=ijK5hyxs0n

    2024

  9. [17]

    Llana: Large language and nerf assistant,

    A. Amaduzzi, P. Z. Ramirez, G. Lisanti, S. Salti, and L. Di Ste- fano, “Llana: Large language and nerf assistant,” arXiv preprint arXiv:2406.11840, 2024

    2024 arXiv

  10. [18]

    Looking at words and points with attention: a benchmark for text-to-shape coherence,

    A. Amaduzzi, G. Lisanti, S. Salti, and L. Di Stefano, “Looking at words and points with attention: a benchmark for text-to-shape coherence,” in 2023 IEEE/CVF International Conference on Computer Vision Work- shops (ICCVW). IEEE Computer Society, 2023, pp. 2860–2869

    2023

  11. [19]

    Objaverse: A universe of annotated 3d objects,

    M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi, “Objaverse: A universe of annotated 3d objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 13 142–13 153

    2023

  12. [20]

    Llama-adapter v2: Parameter-efficient visual instruction model,

    P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al. , “Llama-adapter v2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010 , 2023

    2023 arXiv

  13. [21]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180–15 190

    2023

  14. [22]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y . Wu, Z. Hong, J. Huang, J. Liu et al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 21, 2024, pp. 23 802–23 804

    2024

  15. [23]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424 , 2023

    2023 arXiv

  16. [24]

    Language is not all you need: Aligning perception with language models,

    S. Huang, L. Dong, W. Wang, Y . Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra et al. , “Language is not all you need: Aligning perception with language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  17. [25]

    Kosmos-2: Grounding multimodal large language models to the world,

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, and F. Wei, “Kosmos-2: Grounding multimodal large language models to the world,” arXiv preprint arXiv:2306.14824 , 2023

    2023 arXiv

  18. [26]

    Mimic-it: Multi-modal in-context instruction tuning,

    B. Li, Y . Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, and Z. Liu, “Mimic-it: Multi-modal in-context instruction tuning,” arXiv preprint arXiv:2306.05425, 2023

    2023 arXiv

  19. [27]

    Qwen-vl: A frontier large vision-language model with versatile abilities,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966 , 2023

    2023 arXiv

  20. [28]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  21. [29]

    3d-vista: Pre-trained transformer for 3d vision and text alignment,

    Z. Zhu, X. Ma, Y . Chen, Z. Deng, S. Huang, and Q. Li, “3d-vista: Pre-trained transformer for 3d vision and text alignment,” ICCV, 2023

    2023

  22. [30]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following,

    Z. Guo, R. Zhang, X. Zhu, Y . Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li et al. , “Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following,” arXiv preprint arXiv:2309.00615 , 2023

    2023 arXiv

  23. [31]

    3d con- cept learning and reasoning from multi-view images,

    Y . Hong, C. Lin, Y . Du, Z. Chen, J. B. Tenenbaum, and C. Gan, “3d con- cept learning and reasoning from multi-view images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 9202–9212

    2023

  24. [32]

    Nerf in the wild: Neural radiance fields for unconstrained photo collections,

    R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Doso- vitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7210–7219

    2021

  25. [33]

    Dreamfusion: Text- to-3d using 2d diffusion,

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text- to-3d using 2d diffusion,” in The Eleventh International Conference on Learning Representations, 2022

    2022

  26. [34]

    Nerf-supervision: Learning dense object descriptors from neural radiance fields,

    L. Yen-Chen, P. Florence, J. T. Barron, T.-Y . Lin, A. Rodriguez, and P. Isola, “Nerf-supervision: Learning dense object descriptors from neural radiance fields,” in 2022 international conference on robotics and automation (ICRA) . IEEE, 2022, pp. 6496–6503

    2022

  27. [35]

    Tensorf: Tensorial radiance fields,

    A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in European Conference on Computer Vision (ECCV) , 2022

    2022

  28. [36]

    Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction,

    C. Sun, M. Sun, and H.-T. Chen, “Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5459–5469

    2022

  29. [37]

    Plenoxels: Radiance fields without neural networks,

    S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2022. [Online]. Available: http://dx.doi.org/ 10.1109/CVPR52688.2022.00542

    2022

  30. [38]

    Instant neural graphics primitives with a multiresolution hash encoding,

    T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph. , vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022. [Online]. Available: https://doi.org/10.1145/3528223.3530127

    2022

  31. [39]

    Ditto-nerf: Diffusion-based iterative text to omni-directional 3d model,

    H. Seo, H. Kim, G. Kim, and S. Y . Chun, “Ditto-nerf: Diffusion-based iterative text to omni-directional 3d model,” 2023

    2023

  32. [40]

    Latent-nerf for shape-guided generation of 3d shapes and textures,

    G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or, “Latent-nerf for shape-guided generation of 3d shapes and textures,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 12 663–12 673

    2023

  33. [41]

    Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis,

    K. Jo, G. Shim, S. Jung, S. Yang, and J. Choo, “Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2023, pp. 724–733

    2023

  34. [42]

    Hyp-nerf: Learning improved nerf priors using a hypernetwork,

    B. Sen, G. Singh, A. Agarwal, R. Agaram, M. Krishna, and S. Sridhar, “Hyp-nerf: Learning improved nerf priors using a hypernetwork,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curra...

    2023

  35. [43]

    Instructpix2neRF: Instructed 3d portrait editing from a single image,

    J. Li, S. Liu, Z. Liu, Y . Wang, K. Zheng, J. Xu, J. Li, and J. Zhu, “Instructpix2neRF: Instructed 3d portrait editing from a single image,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=XIxhINXtQk

    2024

  36. [44]

    Understanding pure clip guidance for voxel grid nerf models,

    H.-H. Lee and A. X. Chang, “Understanding pure clip guidance for voxel grid nerf models,” 2022

    2022

  37. [45]

    Clip-nerf: Text-and- image driven manipulation of neural radiance fields,

    C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Clip-nerf: Text-and- image driven manipulation of neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3835–3844

    2022

  38. [46]

    Faceclipnerf: Text-driven 3d face manipulation using deformable neural radiance fields,

    S. Hwang, J. Hyung, D. Kim, M.-J. Kim, and J. Choo, “Faceclipnerf: Text-driven 3d face manipulation using deformable neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 3469–3479

    2023

  39. [47]

    Blending-nerf: Text- driven localized editing in neural radiance fields,

    H. Song, S. Choi, H. Do, C. Lee, and T. Kim, “Blending-nerf: Text- driven localized editing in neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 14 383–14 393

    2023

  40. [48]

    Nerf-art: Text-driven neural radiance fields stylization,

    C. Wang, R. Jiang, M. Chai, M. He, D. Chen, and J. Liao, “Nerf-art: Text-driven neural radiance fields stylization,” IEEE Transactions on Visualization and Computer Graphics , pp. 1–15, 2023

    2023

  41. [49]

    Nerfeditor: Differentiable style decomposition for 3d scene editing,

    C. Sun, Y . Liu, J. Han, and S. Gould, “Nerfeditor: Differentiable style decomposition for 3d scene editing,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024, pp. 7306–7315

    2024

  42. [50]

    Instruct-nerf2nerf: Editing 3d scenes with instructions,

    A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa, “Instruct-nerf2nerf: Editing 3d scenes with instructions,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 19 740–19 750

    2023

  43. [51]

    Morphnerf: Text-guided 3d-aware editing via morphing generative neural radiance fields,

    Y . Yu, R. Wu, Y . Men, S. Lu, M. Cui, X. Xie, and C. Miao, “Morphnerf: Text-guided 3d-aware editing via morphing generative neural radiance fields,” IEEE Transactions on Multimedia , pp. 1–13, 2024

    2024

  44. [52]

    Dreameditor: Text- driven 3d scene editing with neural fields,

    J. Zhuang, C. Wang, L. Lin, L. Liu, and G. Li, “Dreameditor: Text- driven 3d scene editing with neural fields,” in SIGGRAPH Asia 2023 Conference Papers, 2023, pp. 1–10

    2023

  45. [53]

    Componerf: Text-guided multi-object compositional nerf with editable 3d scene layout,

    H. Bai, Y . Lyu, L. Jiang, S. Li, H. Lu, X. Lin, and L. Wang, “Componerf: Text-guided multi-object compositional nerf with editable 3d scene layout,” arXiv preprint arXiv:2303.13843 , 2023

    2023 arXiv

  46. [54]

    Reference-guided 14 controllable inpainting of neural radiance fields,

    A. Mirzaei, T. Aumentado-Armstrong, M. A. Brubaker, J. Kelly, A. Levinshtein, K. G. Derpanis, and I. Gilitschenski, “Reference-guided 14 controllable inpainting of neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , Octo...

    2023

  47. [55]

    Lerf: Language embedded radiance fields,

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “Lerf: Language embedded radiance fields,” in International Conference on Computer Vision (ICCV) , 2023

    2023

  48. [56]

    Decomposing nerf for editing via feature field distillation,

    S. Kobayashi, E. Matsumoto, and V . Sitzmann, “Decomposing nerf for editing via feature field distillation,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, ...

    2022

  49. [57]

    Connecting nerfs, images, and text,

    F. Ballerini, P. Zama Ramirez, R. Mirabella, S. Salti, and L. Di Stefano, “Connecting nerfs, images, and text,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, June 2024

    2024

  50. [58]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763

    2021

  51. [59]

    Predicting neural network accuracy from weights,

    T. Unterthiner, D. Keysers, S. Gelly, O. Bousquet, and I. O. Tol- stikhin, “Predicting neural network accuracy from weights,” arXiv, vol. abs/2002.11448, 2020

    2002 arXiv

  52. [60]

    Self-supervised representation learning on neural network weights for model characteristic prediction,

    K. Sch ¨urholt, D. Kostadinov, and D. Borth, “Self-supervised representation learning on neural network weights for model characteristic prediction,” in Advances in Neural Information Processing Systems , A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021. [O...

    2021

  53. [61]

    Parameter prediction for unseen deep architectures,

    B. Knyazev, M. Drozdzal, G. W. Taylor, and A. Romero, “Parameter prediction for unseen deep architectures,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021. [Online]. Available: https://openreview. net/...

    2021

  54. [62]

    Generating adversarial examples with graph neural networks,

    F. Jaeckle and M. P. Kumar, “Generating adversarial examples with graph neural networks,” in Uncertainty in Artificial Intelligence. PMLR, 2021, pp. 1556–1564

    2021

  55. [63]

    Neural network branching for neural network verification,

    J. Lu and M. P. Kumar, “Neural network branching for neural network verification,” in International Conference on Learning Representations ,

  56. [64]

    From data to functa: Your data point is a function and you can treat it like one,

    E. Dupont, H. Kim, S. A. Eslami, D. J. Rezende, and D. Rosenbaum, “From data to functa: Your data point is a function and you can treat it like one,” in International Conference on Machine Learning . PMLR, 2022, pp. 5694–5725

    2022

  57. [65]

    Deep learning on implicit neural representations of shapes,

    L. De Luigi, A. Cardace, R. Spezialetti, P. Zama Ramirez, S. Salti, and L. Di Stefano, “Deep learning on implicit neural representations of shapes,” in International Conference on Learning Representations (ICLR), 2023

    2023

  58. [66]

    Neural processing of tri-plane hybrid neural fields,

    A. Cardace, P. Z. Ramirez, F. Ballerini, A. Zhou, S. Salti, and L. di Ste- fano, “Neural processing of tri-plane hybrid neural fields,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=zRkM6UcA22

    2024

  59. [67]

    Equivariant architectures for learning in deep weight spaces,

    A. Navon, A. Shamsian, I. Achituve, E. Fetaya, G. Chechik, and H. Maron, “Equivariant architectures for learning in deep weight spaces,” in International Conference on Machine Learning , 2023

    2023

  60. [68]

    Neural functional transformers,

    A. Zhou, K. Yang, Y . Jiang, K. Burns, W. Xu, S. Sokota, J. Z. Kolter, and C. Finn, “Neural functional transformers,” Advances in neural information processing systems , vol. 37, 2023

    2023

  61. [69]

    Permutation equivariant neural functionals,

    A. Zhou, K. Yang, K. Burns, A. Cardace, Y . Jiang, S. Sokota, J. Z. Kolter, and C. Finn, “Permutation equivariant neural functionals,” Advances in neural information processing systems , vol. 37, 2023

    2023

  62. [70]

    Universal neural functionals,

    A. Zhou, C. Finn, and J. Harrison, “Universal neural functionals,” arXiv preprint arXiv:2402.05232, 2024

    2024 arXiv

  63. [71]

    On the algebraic structure of feedforward network weight spaces,

    R. Hecht-Nielsen, “On the algebraic structure of feedforward network weight spaces,” in Advanced Neural Computers . Elsevier, 1990, pp. 129–135

    1990

  64. [72]

    Graph neural networks for learning equivariant representations of neural networks,

    M. Kofinas, B. Knyazev, Y . Zhang, Y . Chen, G. J. Burghouts, E. Gavves, C. G. Snoek, and D. W. Zhang, “Graph neural networks for learning equivariant representations of neural networks,” in The Twelfth Interna- tional Conference on Learning Representations , 2024

    2024

  65. [73]

    Fast r-cnn,

    R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440–1448

    2015

  66. [74]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . pmlr, 2015, pp. 448–456

    2015

  67. [75]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett...

    2017

  68. [76]

    Shapenet: An information- rich 3d model repository,

    A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su et al., “Shapenet: An information- rich 3d model repository,” arXiv preprint arXiv:1512.03012 , 2015

    2015 arXiv

  69. [77]

    High-fidelity 3d textured shapes generation by sparse encoding and adversarial decoding,

    Q. Zuo, X. Gu, Y . Dong, Z. Zhao, W. Yuan, L. Qiu, L. Bo, and Z. Dong, “High-fidelity 3d textured shapes generation by sparse encoding and adversarial decoding,” in European Conference on Computer Vision , 2024

    2024

  70. [78]

    Scalable 3d captioning with pretrained models,

    T. Luo, C. Rockwell, H. Lee, and J. Johnson, “Scalable 3d captioning with pretrained models,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  71. [79]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084 , 2019

    1908 arXiv

  72. [80]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” in 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 . Association for Computational Linguistics (ACL), 2021, pp. 6894–6910

    2021

  73. [81]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  74. [82]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  75. [83]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  76. [84]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023. Andrea Amaduzzi is a fourth-year PhD student at the Computer Vision Laboratory (CVLAB), Uni...

    2023 arXiv

  77. [2020]

    Available: https://openreview.net/forum?id=B1evfa4tPB

    [Online]. Available: https://openreview.net/forum?id=B1evfa4tPB

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.