Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Better captions and a mixture-of-experts perceiver lift an RS multimodal LLM past its predecessor and GPT-4o-mini.

desk verdict A solid extension of the authors' own LHRS-Bot with genuinely useful new data and a believable MoE perceiver, but the LHRS-Bench edge over closed models needs a decontamination check before you trust it. read the letter →

arxiv 2411.09301 v1 pith:DK7RJLXC submitted 2024-11-14 cs.CV

Zhenshi Li , Dilxat Muhtar , Feng Gu , Xueliang Zhang , Pengfeng Xiao , Guangjun He , Xiaoxiang Zhu This is my paper · ORCID
classification cs.CV
keywords remotesensingmultimodallargelanguagemodelvision-languagealignmentimagecaptioningmixtureofexpertsvisualgroundingsceneclassificationinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LHRS-Bot-Nova claims that the main bottleneck in remote-sensing multimodal large language models is the captions used for vision-language alignment, not the language backbone. The authors recaption the LHRS-Align dataset with Share-Captioner, a vision-capable captioner that sees the image together with its OpenStreetMap features, producing LHRS-Align-Recap, whose captions are roughly twice as vocabulary-rich and score 88.12 versus 70.81 on average LongCLIP alignment. They add a spatial-reasoning instruction set and a mixture-of-experts vision perceiver, then train LHRS-Bot-Nova, which reaches 87.36% average visual grounding accuracy versus 80.78% for LHRS-Bot and 34.93% overall on LHRS-Bench versus 30.00% for GPT-4o-mini. If the claim holds, synthetic feature-guided recaptioning plus sparse expert routing is a cheap route to better specialized vision-language models.

What carries the argument

The load-bearing mechanism is two-part. The first is LHRS-Align-Recap, generated by prompting Share-Captioner with each remote-sensing image and its OpenStreetMap key-value features; the resulting captions average 150 words versus 31, contain nearly twice the unique words and trigrams of LHRS-Align, and reach an average LongCLIP score of 88.12 versus 70.81. The second is the MoE vision perceiver, which replaces each feed-forward layer of the original perceiver with four experts and routes each summarized vision token through its top-2 experts, expanding the perceiver's memory for visual detail while keeping inference activation roughly constant. Learnable queries with decreasing allocation (112, 96, 64) compress multi-level SigLIP features from layers $\{N_L/3, 2N_L/3, N_L-1\}$, and the concatenated tokens feed LLaMA3-8B through LoRA adapters. The evaluation protocol that carries the LHRS-Bench claim is strict letter-only matching combined with CircularEval option rotation.

What would settle it

Have remote-sensing experts check a random sample of LHRS-Align-Recap captions against the images and OSM tags; if a substantial fraction contain invented objects or mismatched attributes, the pretraining gain would come from caption style rather than factual alignment, falsifying the claim that feature-guided recaptioning is what improves the model.

Watch

Extended reading notes

Core claim

The paper's central discovery is that caption quality, not model size, is the binding constraint for remote-sensing MLLMs, and that captions generated by a multimodal model that can see the image while being prompted with OSM key-value features align better with imagery than captions written by a blind LLM from text tags alone. Trained in the same three-stage curriculum as LHRS-Bot, LHRS-Bot-Nova improves average scene classification from 71.83% to 76.60%, visual grounding from 80.78% to 87.36%, and LHRS-Bench overall to 34.93%, edging out GPT-4o-mini by 4.93 points. Ablations show the recaptioned pretraining data contributes the largest gain, including a 20.39-point jump in classification accuracy over pretraining with the original LHRS-Align, while the MoE vision perceiver adds up to 6.24 points in grounding and 4.47 points in LHRS-Bench. The paper also proposes a strict letter-only MCQ evaluation combined with CircularEval option rotation, which lowers reported accuracies by around 20% relative to lax matching and thereby gives a more conservative measure of RS instruction-following ability.

Load-bearing premise

The synthetic captions produced by Share-Captioner are factually faithful to what is actually visible in each image and to its OpenStreetMap feature tags, so pretraining on them improves vision-language alignment rather than teaching the model to repeat invented geographic details.

Editorial extensions

If this is right

  • LHRS-Align-Recap can serve as a drop-in replacement for LHRS-Align in pretraining other RS MLLMs, since the ablation shows consistent gains across classification, VQA, grounding, and LHRS-Bench when the recaptioned dataset is used alone.
  • MoE vision perceivers yield better visual grounding than vanilla perceivers at similar inference cost, so the design should transfer to other dense-perception multimodal models beyond remote sensing.
  • Strict letter-only scoring with CircularEval lowers MCQ accuracies by around 20% relative to lax matching, so future RS MLLM comparisons should report this protocol to avoid inflated numbers.
  • The combination of LRV-Instruct negative samples gives the model the ability to refuse misleading instructions, which is a step toward more reliable RS assistants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same feature-guided recaptioning recipe could extend to other geospatial modalities such as SAR, multispectral, or time-series imagery, as long as a vision-capable captioner can consume those inputs and OSM-style ancillary features are available.
  • If caption faithfulness is the real driver, then using a small human-validated subset to filter the highest-quality synthetic captions could concentrate training signal and potentially exceed the reported gains.
  • The strict letter-only protocol may over-penalize models that answer correctly but add explanation; pairing it with open-ended generation evaluation would give a more complete measure of RS MLLM capability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LHRS-Bot-Nova, a remote-sensing MLLM built from SigLIP-L/14 at 336x336 resolution, a query-based vision perceiver with a top-2 MoE FFN, and LLaMA3-8B fine-tuned with LoRA. The authors contribute a recaptioned pretraining corpus (LHRS-Align-Recap), a spatial-instruction dataset (LHRS-Instruct-Plus), and the integration of LRV-Instruct as a negative-sample robustness dataset. Training follows the three-stage curriculum of the earlier LHRS-Bot. Evaluation covers seven scene-classification datasets, RSVQA-HR/LR, RSVG/DIOR-RSVG visual grounding, and the LHRS-Bench MCQ benchmark. Headline results are 76.60% vs 71.83% average classification over LHRS-Bot, 87.36% vs 80.78% average visual grounding, and 34.93% vs 30.00% for GPT-4o-mini on LHRS-Bench. The ablations attribute most of the gains to recaptioning and to the MoE perceiver.

Significance. If the claims hold, the paper is a useful engineering contribution: the MoE vision perceiver increases capacity without additional inference activation, LHRS-Align-Recap addresses a real caption-quality bottleneck in RS vision-language pretraining, and the CircularEval-based MCQ protocol is a sensible evaluation practice. The headline ablations are matched in data and hyperparameters, which is the right experimental design. However, the central generalization claims currently rest on unverified disjointness between training data and LHRS-Bench, and the single-run point estimates do not support the word 'significant' as used in the text. The paper's lasting value depends on the dataset release and on the decontamination and caption-fidelity checks requested below.

major comments (4)
  1. [Section 5.3, Table 9; training data in Table 3] The central claim that LHRS-Bot-Nova outperforms GPT-4o-mini on LHRS-Bench (34.93% vs 30.00%) presupposes that LHRS-Bench is a held-out benchmark for this model. The paper never establishes this. LHRS-Bench was introduced in the authors' earlier LHRS-Bot paper, and Table 3 shows that LHRS-Instruct, used in Stage 3, contains 29,671 LHRS-Align instances with instruction types 'Conversation, Detailed description, Visual reasoning'. If LHRS-Bench questions or images derive from the same LHRS-Align/OSM caption pool, the comparison against closed models that never trained on that pool is not a valid generalization test. Please provide an image-overlap and question/answer-source disjointness analysis between all training stages and LHRS-Bench, and report LHRS-Bench accuracy separately for items that do and do not overlap with the training data.
  2. [Section 5.2, Table 6; Table 3] The average classification accuracy (76.60% vs 71.83%) and the surrounding 'significant improvement' language conflate zero-shot and trained-on datasets. Table 3 lists NWPU (111,755 instances), fMoW (5,352), and METER-ML (1,400) as instruction-tuning data, and these are exactly the non-zero-shot columns in Table 6. Only AID, WHU-RS19, SIRI-WHU, and EuroSAT are zero-shot, and on AID LHRS-Bot-Nova actually drops relative to LHRS-Bot (91.26% to 88.32%). Please report the zero-shot average separately and state explicitly which subsets drive the overall gain.
  3. [Section 3.1, Tables 2 and 10] The quality of LHRS-Align-Recap is assessed only through vocabulary counts, sentence lengths, and LongCLIP scores, but the data-quality claim is load-bearing: the Table 10 ablation attributes a 20.39 percentage-point classification gain and a 13.91 percentage-point LHRS-Bench gain to this dataset. A CLIP-style score cannot detect factual errors or hallucinations introduced by Share-Captioner, especially in the OSM feature tags that are fed to the captioner. Please add a human-annotated factuality sample, or at least an LLM-based consistency check against image content and OSM tags, and report caption-error statistics.
  4. [Tables 6, 8, 10, and 11] All reported accuracies are single-run point estimates without error bars, confidence intervals, or significance tests. Differences of a few percentage points are described as 'significant' (e.g., the 0.05% overall VQA gain over LHRS-Bot in Table 7 is not interpretable without variance estimates). At minimum, report multiple seeds for the central ablations in Tables 10 and 11, or bootstrap confidence intervals computed over evaluation items, and adjust the causal language accordingly.
minor comments (5)
  1. [Section 4.2] The pre-training description refers to 'LHRS-Bot-Recap'; the dataset name used elsewhere is LHRS-Align-Recap.
  2. [Table 2] The numeric formatting in Table 2 is garbled, e.g., '1 .12106 31'; please correct the alignment of the unique-word and unique-trigram columns.
  3. [Section 5.3, footnote] The footnote states that gpt-4o-mini required one-shot prompting to follow the letter-only instruction; this should be discussed as a limitation, since it makes the comparison with other models less uniform.
  4. [Figure 4] The MoE module is drawn as repeated FFN blocks without clear labels for the router and top-k path; adding a legend or labels would make the architecture self-contained.
  5. [Table 3 and Section 3.2] Dataset names are used inconsistently, including 'F AIR1M' in Table 3 and 'LR V-Instruct' vs 'LRV-Instruct' in the text; please make the naming uniform.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: main improvements are supported by external zero-shot benchmarks and one-factor ablations; self-built LHRS-Bench and trained-on classification sets pose benchmark-hygiene risks but do not make the derivation self-referential.

full rationale

The paper's derivation chain is empirical and controlled rather than definitional: LHRS-Align-Recap captions are generated by Share-Captioner from images and OSM features, used for pretraining, and the resulting model is evaluated on external benchmarks (AID, WHU-RS19, SIRI-WHU, EuroSAT zero-shot; RSVQA-HR/LR; RSVG/DIOR-RSVG) plus the authors' own LHRS-Bench. The ablations in Tables 10 and 11 change one factor at a time (recaptioned vs. original captions; MoE vs. vanilla perceiver), so the attributed gains are not equivalent to the inputs by construction. The paper does inherit architecture and curriculum choices from Muhtar et al. (2024), and it evaluates on the same group's LHRS-Bench without reporting a disjointness check; additionally, Table 3 shows fMoW, METER-ML, and NWPU appearing in instruction tuning while Table 6 reports classification accuracies on those benchmarks. These are benchmark-hygiene and comparison-fairness concerns that could inflate specific numbers, and the paper would be stronger with explicit decontamination evidence. However, none of these concerns reduces a claimed result to its own input: the zero-shot gains on four external datasets and the one-factor ablations remain independent evidence, so no circular step meeting the quote-and-reduction standard is present.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central empirical claims rest on the quality of synthetic captions and conversations, evaluation fairness, and standard ML assumptions; no unobserved physical or theoretical entities are introduced. Key hand-set hyperparameters (query allocation, MoE expert count, activated experts, LoRA rank/alpha) are not swept, so the optimality of the reported configuration is assumed.

free parameters (4)
  • Vision perceiver query allocation per level = {112, 96, 64}
    Chosen by hand in Section 5.1; no ablation is reported over query counts, yet this determines how many visual tokens reach the LLM.
  • Number of MoE experts and activated experts = Ne=4, K=2
    Chosen in Section 5.1; no sensitivity analysis is provided, so the claimed benefit of the MoE perceiver is only shown for this one configuration.
  • LoRA rank and alpha = r=128, alpha=256
    Chosen in Section 5.1 as a high-rank LoRA setting; no comparison with other ranks is reported.
  • Input image resolution for vision encoder = 336x336
    Fixed by the SigLIP-L/14 choice in Section 4.1; larger resolutions are motivated by prior work but not ablated here.
assumptions (6)
  • domain assumption OSM key-value features attached to LHRS-Align images are accurate and complete enough to guide caption generation.
    The recaptioning prompt (Table 1) feeds OSM features to Share-Captioner; if the tags are sparse or wrong, the generated captions inherit those errors and the claimed alignment improvement is overstated.
  • domain assumption Share-Captioner produces factually faithful captions of the remote sensing images, i.e., it does not hallucinate objects, attributes, or spatial relations.
    Section 3.1 treats Share-Captioner output as ground truth for pretraining; quality is checked only by vocabulary statistics and LongCLIP scores, not by human or automated factual verification.
  • domain assumption GPT-4V-generated LHRS-Instruct-Plus conversations are spatially and linguistically correct.
    Section 3.2 builds LHRS-Instruct-Plus by prompting GPT-4V with bounding boxes and categories; no quality filter or manual verification is described, and the prompt example in Table 4 contains a duplicated question.
  • domain assumption LongCLIP score is a valid proxy for vision-language alignment quality.
    Section 3.1 uses higher LongCLIP scores as evidence that LHRS-Align-Recap is better aligned; CLIP-style scores can favor longer or simpler captions and do not measure factual accuracy.
  • domain assumption The datasets labeled zero-shot (AID, WHU-RS19, SIRI-WHU, EuroSAT) are absent from all training stages.
    The paper states they are absent from the multi-task instruction data (Section 5.1), but does not show a deduplication check against the pretraining data or the recaptioning pipeline.
  • domain assumption Averaging accuracy over all seven classification datasets fairly compares models even though fMoW and METER-ML appear in the training mixture.
    Table 6 reports one average across zero-shot and trained sets; the comparison against general MLLMs is therefore not apples-to-apples, and the average should be disaggregated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation." pith.science (2026). https://pith.science/paper/DK7RJLXC

@misc{pith2026241109301,
  author       = {Pith},
  title        = {Pith review of: LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DK7RJLXC}},
  note         = {Machine review of arXiv:2411.09301}
}
read the original abstract

Automatically and rapidly understanding Earth's surface is fundamental to our grasp of the living environment and informed decision-making. This underscores the need for a unified system with comprehensive capabilities in analyzing Earth's surface to address a wide range of human needs. The emergence of multimodal large language models (MLLMs) has great potential in boosting the efficiency and convenience of intelligent Earth observation. These models can engage in human-like conversations, serve as unified platforms for understanding images, follow diverse instructions, and provide insightful feedbacks. In this study, we introduce LHRS-Bot-Nova, an MLLM specialized in understanding remote sensing (RS) images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions. LHRS-Bot-Nova features an enhanced vision encoder and a novel bridge layer, enabling efficient visual compression and better language-vision alignment. To further enhance RS-oriented vision-language alignment, we propose a large-scale RS image-caption dataset, generated through feature-guided image recaptioning. Additionally, we introduce an instruction dataset specifically designed to improve spatial recognition abilities. Extensive experiments demonstrate superior performance of LHRS-Bot-Nova across various RS image understanding tasks. We also evaluate different MLLM performances in complex RS perception and instruction following using a complicated multi-choice question evaluation benchmark, providing a reliable guide for future model selection and improvement. Data, code, and models will be available at https://github.com/NJU-LHRS/LHRS-Bot.

Figures

Figures reproduced from arXiv: 2411.09301 by the authors.

Figure 1
Figure 1. Four examples from LHRS-Align and LHRS-Align-Recap. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Caption length (left) and image-caption LongCLIP score (right) distributions for the LHRS-Align [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Six examples of various instruction tasks from the instruction datasets used to train LHRS-Bot [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Architecture of LHRS-Bot-Nova. LHRS-Bot-Nova employs learnable queries in conjunction with [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Four conversation examples between user and LHRS-Bot-Nova. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The loss curve for different image-text pre-training datasets at stage 1 training. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training on multi-tool visual reasoning trajectories (zoom, grounding, lines) with an attention-focused RL objective improves UHR remote-sensing VQA accuracy over single-tool zoom-in and larger base models.

  2. Few-Shot Vision-Language Reasoning for Satellite Imagery via Verifiable Rewards

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A few-shot RLVR method using only rule-based rewards lifts a 2B vision-language model's remote sensing accuracy by double digits, with 128 examples rivaling thousands.

  3. Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Chain-of-Talkers (CoTalk) has annotators sequentially dictate only the missing visual details, and it reports modest gains in annotation speed and caption density over parallel typed annotation.

  4. Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A remote sensing LVLM that augments visual features with retrieved captions and routes them through level-specific experts improves performance on several RS vision-language benchmarks.

  5. A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Features

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A lightweight projection layer trained to align frozen SpectralGPT multispectral features with LLaMA-3 text embeddings markedly improves EuroSAT classification and enables multispectral scene description.

  6. Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes

    cs.LG 2025-04 conditional novelty 4.0 of 10

    The authors argue that spatial reasoning in multimodal LLMs requires dedicated new recipes in data, architecture, and training objectives, and will not emerge from scaling alone.

Reference graph

Works this paper leans on

31 extracted references · 2 canonical work pages · cited by 6 Pith papers

  1. [2]

    arXiv preprint arXiv:2308.12966

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 . Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., Shou, M.Z.,

    arXiv

  2. [6]

    arXiv:2305.06500

    Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500. Ding, J., Xue, N., Xia, G.S., Bai, X., Yang, W., Yang, M.Y., Belongie, S., Luo, J., Datcu, M., Pelillo, M., et al.,

    arXiv

  3. [7]

    arXiv preprint arXiv:2407.21783

    The llama 3 herd of models. arXiv preprint arXiv:2407.21783 . Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., Schmidt, L.,

    arXiv

  4. [8]

    From multimodal llm to human-level ai: Modality, instruction, reasoning, efficiency and beyond, in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries, pp. 1–8. Gadre, S.Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wort...

    work page 2024

  5. [10]

    URL: https://arxiv.org/abs/ 2106.09685, arXiv:2106.09685

    Lora: Low-rank adaptation of large language models. URL: https://arxiv.org/abs/ 2106.09685, arXiv:2106.09685. Hu, Y., Yuan, J., Wen, C., Lu, X., Li, X.,

    arXiv

  6. [11]

    arXiv preprint arXiv:2307.15266

    Rsgpt: A remote sensing vision language model and benchmark. arXiv preprint arXiv:2307.15266 . Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., de las Casas, D., Hanna, E.B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L.R., Saulnier, L., Lachaux, M.A., Stock, P., Subramanian, S., Yang, S., Antoniak...

    arXiv

  7. [12]

    URL: https://arxiv.org/abs/2401.04088, arXiv:2401.04088

    Mixtral of experts. URL: https://arxiv.org/abs/2401.04088, arXiv:2401.04088. Kuckreja, K., Danish, M.S., Naseer, M., Das, A., Khan, S., Khan, F.S.,

    arXiv

  8. [13]

    27831–27840

    Geochat: Grounded large vision-language model for remote sensing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27831–27840. 25 Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y., 2024a. Seed-bench: Bench- marking multimodal large language models, in: Proceedings of the IEEE/CVF Conference on Compu...

    arXiv

Show all 31 references
  1. [15]

    arXiv preprint arXiv:2406.10100

    Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint arXiv:2406.10100 . Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., Johnson, B.A.,

    arXiv

  2. [16]

    arXiv preprint arXiv:2402.02544

    Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXiv:2402.02544 . 27 Myrzakhan, A., Bsharat, S.M., Shen, Z.,

    arXiv

  3. [17]

    arXiv preprint arXiv:2406.07545

    Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545 . Nguyen, T., Gadre, S.Y., Ilharco, G., Oh, S., Schmidt, L.,

    arXiv

  4. [18]

    arXiv preprint arXiv:2403.20213

    H2rsvlm: Towards helpful and honest remote sensing large vision language model. arXiv preprint arXiv:2403.20213 . Qian, Z., Chen, M., Sun, Z., Zhang, F., Xu, Q., Guo, J., Xie, Z., Zhang, Z.,

    arXiv

  5. [19]

    arXiv preprint arXiv:2312.17016

    On the promises and challenges of multimodal foundation models for geographical, environmental, agricultural, and urban planning applications. arXiv preprint arXiv:2312.17016 . Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A....

    arXiv

  6. [20]

    arXiv preprint arXiv:2312.11805

    Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 . Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S.C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, A., Fergus, R., LeCun, Y., Xie, S.,

    arXiv

  7. [21]

    URL:https://arxiv.org/abs/2406

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms. URL:https://arxiv.org/abs/2406. 16860, arXiv:2406.16860. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.,

    arXiv

  8. [22]

    arXiv preprint arXiv:2307.09288

    Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 . Wang, H., Zhao, S., Qiang, Z., Qin, B., Liu, T., 2024a. Beyond the answers: Reviewing 29 the rationality of multiple choice question answering for the evaluation of large language models. ar...

    arXiv

  9. [23]

    arXiv preprint arXiv:2403.15356

    Neural plasticity-inspired foundation model for observing the earth crossing modalities. arXiv preprint arXiv:2403.15356 . Xu, H., Xie, S., Tan, X.E., Huang, P.Y., Howes, R., Sharma, V., Li, S.W., Ghosh, G., Zettle- moyer, L., Feichtenhofer, C.,

  10. [24]

    arXiv preprint arXiv:2309.16671

    Demystifying clip data. arXiv preprint arXiv:2309.16671 . Xu, Q., Shi, Y., Bamber, J.L., Ouyang, C., Zhu, X.X., 2024a. Large-scale flood modeling and forecasting with floodcast. Water Research , 122162. Xu, Z., Feng, C., Shao, R., Ashby, T., Shen, Y., Jin, D., Cheng, Y., Wang,...

    arXiv

  11. [25]

    arXiv preprint arXiv:2306.13549

    A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 . Yuan, Z., Zhang, W., Fu, K., Li, X., Deng, C., Wang, H., Sun, X.,

    arXiv

  12. [26]

    URL: https://arxiv.org/abs/2303.15343, arXiv:2303.15343

    Sigmoid loss for language image pre-training. URL: https://arxiv.org/abs/2303.15343, arXiv:2303.15343. Zhan, Y., Xiong, Z., Yuan, Y.,

    arXiv

  13. [27]

    arXiv preprint arXiv:2401.09712

    Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model. arXiv preprint arXiv:2401.09712 . Zhang, B., Zhang, P., Dong, X., Zang, Y., Wang, J., 2024a. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2...

    arXiv

  14. [28]

    arXiv preprint arXiv:2401.17600

    Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data. arXiv preprint arXiv:2401.17600 . Zhang, L., Yang, L., Zohner, C.M., Crowther, T.W., Li, M., Shen, F., Guo, M., Qin, J., Yao, L., Zhou, C.,

    arXiv

  15. [30]

    arXiv preprint arXiv:2303.18223

    A survey of large language models. arXiv preprint arXiv:2303.18223 . Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.,

    arXiv

  16. [31]

    arXiv preprint arXiv:2406.09385

    Towards vision- language geo-foundation model: A survey. arXiv preprint arXiv:2406.09385 . Zhu, B., Lui, N., Irvin, J., Le, J., Tadwalkar, S., Wang, C., Ouyang, Z., Liu, F.Y., Ng, A.Y., Jackson, R.B., 2022a. Meter-ml: A multi-sensor earth observation benchmark for automated me...

  17. [36]

    Mibench: Evaluating multimodal large language models over multiple images

    26 Liu, H., Zhang, X., Xu, H., Shi, Y., Jiang, C., Yan, M., Zhang, J., Huang, F., Yuan, C., Li, B., et al., 2024c. Mibench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272 . Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao...

    arXiv

  18. [294]

    Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition

    31 Zhang, P., Wang, X.D.B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Ding, S., Zhang, S., Duan, H., Yan, H., et al., 2023a. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 . Zhang, S., Dong,...

    arXiv

  19. [2020]

    Advances in neural information processing systems 33, 1877–1901

    Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901. Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M., 2023a. Minigpt-v2: large language model as a unified inte...

    1901 arXiv

  20. [2021]

    URL: https://arxiv.org/abs/2012.14913, arXiv:2012.14913

    Transformer feed-forward layers are key-value memories. URL: https://arxiv.org/abs/2012.14913, arXiv:2012.14913. 24 Guo, X., Lao, J., Dang, B., Zhang, Y., Yu, L., Ru, L., Zhong, L., Huang, Z., Wu, K., Hu, D., et al.,

    2012 arXiv

  21. [2022]

    arXiv preprint arXiv:2210.09461

    Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 . Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.,

    arXiv

  22. [2023]

    arXiv preprint arXiv:2303.08774

    Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.,

    arXiv

  23. [2024]

    arXiv preprint arXiv:2404.18930

    Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 . Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.