Pith. sign in

REVIEW 20 cited by

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.12570 v1 pith:MJF5YUCT submitted 2024-08-22 cs.CL cs.LG

Jamba Team: Barak Lenz , Alan Arazi , Amir Bergman , Avshalom Manevich , Barak Peleg , Ben Aviram , Chen Almagor , Clara Fridman
show 52 more authors
Dan Padnos Daniel Gissin Daniel Jannai Dor Muhlgay Dor Zimberg Edden M Gerber Elad Dolev Eran Krakovsky Erez Safahi Erez Schwartz Gal Cohen Gal Shachaf Haim Rozenblum Hofit Bata Ido Blass Inbal Magar Itay Dalmedigos Jhonathan Osin Julie Fadlon Maria Rozman Matan Danos Michael Gokhman Mor Zusman Naama Gidron Nir Ratner Noam Gat Noam Rozen Oded Fried Ohad Leshno Omer Antverg Omri Abend Opher Lieber Or Dagan Orit Cohavi Raz Alon Ro'i Belson Roi Cohen Rom Gilad Roman Glozman Shahar Lev Shaked Meirom Tal Delbari Tal Ness Tomer Asida Tom Ben Gal Tom Braude Uriya Pumerantz Yehoshua Cohen Yonatan Belinkov Yuval Globerson Yuval Peleg Levy Yoav Shoham
This is my paper · ORCID
classification cs.CLcs.LG
keywords modelsjamba-1jambalargemodelactivearchitecturebenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Marconi: Prefix Caching for the Era of Hybrid LLMs

    cs.DC 2024-11 conditional novelty 7.0 of 10

    A prefix cache for hybrid attention-SSM LLMs that selectively checkpoints SSM states and evicts by FLOP efficiency, improving token hit rates and time-to-first-token.

  2. Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

    cs.AI 2026-03 conditional novelty 6.0 of 10

    Hybrid Transformer-Mamba models beat pure transformers and SSMs on n-gram retrieval efficiency and length generalization, but transformers stay ahead on two-hop position lookup, with SSM-based models uniquely forming ...

  3. Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

    cs.CY 2025-10 conditional novelty 6.0 of 10

    Reputable news sites block AI crawlers at roughly six times the rate of misinformation sites, and this gap is widening over time.

  4. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.

  5. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  6. Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

    cs.CL 2025-05 conditional novelty 6.0 of 10

    FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.

  7. Overflow Prevention Enhances Long-Context Recurrent LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OPRM, a training-free chunk-and-select inference method, improves recurrent LLMs by 14-51% on LongBench and sets a 7B-class LongBench v2 record, while raising doubts about whether recurrent models exploit long-range d...

  8. Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarization

    cs.CL 2025-04 conditional novelty 6.0 of 10

    A silver-reference panel plus minimum Bayes risk decoding selects the retrieval context length for a particular retriever, summarizer, and dataset, and the chosen length beats fixed benchmark estimates on SummHay.

  9. Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Full-context and retrieval-based methods outperform hierarchical and incremental compression for large-scale multi-document summarization, though compression methods show strong intermediate information retention.

  10. NoLiMa: Long-Context Evaluation Beyond Literal Matching

    cs.CL 2025-02 conditional novelty 6.0 of 10

    NoLiMa shows that long-context language models' strong retrieval performance largely depends on literal matches, and that accuracy drops sharply once those matches are removed.

  11. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new synthetic benchmark reveals that LLM reasoning accuracy decays sigmoidally with problem complexity and that repeated sampling has poor scaling efficiency.

  12. We're Different, We're the Same: Creative Homogeneity Across LLMs

    cs.CY 2025-01 conditional novelty 6.0 of 10

    Across three divergent-thinking tests, responses from seven LLM families were substantially more similar to one another than responses from 102 humans were to one another.

  13. GG-SSMs: Graph-Generating State Space Models

    cs.LG 2024-12 reject novelty 6.0 of 10

    GG-SSMs dynamically build a minimum spanning tree over feature embeddings and propagate SSM state along it, claiming state-of-the-art results on vision and time-series benchmarks.

  14. The Zamba2 Suite: Technical Report

    cs.LG 2024-11 conditional novelty 6.0 of 10

    This paper introduces Zamba2, a suite of 1.2B, 2.7B, and 7.4B hybrid Mamba2-transformer models that claims state-of-the-art small-model quality and 30-50% lower time-to-first-token, with open weights and a 5T-token pr...

  15. State Space Models are Strong Text Rerankers

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Mamba-1 and Mamba-2 rerankers match comparably sized transformers on ranking accuracy but are less efficient in training and inference, with Mamba-2 improving on both fronts.

  16. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  17. CryptoX : Compositional Reasoning Evaluation of Large Language Models

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A benchmark that encodes prompts in secret codes and measures how much accuracy models lose, showing most LLMs, especially open-source ones, struggle on this two-step compositional task.

  18. From Divergence to Consensus: Evaluating the Role of Large Language Models in Facilitating Agreement through Adaptive Strategies

    cs.HC 2025-02 conditional novelty 4.0 of 10

    In a 75-session pilot with two Greek students per session, ChatGPT 4.0 produced consensus proposals with higher average cosine similarity to initial participant opinions and required fewer iterations than Mistral Larg...

  19. 7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement

    cs.CL 2024-12 conditional novelty 4.0 of 10

    The authors trained and openly released a 7B LLM, an instruction-tuned variant, a GRPO-based reasoning variant, and a VLM, claiming competitive or superior performance on zero-shot, few-shot, CoT, and VLM benchmarks.

  20. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

Pith tools