Pith. sign in

REVIEW 12 cited by

Continual learning with hypernetworks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1906.00695 v4 pith:G36A5QQG submitted 2019-06-03 cs.LG cs.AIstat.ML

Johannes von Oswald , Christian Henning , Benjamin F. Grewe , João Sacramento This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords hypernetworkstask-conditionedlearningtaskcontinualhypernetworklongmemory
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Architecture Generalization with MetaNCA

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A learned local rule (Weight Transformer) iteratively self-organizes task-network weights from local graph neighborhoods and generalizes across unseen MLP, CNN, and ResNet architectures up to ~2M parameters.

  2. Learning to Remember, Learn, and Forget in Attention-Based Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Palimpsa adds a per-slot importance/precision state to gated linear attention, letting a fixed-size memory forget stale information and protect important information, and recovers Mamba2 as a high-forgetting limit.

  3. Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.

  4. Privacy-Aware Lifelong Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    PALL learns tasks in a single network as sparse subnetworks and performs exact task unlearning by resetting the task's trained parameters and briefly retraining shared ones from a rehearsal buffer.

  5. Learning to Forget using Hypernetworks

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A diffusion-based hypernetwork can generate classifier weights with near-zero accuracy on a requested forget class and near-retrained accuracy on retained classes.

  6. Context-Aware Force Estimation for Deformable Tool Manipulation in Robotic Environmental Swabbing via Few-Shot Continual Adaptation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A frozen LSTM backbone with FiLM-based few-shot context adaptation estimates tip-level contact forces for deformable swabbing tools across nine surface-tool regimes using only wrist-mounted proprioception.

  7. An open dataset of neural networks for hypernetwork research

    cs.LG 2025-07 reject novelty 5.0 of 10

    A public dataset of 10,000 LeNet-5 networks split into 10 Imagenette classes is released, with a 72% Naive Bayes baseline for classifying networks by their weights.

  8. Text-to-LoRA: Instant Transformer Adaption

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A hypernetwork can generate task-specific LoRA adapters from a text description, and when trained with supervised fine-tuning it zero-shot outperforms a multi-task LoRA baseline on ten benchmarks.

  9. IRS: Incremental Relationship-guided Segmentation for Digital Pathology

    eess.IV 2025-05 conditional novelty 5.0 of 10

    IRS uses anatomical relationships between old and new classes to guide knowledge distillation in a prompt-driven mixture-of-experts network, improving class-incremental segmentation of kidney pathology compared to baselines.

  10. On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning

    cs.CV 2025-01 conditional novelty 5.0 of 10

    DCNet improves exemplar-free class-incremental learning by embedding classes into mutually orthogonal hyperspherical directions and adaptively compensating intra-class aggregation, outperforming prior exemplar-free an...

  11. Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation

    cs.CV 2025-01 reject novelty 4.0 of 10

    Combining ControlNet and GLIGEN, ObjectDiffusion conditions Stable Diffusion on bounding boxes and open-ended object names, reporting improved AP50, AR, and FID on COCO2017.

  12. Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review

    cs.LG 2025-06 conditional novelty 2.0 of 10

    A survey that categorizes continual reinforcement learning methods, environments, and evaluation metrics for deep RL, with a focus on robotics.

Pith tools