REVIEW 6 cited by
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.
Forward citations
Cited by 6 Pith papers
-
MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning
MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.
-
KoRe: Compact Knowledge Representations for Large Language Models
KoRe encodes 1-hop knowledge graph subgraphs as compact discrete tokens for injection into LLMs, achieving competitive benchmark performance with up to 10x token reduction.
-
AI for the Open-World: the Learning Principles
Open-world AI requires rich features, disentangled representations, and inference-time learning; the thesis presents techniques and large-scale experiments supporting these principles.
-
Automatic Demonstration Selection for LLM-based Tabular Data Classification
A spectral-gap heuristic on a token-ID similarity graph automatically chooses the number of in-context demonstrations for LLM-based tabular classification, with mixed empirical results against random selection.
-
Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph
FastToG lets LLMs reason 'community by community' over knowledge graphs, reporting higher accuracy and faster reasoning than Think-on-Graph.
-
LQ-Adapter: ViT-Adapter with Learnable Queries for Gallbladder Cancer Detection from Ultrasound Image
A modified ViT-Adapter with learnable queries improves gallbladder cancer bounding-box detection on ultrasound images, beating DINO and FocalNet-DINO.
Discussion (0). Continue with ORCID to comment.