Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs

Always Active

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.

Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.

Home Editors Pick Agentic AI Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs

Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs

Stanford researchers have released Memorilla, an open-source memory system for LLMs. The Memorilla memory system reads a whole document collection and hands a frozen LLM just 16 memory tokens. On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, against 80.43% with every document in context. It did that with 108x fewer document tokens. Code, training recipes and checkpoints are public under Apache 2.0. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.

TL;DR

• Size: 194M-parameter trainable memory module. It always outputs 16 memory tokens, whatever the collection size. The encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) stay frozen.
• Runs on: Linux with CUDA GPUs and Python 3.11. The quickstart needs 1 GPU with 80 GB (recipes ran on H100 80GB). No quantized variants disclosed.
• Performance: Highest reported average across 8 benchmarks (49.68), top score on 5 of 8.
• Best: PersonaMem-v2 accuracy of 44.60%, versus 27.92% for full context.
• Worst: PubMedQA accuracy of 59.00%, versus 69.00% for LLMLingua-2.

What is Memorilla?

Memorilla is a memory module that turns a document collection into 16 vectors for a frozen LLM. The work comes from Jure Leskovec’s SNAP group, with co-authors from Toronto, Penn and MIT.

▲ AI DEV SIGNALS · FREE NEWSLETTER

Get the AI releases that matter, before they trend.

You’re in. Check your inbox to confirm.

▲ AI DEV SIGNALSFREE NEWSLETTER

Get the AI releases that matter, before they trend.

Model releases, open weights, agents and papers with code — the 5 that matter, in 1 email. Free.

You’re in. Check your inbox to confirm.

A frozen LLM can only use what sits in its context window. Full-context prompting grows slower as history grows. Retrieval pastes the chunks most similar to the question, and can miss weaker clues. Memorilla takes a third route: it learns what each question needs from the whole store.

How does Memorilla work?

The pipeline has 3 steps:

• Encode once. A frozen Qwen3-Embedding-4B embeds each document a single time. New documents never force re-encoding of old ones.
• Read per question. 16 learned slots attend to the question and the full store. The module uses 2 cross-attention layers with 8 heads.
• Inject. The slots are projected into Qwen3-8B’s input embedding space and prepended to the question.

Only the memory module trains: 194,123,776 parameters for the 8B decoder. Training runs in 3 stages. Stage 1 uses about 1.98M Wikipedia paraphrase-reconstruction examples. Stage 2 adds about 1.34M examples from 10 QA and summarization datasets. Stage 3 fine-tunes on all 8 benchmarks. The full recipe takes about 19 hours on 4 H100 GPUs.

How does Memorilla perform on benchmarks?

All baselines use the same frozen Qwen3-8B, per the paper’s main table. Scores are self-reported by the research team.

• FactKG: 84.33% accuracy, versus 80.43% full context and 80.38% MemoRAG.
• PersonaMem-v2: 44.60%, versus 30.90% for xRAG, the next best method.
• TriviaQA: 57.00 exact match, just ahead of MemoRAG at 56.60.
• LaMP-4 / LaMP-7: 18.15 and 48.37 ROUGE-L, both the best scores.
• Average: 49.68, versus 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0.

Memorilla loses on 3 benchmarks. On PubMedQA it scores 59.00%, while LLMLingua-2 reaches 69.00%. Those abstracts total only 324 tokens, so compression has little to gain. MemoRAG leads on PersonalizationV4 (67.88% vs 66.65%) and NarrativeQA ROUGE-L (20.51 vs 19.31). MemoRAG used about 690x and 50x more document tokens there.

Compression peaks on NarrativeQA: 16 tokens replace 67,385 document tokens, a 4,212x cut. As agent memory, 8 memory tokens lift a Qwen3-4B TextWorld agent from 18.6% to 24.2% average success.

How fast is Memorilla compared to full context?

On 1 H100, Memorilla answers in 0.74 s at 512 documents (about 128K tokens). Full context slows from 0.75 s to 19.9 s, which is 27x slower. The memory module itself takes about 2 ms.

An optional memory cache precomputes cross-attention keys and values. On 1 B200, a question then takes 8.1 ms at 1M documents and 33 ms at 4M. Withtorch.compile, the 1M case drops to 5.0 ms. The cache costs 80 GiB at 4M documents.

How do you run Memorilla?

Install withuv on Linux, then load MemoryVLLM.from_pretrained. The first run downloads about 25 GB of models. Checkpoints live at memorilla/Memorilla-Qwen3-8B as Stage 1, 2, 3, PV4 and PMv2 folders.

Each checkpoint must run with the decoder it was trained with (evaluation docs). Custom data needs parquet files withcollection_id, question, answer and documents columns. The research team also released PersonalizationV4: 149 synthetic users, 200 conversations each, under CC BY 4.0.

Memorilla vs Mem0, MemoRAG and xRAG

Benchmark scores come from Memorilla’s paper, which ran every baseline on the same Qwen3-8B. Licenses come from each project’s repository.

Key Takeaways

• 16 memory tokens replace whole document collections for a frozen Qwen3-8B.
• FactKG: 84.33% vs 80.43% full context, with 108x fewer tokens.
• Highest reported 8-benchmark average: 49.68.
• Weak spot: PubMedQA at 59.00% and fine-grained facts.
• Apache 2.0 code; 194M-parameter checkpoints on Hugging Face.

Check out the paper, browse the GitHub repo, and download the checkpoints. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.

Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

YOU READ TO THE END

Get the stories that didn’t make it on the homepage.

You’re in. Check your inbox to confirm.

Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

▲ AI DEV SIGNALS

The newsletter 70,000+ AI builders read

Model releases, open weights and papers with code — in 1 email. Free.

You’re in. Check your inbox to confirm. Browse past issues on aidevsignals.com →

Get AI Dev Signals

The newsletter 70,000+ AI builders read. Model releases, tools and research, without the hype.

注意转载授权 · 阅读原文