Sparse Mixture-of-Experts for Long-Context Transformers
rel 0.94We propose a sparse routing scheme that activates only k of n expert blocks per token, yielding near-linear scaling…
Read, search, ask, synthesize, collaborate — all grounded in your library. Every claim traceable to a page.
One question, hundreds of papers, every source cited.
Narrow to one topic. Same chat, focused answers.
Crop a chart or equation, ask about it. Vision-grounded.
Structured draft. Per-section regeneration. Cited claims.
Hand off drafts to the writing workflow you already have.
Seminal works, clusters, outliers — at a glance.
Semantic Scholar, Crossref, OpenAlex, arXiv — in one pane.
For every paper: what to read next, ranked by relevance.
Connect once. New papers flow in automatically — or drag in PDFs.
Build a prompt chain once. Run it across a folder.
TipTap editor, live cursors, threaded comments — CRDT sync.
Cards with linked papers. Templates for reading pipelines.
Formulas. Tables. Figures. Every layer.
Maximizes expected reward by adjusting policy parameters θ in the direction of higher-advantage actions. Variance-reduced via the advantage function Aπ in place of raw returns.
Linear accuracy gain with model size; diminishing returns past 56M. 7× FLOPs cost from S→L for +3.2 pts top-1.
Heatmap shows sharp peaks in layers 1–2 for content tokens, with broadening receptive field by layer 4. Consistent with finding in §4.2.
Pick a folder. Get a structured review, every claim cited.
Attention reshaped how sequence models capture long-range dependencies3. The field consolidated around self-attention after1 showed recurrence could be discarded; pretraining work2 then proved attention layers transfer across NLP tasks.
Three themes emerge. Architectural minimalism1 prioritizes parallelism. Bidirectional context2 reframes pretraining as denoising. Soft alignment3 anticipated both, reframing translation as attention-weighted lookup.
All three works rely on dot-product or additive attention13, scale to large parallel training, and report ablations on heads and alignment depth. Evaluation centers on translation BLEU13 and downstream NLP benchmarks2.
Two gaps stand out. The corpus is silent on the compute–quality trade-off — none report wall-clock or energy budgets. Second, the analysis of failure modes (attention sinks, head collapse) has not yet entered this slice of the literature.
The arc runs from soft alignment as an addition to RNNs3, to attention as the only mechanism1, to attention as substrate for transferable pretraining2. Future reviews should incorporate efficiency- and analysis-oriented work.
Visualize how papers in your library reference each other. Seminal works pull to the center; clusters self-organize by topic; outliers float free. Hover any node to see what it cites and what cites it.
Semantic Scholar, Crossref, OpenAlex, arXiv — searched in parallel, deduped, reranked. Ask why a result matters. Save, cite, or chat with the abstract.
We propose a sparse routing scheme that activates only k of n expert blocks per token, yielding near-linear scaling…
Empirical analysis of failure modes in extended-context decoders; introduces a diagnostic for measuring attention dispersal across…
Production deployment notes on a 12-billion-parameter encoder–decoder serving 9 languages, with latency budgets and…
Real-time co-authoring. Kanban for the reading pipeline. Reusable workflows that batch-extract across a folder.
Co-author in real time.
The encoder layers concentrate on tokens 5–6 by layer 2, then disperse by layer 4 — see Vaswani §3.
Open question: does this still hold past 8k context?
Manage the reading pipeline.
Run a prompt chain across a folder.
Connect Zotero or Google Drive and your papers flow into Literica AI — no re-uploading, no duplicate libraries. Or drag PDFs straight in.
Connect your account and pick the collections you want indexed. New papers added in Zotero appear in Literica AI automatically.
Pick PDFs from any Drive folder you already organize papers in. Updates sync when files change.
Or just drop PDFs into the workspace. Scanned, multi-column, equation-heavy — all parsed the same way.
Free to start. No credit card. Your library stays yours — we don't train on it.