RAG Resources

• Files from my "Using Retrieval Augmented Generation (RAG), Langchain, and LLMs for Cybersecurity Operations" Course
• LangFlow - a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database.

Disadvantages of RAG

• Disadvantages of RAG

RAG Patterns

• Generative AI Lifecycle Patterns
• Why do RAG pipelines fail? Advanced RAG Patterns — Part1 Ozgur Guler
• How to improve RAG peformance — Advanced RAG Patterns — Part2
• Patterns for Building LLM-based Systems & Products
• AI Engineer Summit - Building Blocks for LLM Systems & Products
• Technical Considerations for Complex RAG

Dialogue Routing

• Routing in RAG-Driven Applications

Retrieval

Vector Retrieval

• Boosting RAG: Picking the Best Embedding & Reranker models
• What We Need to Know Before Adopting a Vector Database

Chunking

• Chunking Strategies for LLM Applications
• Evaluating the Ideal Chunk Size for a RAG System using LlamaIndex
• How to Chunk Text Data — A Comparative Analysis

Embeddings

Vector Search

• Awesome Search
• Advanced RAG Retrieval Strategies: Sentence Window Retrieval

Not Vector Retrieval

• Vector Search Is Not All You Need
• Build a search engine, not a vector DB
• Improving RAG (Retrieval Augmented Generation) Answer Quality with Re-ranker
• From Search to Synthesis: Enhancing RAG with BM25 and Reciprocal Rank Fusion

Generation

Prompts

• Emerging RAG & Prompt Engineering Architectures for LLMs
• How to Cut RAG Costs by 80% Using Prompt Compression

Prompting strategies

Multi-Modal RAG

• Multi-Modal RAG

Multi-index RAG

• Having all of your data stored in one collection isn't always the best for RAG apps

Multi-Document

• Advanced RAG — Multi-Documents Agent with LlamaIndex

FLARE

• Better RAG with Active Retrieval Augmented Generation FLARE

Chain-of-Verification

• in-Of-Verification Reduces Hallucination in LLMs

Chain-Of-Thought

• Chain-Of-Thought Prompting In LLMs

Context

• The Needle In a Haystack Test
• Conversational Memory for LLMs with Langchain

Long context RAG

• The next generation of RAG: Long-Context RAG
• NVIDIA Research: RAG with Long Context LLMs

Knowledge and Knowledge Graphs

• Graph RAG: Unleashing the Power of Knowledge Graphs with LLM
• Embeddings + Knowledge Graphs: The Ultimate Tools for RAG Systems
• The Practical Benefits to Grounding an LLM in a Knowledge Graph Daniel Bukowski

• Implement RAG with Knowledge Graph and Llama-Index

• Awesome Knowledge Graphs

• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

Automated prompt optimization

Hallucination

• How to Detect Hallucinations in LLMs
• Measuring Hallucinations in RAG Systems

Guardrails

• Safeguarding LLMs with Guardrails
• NeMo Guardrails: The Missing Manual

LLM Models

Finetuning and Pretraining

• Fine-Tuning Llama 2.0 with Single GPU Magic
• Practitioners guide to fine-tune LLMs for domain-specific use case
• Are You Pre-training your RAG Models on Your Raw Text?
• Combine Multiple LoRA Adapters for Llama 2
• RAG vs Finetuning — Which Is the Best Tool to Boost Your LLM Application?

Evaluation of RAGs

• RAG Evaluation
• Evaluating RAG: A journey through metrics
• Exploring End-to-End Evaluation of RAG Pipelines
• Evaluation Driven Development, the Swiss Army Knife for RAG Pipelines
• Evaluating the Ideal Chunk Size for a RAG System using LlamaIndex

Performance and cost

• Secrets to Optimizing RAG LLM Apps for Better Performance, Accuracy and Lower Costs!

Privacy

• Masking PII Data in RAG Pipeline

Security

• Hijacking Chatbots: Dangerous Methods Manipulating GPTs

Applications of RAG

Chatbots

Tools

• Three Open-Source RAG Tools You Need to Know About
• HayStack
• RAGAS

DSPy

• DSPy — Does It Live Up To The Hype?

AutoRAG

• AutoRAG - AutoML tool for RAG. Automatically optimize RAG pipeline with single YAML file.

AutoGPT

Langchain

• Langchain
• Langchain is NOT for production use. Here is why ..

LlamaIndex

• LlamaIndex

• Building Production-Ready LLM Apps with LlamaIndex: Document Metadata for Higher Accuracy Retrieval

• Building Production-Ready LLM Apps With LlamaIndex: Recursive Document Agents for Dynamic Retrieval

Vendor-specific examples

• RAG Pipeline with Mistral 7B Instruct Model in Colab: A Step-by-Step Guide Qendel AI GoPenAI

Elastcisearch + OpenAI

• ChatGPT and Elasticsearch: OpenAI meets private data

OpenAI and ChatGPT

• Compare PDF Question Answering Systems Build with OpenAI and Google VertexAI

Tools and fucntions

• Unlocking the Power of the OpenAI API: Master Function-Calling with Practical Examples
• penAI/Chat-GPT Function Calling : for Enhanced AI Interactions

Vespa

• Hands-On RAG guide for personal data with Vespa and LLamaIndex

Qdrant

Running RAGs in production

Vectors corner

• Similarity Search, Part 2: Product Quantization
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval
• Cohere int8 & binary Embeddings - Scale Your Vector Database to Large Datasets Image of Nils Reimers

Research Papers

Survey and Benchmark

Benchmarking Large Language Models in Retrieval-Augmented Generation \ Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun \ arXiv 2023. [Paper][Github] \ 4 Sep 2023

Retrieval-enhanced LLMs

Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models \ Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, Dong Yu \ arxiv - Nov 2023 [Paper]

REST: Retrieval-Based Speculative Decoding \ Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, Di He \ arXiv - Nov 2023 [Paper][Github]

Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Anonymous
ICLR 24 – Oct 2023 [paper]

Self-Knowledge Guided Retrieval Augmentation for Large Language Models \ Yile Wang, Peng Li, Maosong Sun, Yang Liu \ arXiv - Oct 2023 [Ppaer]

Retrieval meets Long Context Large Language Models \ Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro \ arxiv - Oct 2023 [Paper]

DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts
arXiv – Oct 2023 [paper] [code]

Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su
ICLR 24 – May 2023 [paper] [code]

Active Retrieval Augmented Generation
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig
arXiv – May 2023 [paper] [code]

REPLUG: Retrieval-Augmented Black-Box Language Models
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih
arXiv – Jan 2023 [paper]

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela NeurIPS 2020 - May 2020 [Paper]

RAG Instruction Tuning

RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Anonymous
ICLR 24 – Oct 23 [paper]

InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro \ arXiv - Oct 23 [paper]

RAG In-Context Learning

In-Context Retrieval-Augmented Language Models
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham
AI21 Labs – Jan 2023 [paper] [code]

RAG Embeddings

RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling \ Jingcheng Deng, Liang Pang, Huawei Shen, Xueqi Cheng \ EMNLP 2023 - Oct 2023 [Paper][Github]

Text Embeddings Reveal (Almost) As Much As Text \ John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush \ EMNLP 2023 - Oct 2023 [Paper][Github]

Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents \ Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, Han Xiao \ arXiv - Oct 2023. [Paper][Model]

RAG Simulators

KAUCUS: Knowledge Augmented User Simulators for Training Language Model Assistants \ Kaustubh D. Dhole \ Simulation of Conversational Intelligence in Chat, EACL 2024 [Paper]

RAG Search

RAG Long-text and Memory

HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models \ Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su \ arXiv - May 2024 [paper] [GitHub]

Understanding Retrieval Augmentation for Long-Form Question Answering \ Hung-Ting Chen, Fangyuan Xu, Shane A. Arora, Eunsol Choi \ arXiv - Oct 2023 [Paper]

RAG Evaluation

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems \ Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia \ arXiv - Nov 2023. [Paper] [Github]

RAG Optimization

Learning to Filter Context for Retrieval-Augmented Generation \ Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, Graham Neubig \ arxiv- Nov 2023 [Paper][Github]

Large Language Models Can Be Easily Distracted by Irrelevant Context \ Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou \ ICML 2023 - Jan 2023 [Paper][Github]

Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks \ Akari Asai, Matt Gardner, Hannaneh Hajishirzi \ NAACL 2022 - Dec 2021 [Paper][Github]

When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories \ Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi \ ACL 2023 - Dec 2022 [Paper][Github]

RAG Application

Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination \ Haoqiang Kang, Xiao-Yang Liu \ arXiv - Nov 2023 [Paper]

Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature \ Alejandro Lozano, Scott L Fleming, Chia-Chun Chiang, Nigam Shah \ arXiv - Oct 2023. [Paper]

PEARL: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers \ Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, Tara Safavi \ arXiv - Nov 2023. [Paper]