[WIP] RAG-Collections

This is a summarized collection of RAG papers, tutorials, tools, applications and databases from multiple repositories with latest update.

Existing Repositories

Repository	Papers	Workshops/Tutorials	Tools	Applications	Database
Awesome-LLM-RAG	Yes*	Yes*
*Awesome-RAG	Yes	Yes*	Partial	partial
awesome-rag	Yes	Yes	Partial
Awesome-LLM-with-RAG	Yes	Partial
*RAG-Survey	Yes*	Yes*	Partial
*OpenRAG Base	Yes*	Yes*	Yes	Yes*
Awesome-RAG	Yes*	Yes*	Yes		Yes
awesome-llm-reader	Yes
Awesome RAG	Yes
Awesome-RAG		Yes	Yes*
awesome-papers-for-rag		Yes*
RAG-Survey		Yes*
- Awesome-RAG-Evaluation	Yes*			Yes

Tools

General LLM

LangChain : It is a framework for developing applications powered by LLMs.It contains a versatile Python library that simplifies the development of applications involving LLMS and RAG implementations. It features a modular architecture, allowing developers to integrate various components such as retrieval systems and knowledge bases easily. One of its standout features is memory management, enabling models to maintain conversational context across interactions. LangChain supports a wide range of pre-trained models and offers seamless integration with structured databases and unstructured text data.
Llamaindex : it is a data framework for your LLM application. Building with LlamaIndex typically involves working with LlamaIndex core and a chosen set of integrations (or plugins). Its Python library is namespaced such that import statements which include core imply that the core package is being used. In contrast, those statements without core imply that an integration package is being used.
Haystack : It is an open-source framework developed by Deepset for building end-to-end question-answering systems that utilize RAG techniques. It includes components for document stores, retrievers, and generators, allowing developers to create sophisticated applications that combine retrieval and generation capabilities efficiently. Haystack is particularly useful for creating applications that require in-depth natural language processing with substantial knowledge integration.
Hugging Face Transformers : This library provides a wide array of pre-trained models for various natural language processing tasks, including RAG implementations. Hugging Face Transformers supports both retrieval and generation tasks, allowing developers to leverage state-of-the-art language models easily. The library is highly extensible and integrates well with other tools and frameworks, making it a popular choice for building AI applications.
Semantic Kernel by Microsoft : It is an open-source project developed by Microsoft, designed to facilitate the integration of LLMs with existing software applica-tions. The core idea behind Semantic Kernel is to provide a framework that allows de-velopers to easily create, orchestrate, and customize workflows around LLMs by utiliz-ing a variety of skills like prompt chaining, memory, and learning capabilities. The tool enables developers to embed LLM capabilities directly into their applications while maintaining flexibility and control. It is highly customizable and supports a modular approach, allowing users to combine different AI services and integrate seamlessly with other Microsoft tools like Azure AI and OpenAI. In essence, Semantic Kernel helps streamline LLM-based operations in enterprise or software development con-texts.
embedchain / mem0 : It enhances AI assistants and agents with an intelligent memory layer, enabling personalized AI interactions. Mem0 remembers user preferences, adapts to individual needs, and continuously improves over time, making it ideal for customer support chatbots, AI assistants, and autonomous systems. Mem0 leverages a hybrid database approach to manage and retrieve long-term memories for AI agents and assistants. Each memory is associated with a unique identifier, such as a user ID or agent ID, allowing Mem0 to organize and access memories specific to an individual or context.
GenAI Stack The GenAI Stack will get you started building your own GenAI application in no time: Langchain + Docker + Neo4j + Ollama.
AIConfig a framework that makes it easy to build generative AI applications for production. It manages generative AI prompts, models and model parameters as JSON-serializable configs that can be version controlled, evaluated, monitored and opened in a local editor for rapid prototyping. It allows you to store and iterate on generative AI behavior separately from your application code, offering a streamlined AI development workflow

Specific RAG

RAGFlow : It is an open-source framework built to streamline the development and deployment of RAG systems. It focuses on modularity and easy integration with a variety of data sources, including documents, images, and structured datasets. With RAGFlow, users can efficiently implement RAG pipelines, leveraging LLMs to provide grounded, context-aware responses by integrating external data sources. RAGFlow supports both cloud and on-premises deployments, making it flexible for various use cases.
GraphRAG By Microsoft : GraphRAG organizes information into interconnected graphs to enhance retrieval and context understanding in RAG systems. This graph-based approach helps capture the relationships between different pieces of infor-mation, leading to more accurate and relevant LLM responses. It is particularly useful for applications that require in-depth, structured retrieval, such as legal document analysis, scientific research, and knowledge management.
AnythingLLM : It is a flexible framework that allows developers to create cus-tomized RAG systems by integrating various retrieval and generation components. This framework supports integration with multiple APIs and data sources, making it ideal for a wide range of applications, from small-scale projects to large enterprise-level systems. Its modular design enables developers to build dynamic pipelines tai-lored to specific use cases.
Cognita by TrueFoundry : It is an open-source framework from designed to sim-plify the development and deployment of RAG systems. It provides a modular, API-driven architecture that supports customization and scalability, allowing users to handle large datasets efficiently. Key features include incremental indexing, multiple query handling, autoscaling, and support for various vector databases like Qdrant and embedding models from OpenAI and Cohere. Cognita also offers a user-friendly UI, making it suitable for both technical and non-technical users to experiment and deploy RAG systems. In addition, it uses Langchain/Llamaindex under the hood, which means we can always integrate the latest modules/functions from these two popular tools.
Weaviate Verba : An open-source RAG application designed to make RAG ac-cessible to users without extensive technical expertise. It features a user-friendly inter-face for customizing RAG pipelines and interacting with generated outputs. It can connect either locally with Ollama and Huggingface or through LLM providers such as Anthrophic, Cohere, and OpenAI.
FlashRAG a Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. the toolkit includes 32 pre-processed benchmark RAG datasets and 15 state-of-the-art RAG algorithms.
AutoRAG : This tool automatically identifies the best RAG pipeline for your specif-ic data and use case. With many RAG pipelines and modules available, manually test-ing them all is time-consuming and difficult. It simplifies this by automating the eval-uation of different module combinations, helping you quickly find the optimal solution based on your own evaluation data. Try it now to streamline your search for the best RAG pipeline.
llmware : it provides a unified framework for building LLM-based applications (e.g, RAG, Agents), using small, specialized models that can be deployed privately, integrated with enterprise knowledge sources safely and securely, and cost-effectively tuned and adapted for any business process.
NeumAI by NeumTry : NeumAI is an open-source AI framework designed to of-fer a user-friendly platform for building and deploying ML/ AI models. The focus of NeumAI is to simplify AI model development by providing intuitive tools that mini-mize the complexity of setting up training environments, managing data, and scaling machine learning operations. The framework is built to be accessible for both beginners and experienced developers, emphasizing easy-to-understand APIs and modular com-ponents. NeumAI supports deep learning models and offers features that help with model optimization, experiment tracking, and collaborative AI development. It is de-signed to be highly flexible and adaptable for various AI applications, from research to production environments.
beyondllm : It offers an all-in-one toolkit for experimentation, evaluation, and deployment of RAG systems, simplifying the process with automated integration, customizable evaluation metrics, and support for various Large Language Models (LLMs) tailored to specific needs, ultimately aiming to reduce LLM hallucination risks and enhance reliability.
MindSQL : It is a Python RAG Library designed to streamline the interaction between users and their databases using just a few lines of code. With seamless integration for renowned databases such as PostgreSQL, MySQL, and SQLite, MindSQL also extends its capabilities to major databases like Snowflake and BigQuery by extending the IDatabase Interface. This library utilizes LLM like GPT-4, Llama 2, Google Gemini, and supports knowledge bases like ChromaDB and Faiss.

Evaluation

We can use the general frameworks, such as Amazon Bedrock, Azure AI Studio Google's Vertex AI Studio, LangSmith and TruLens and DeepEval, for the evaluations. Or we can customize our datasets, e.g., SuperAnnotate. Below are some dedicated tools for RAG.

Phoenix It is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙LlamaIndex, 🦜⛓LangChain, Haystack, 🧩DSPy) and LLM providers (OpenAI, Bedrock, and more).
TruLens A tool can systematically evaluate and track your LLM experiments. the core concepts behind TruLens include Feedback Functions, The RAG Triad, and Honest, Harmless and Helpful Evals.
ragas Ragas is a framework that helps you evaluate your RAG pipelines.Dedicated solutions to evaluate, monitor and improve performance of LLM & RAG application in production including custom models for production quality monitoring
CRUD_RAG This project fully supports the Chinese RAG system evaluation, which includes native Chinese datasets, evaluation tasks, and baseline models; It covers CRUD (Create, Read, Update, Delete) operations, which are used to evaluate the RAG system's ability to add, reduce, correct information, as well as to answer questions based on the retrieve information;
RGB An implementation for Benchmarking Large Language Models in Retrieval-Augmented Generation arxiv
ARES ARES is a groundbreaking framework for evaluating Retrieval-Augmented Generation (RAG) models. The automated process combines synthetic data generation with fine-tuned classifiers to efficiently assess context relevance, answer faithfulness, and answer relevance, minimizing the need for extensive human annotations. ARES employs synthetic query generation and Prediction-Powered Inference (PPI), providing accurate evaluations with statistical confidence.
ALCE This repository contains the code and data for paper Enabling Large Language Models to Generate Text with Citations (https://arxiv.org/abs/2305.14627). In this paper, ALCE, a benchmark for Automatic LLMs' Citation Evaluation, is proposed. ALCE contains three datasets: ASQA, QAMPARI, and ELI5. it provides automatic evaluation code of LLM generations around three dimensions: fluency, correctness, and citation quality. This repository also includes code to reproduce the baselines in the paper.
RAG Evaluator : It is a Python library for evaluating Retrieval-Augmented Generation (RAG) systems. It provides various metrics to evaluate the quality of generated text against reference text.