Merge pull request langchain-ai#110 from langchain-ai/molly/ingest-im…

…provements

ingest improvements

.gitignore

@@ -137,3 +137,4 @@ dmypy.json
 
 vectorstore.pkl
 langchain.readthedocs.io/
+.vercel

Makefile

@@ -1,6 +1,6 @@
 .PHONY: start
 start:
-	uvicorn main:app --reload --port 9000
+	uvicorn main:app --reload --port 8080
 
 .PHONY: format
 format:

Procfile

@@ -0,0 +1,2 @@
+# Modify this Procfile to fit your needs
+web: uvicorn main:app --host 0.0.0.0 --port 8080

README.md

@@ -1,4 +1,4 @@
-# 🦜️🔗 ChatLangChain
+# 🦜️🔗 Chat LangChain
 
 This repo is an implementation of a locally hosted chatbot specifically focused on question answering over the [LangChain documentation](https://langchain.readthedocs.io/en/latest/).
 Built with [LangChain](https://github.com/hwchase17/langchain/) and [FastAPI](https://fastapi.tiangolo.com/).
@@ -7,35 +7,50 @@ The app leverages LangChain's streaming support and async API to update the page
 
 ## ✅ Running locally
 1. Install dependencies: `pip install -r requirements.txt`
-1. Run `ingest.sh` to ingest LangChain docs data into the vectorstore (only needs to be done once).
+1. Run `python ingest.py` to ingest LangChain docs data into the Weaviate vectorstore (only needs to be done once).
    1. You can use other [Document Loaders](https://langchain.readthedocs.io/en/latest/modules/document_loaders.html) to load your own data into the vectorstore.
-1. Run the app: `make start`
-   1. To enable tracing, make sure `langchain-server` is running locally and pass `tracing=True` to `get_chain` in `main.py`. You can find more documentation [here](https://langchain.readthedocs.io/en/latest/tracing.html).
-1. Open [localhost:9000](http://localhost:9000) in your browser.
+1. Run the app: `make start` for backend and `npm run dev` for frontend (cd into chat-langchain first)
+   1. Make sure to enter your environment variables to configure the application: 
+   ```
+   export OPENAI_API_KEY=
+   export WEAVIATE_URL=
+   export WEAVIATE_API_KEY=
+   
+   # for tracing
+   export LANGCHAIN_TRACING_V2=true
+   export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
+   export LANGCHAIN_API_KEY=
+   export LANGCHAIN_PROJECT=
+   ```
+1. Open [localhost:3000](http://localhost:3000) in your browser.
 
 ## 🚀 Important Links
 
-Deployed version (to be updated soon): [chat.langchain.dev](https://chat.langchain.dev)
+Deployed version: [chat.langchain.com](https://chat.langchain.com)
 
-Hugging Face Space (to be updated soon): [huggingface.co/spaces/hwchase17/chat-langchain](https://huggingface.co/spaces/hwchase17/chat-langchain)
-
-Blog Posts: 
-* [Initial Launch](https://blog.langchain.dev/langchain-chat/)
-* [Streaming Support](https://blog.langchain.dev/streaming-support-in-langchain/)
 
 ## 📚 Technical description
 
 There are two components: ingestion and question-answering.
 
 Ingestion has the following steps:
 
-1. Pull html from documentation site
-2. Load html with LangChain's [ReadTheDocs Loader](https://langchain.readthedocs.io/en/latest/modules/document_loaders/examples/readthedocs_documentation.html)
-3. Split documents with LangChain's [TextSplitter](https://langchain.readthedocs.io/en/latest/reference/modules/text_splitter.html)
-4. Create a vectorstore of embeddings, using LangChain's [vectorstore wrapper](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) (with OpenAI's embeddings and FAISS vectorstore).
+1. Pull html from documentation site as well as the Github Codebase
+2. Load html with LangChain's [RecursiveURLLoader Loader](https://python.langchain.com/docs/integrations/document_loaders/recursive_url_loader)
+2. Transform html to text with [Html2TextTransformer](https://python.langchain.com/docs/integrations/document_transformers/html2text)
+3. Split documents with LangChain's [RecursiveCharacterTextSplitter](https://api.python.langchain.com/en/latest/text_splitter/langchain.text_splitter.RecursiveCharacterTextSplitter.html)
+4. Create a vectorstore of embeddings, using LangChain's [Weaviate vectorstore wrapper](https://python.langchain.com/docs/integrations/vectorstores/weaviate) (with OpenAI's embeddings).
 
-Question-Answering has the following steps, all handled by [ChatVectorDBChain](https://langchain.readthedocs.io/en/latest/modules/indexes/chain_examples/chat_vector_db.html):
+Question-Answering has the following steps, all handled by [OpenAIFunctionsAgent](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent):
 
-1. Given the chat history and new user input, determine what a standalone question would be (using GPT-3).
+1. Given the chat history and new user input, determine what a standalone question would be (using GPT-3.5).
 2. Given that standalone question, look up relevant documents from the vectorstore.
-3. Pass the standalone question and relevant documents to GPT-3 to generate a final answer.
+3. Pass the standalone question and relevant documents to GPT-4 to generate and stream the final answer.
+4. Generate a trace URL for the current chat session, as well as the endpoint to collect feedback. 
+
+## Deprecated Links
+Hugging Face Space (to be updated soon): [huggingface.co/spaces/hwchase17/chat-langchain](https://huggingface.co/spaces/hwchase17/chat-langchain)
+
+Blog Posts: 
+* [Initial Launch](https://blog.langchain.dev/langchain-chat/)
+* [Streaming Support](https://blog.langchain.dev/streaming-support-in-langchain/)

_scripts/evaluate_chains.py

@@ -0,0 +1,182 @@
+import argparse
+import functools
+import os
+from typing import Literal, Optional, Union
+
+from langsmith.evaluation.evaluator import EvaluationResult
+from langsmith.schemas import Example, Run
+
+import weaviate
+from langchain import prompts
+from langchain.chains import RetrievalQA
+from langchain.chat_models import ChatAnthropic, ChatOpenAI
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.memory import ConversationBufferMemory
+from langchain.schema.retriever import BaseRetriever
+from langchain.schema.runnable import Runnable, RunnableMap
+from langchain.schema.output_parser import StrOutputParser
+from langchain.smith import RunEvalConfig
+from langchain.vectorstores import Weaviate
+from langchain.prompts import PromptTemplate, ChatPromptTemplate, MessagesPlaceholder
+from langsmith import Client
+from langsmith import RunEvaluator
+from langchain import load as langchain_load
+from operator import itemgetter
+import json
+
+_PROVIDER_MAP = {
+    "openai": ChatOpenAI,
+    "anthropic": ChatAnthropic,
+}
+
+_MODEL_MAP = {
+    "openai": "gpt-3.5-turbo",
+    "anthropic": "claude-2",
+}
+
+def create_chain(
+    retriever: BaseRetriever,
+    model_provider: Union[Literal["openai"], Literal["anthropic"]],
+    chat_history: Optional[list] = None,
+    model: Optional[str] = None,
+    temperature: float = 0.0,
+) -> Runnable:
+    model_name = model or _MODEL_MAP[model_provider]
+    model = _PROVIDER_MAP[model_provider](model=model_name, temperature=temperature)
+
+    _template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.
+
+    Chat History:
+    {chat_history}
+    Follow Up Input: {question}
+    Standalone Question:"""
+
+
+    CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)
+
+    _template = """
+    You are an expert programmer and problem-solver, tasked to answer any question about Langchain. Using the provided context, answer the user's question to the best of your ability using the resources provided.
+    If you really don't know the answer, just say "Hmm, I'm not sure." Don't try to make up an answer.
+    Anything between the following markdown blocks is retrieved from a knowledge bank, not part of the conversation with the user. 
+    <context>
+        {context} 
+    <context/>"""
+
+    if chat_history:
+        _inputs = RunnableMap(
+                {
+                    "standalone_question": {
+                        "question": lambda x: x["question"],
+                        "chat_history": lambda x: x["chat_history"],
+                    } | CONDENSE_QUESTION_PROMPT | model | StrOutputParser(),
+                    "question": lambda x: x["question"],
+                    "chat_history": lambda x: x["chat_history"],
+                }
+            )
+        _context = {
+            "context": itemgetter("standalone_question") | retriever,
+            "question": lambda x: x["question"], 
+            "chat_history": lambda x: x["chat_history"],
+        }
+        prompt = ChatPromptTemplate.from_messages([
+            ("system", _template),
+            MessagesPlaceholder(variable_name="chat_history"),
+            ("human", "{question}"),
+        ])
+    else:
+        _inputs = RunnableMap(
+            {
+                "question": lambda x: x["question"],
+                "chat_history": lambda x: [],
+            }
+        )
+        _context = {
+            "context": itemgetter("question") | retriever,
+            "question": lambda x: x["question"], 
+            "chat_history": lambda x: [],
+        }
+        prompt = ChatPromptTemplate.from_messages([
+            ("system", _template),
+            ("human", "{question}"),
+        ])
+
+    chain = (
+        _inputs
+        | _context
+        | prompt 
+        | model
+        | StrOutputParser()
+    )
+
+    return chain
+
+
+def _get_retriever():
+    WEAVIATE_URL = os.environ["WEAVIATE_URL"]
+    WEAVIATE_API_KEY = os.environ["WEAVIATE_API_KEY"]
+
+    embeddings = OpenAIEmbeddings(chunk_size=200)
+    client = weaviate.Client(
+        url=WEAVIATE_URL,
+        auth_client_secret=weaviate.AuthApiKey(api_key=WEAVIATE_API_KEY),
+    )
+    weaviate_client = Weaviate(
+        client=client,
+        index_name="LangChain_test_idx",
+        text_key="text",
+        embedding=embeddings,
+        by_text=False,
+        attributes=["source"],
+    )
+    return weaviate_client.as_retriever(search_kwargs=dict(k=10))
+
+class CustomHallucinationEvaluator(RunEvaluator):
+
+    @staticmethod
+    def _get_llm_runs(run: Run) -> Run:
+        runs = []
+        for child in (run.child_runs or []):
+            if run.run_type == "llm":
+                runs.append(child)
+            else:
+                runs.extend(CustomHallucinationEvaluator._get_llm_runs(child))
+
+
+    def evaluate_run(self, run: Run, example: Example | None = None) -> EvaluationResult:
+        llm_runs = self._get_llm_runs(run)
+        if not llm_runs:
+            return EvaluationResult(key="hallucination", comment="No LLM runs found")
+        if len(llm_runs) > 0:
+            return EvaluationResult(key="hallucination", comment="Too many LLM runs found")
+        llm_run = llm_runs[0]
+        messages = llm_run.inputs["messages"]
+        langchain_load(json.dumps(messages))
+
+
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--dataset-name", default="Chat LangChain Questions")
+    parser.add_argument("--model-provider", default="openai")
+    parser.add_argument("--prompt-type", default="chat")
+    args = parser.parse_args()
+    client = Client()
+    # Check dataset exists
+    ds = client.read_dataset(dataset_name=args.dataset_name)
+    retriever = _get_retriever()
+    constructor = functools.partial(
+        create_chain,
+        retriever=retriever,
+        model_provider=args.model_provider,
+    )
+    chain = constructor()
+    eval_config = RunEvalConfig(evaluators=["qa"], prediction_key="output")
+    results = client.run_on_dataset(
+        dataset_name=args.dataset_name,
+        llm_or_chain_factory=constructor,
+        evaluation=eval_config,
+        verbose=True,
+    )
+    proj = client.read_project(project_name=results["project_name"])
+    print(proj.feedback_stats)