Langchain

I’ve been playing around a bit with the langchain library to better understand the kind of things people are doing with complex prompts to extend the utility of LLMs. One example I looked at was how to use the library to enable queries over your own documents. There’s a good example using a notion based employee handbook at hwchase17/notion-qa.

I think the idea is that you only want answers from the handbook, but you want to use something like GPT3 to make the process more user friendly. GPT3 doesn’t know what’s in the handbook so how does this work?

Step 1 is to take the handbook and tokenize is up and index it for searching using faiss. This can be done once ahead of time.

Now for each question we’d like to ask of the handbook we go through a 3 step process.

Step 1: Using the user’s question as the query, do a faiss similarity search to extract snippets from any documents which have information which might be relevant

Step 2: Use the LLM to try to extract more targeted responses from these snippets using a prompt like:

Use the following portion of a long document to see if any of the text is relevant to answer the question. Return any relevant text verbatim.

Collect all of these “optimized snippets”

Step 3: Now ask the LLM to make a summary of these snippets to provide a good overall answer to the query using a prompt like:

Given the following extracted parts of a long document and a question, create a final answer. If you don’t know the answer, just say that you don’t know. Don’t try to make up an answer.

This idea works ok but its tough to know how often the answer is a good one. In some experiments the final result was not so great. You can set it up so that it will keep track of which documents had the information and make it easy for the user to go back and look at those. Still, it seems like a lot of tweaking and experimenting with the prompts will be needed to always get the results you want and its not clear how “worth it” it is do use the LLM vs just showing the original search results with snippets and letting the user click through.