Interacting with Large Language Models Programmatically

Additional Context with Retrieval Augmented Generation

---

When interacting with a large language model, the response from the model is formed based on context. The context can be the prompt that is given to the model — like in the stateless chat, or it can be the conversation history — like in the stateful chat.

In both of these instances, the response from the model is based on the prompt and the data that the model has been trained on. Training large language models takes plenty of time and resources, and thus, the models do not have the latest information such as recent news or the latest research in their training data.

Retrieval-Augmented Generation

Retrieval-Augmented Generation is an approach that allows providing additional information for the models. The key idea in retrieval-augmented generation is to create a local database that holds the latest information. Then, when prompting a large language model, the system first searches the local database for relevant information, and provides the found information as a context to the query sent to the the large language model. The idea is summarized in the Figure 1 below.

Fig 1. — Retrieval-Augmented Generation. Image by Gknor, licensed with CC-BY-SA 4.0.

Embedchain

There exists a range of libraries that can be used to build retrieval-augmented generation systems. Here, we look into using Embedchain, which is among the easier to use ones.

To get started with Embedchain, we need to install it. We can do this by running the following command.

pip install embedchain

This installs the Embedchain library and the relevant dependencies, including a database that is used for storing information. Using Embedchain involves creating an application, adding documents to it, and then querying the application. Adding a document means that Embedchain loads a document and stores it into its database. Documents can be added in many formats, including using web pages, PDF files, and so on. The database is stored locally.

Embedchain provides integrations to existing large language model APIs such as the OpenAI API and HuggingFace API. Let’s look into using these next.

OpenAI API

To use Embedchain with the OpenAI API, we need to provide the application an OpenAI API key, and to load the documents to query. In the following, we load information about Jean Sibelius to the model, and then query the model about his birth date.

import os
from embedchain import App

os.environ["OPENAI_API_KEY"] = "Your secret OpenAI API key"

app = App()
app.add("https://en.wikipedia.org/wiki/Jean_Sibelius")

response = app.query("When was Jean Sibelius born?")
print(response)

When you run the program, you see the response from the model printed to the console. One possible output is as follows.

Jean Sibelius was born on 8 December 1865.

In addition, you might notice notice that a folder called db is created to the folder that hosts the code. The folder db stores the database into which the documents are added.

HuggingFace API

Alternatively, we can use the HuggingFace API with Embedchain. To do this, we need to provide the application with the HuggingFace API key, the model to use, and an embedding model to use that is used to encode the documents. As the encoding happens locally when working with HuggingFace, we also need to install the embedding model. In this example, we use the sentence-transformers library to encode the documents. It is installed as follows.

pip install sentence-transformers

With the embedding model installed, we can now use Embedchain with the HuggingFace API. The example uses the Phi-3-mini-4k-instruct model and a all-mpnet-base-v2 embedding model.

The concrete task is similar to the earlier — we are storing information about Jean Sibelius, and then asking for his year of birth.

import os
from embedchain import App

os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "Your secret HuggingFace API key"

config = {
  'llm': {
    'provider': 'huggingface',
    'config': {
      'model': 'microsoft/Phi-3-mini-4k-instruct',
      'top_p': 0.5
    }
  },
  'embedder': {
    'provider': 'huggingface',
    'config': {
      'model': 'sentence-transformers/all-mpnet-base-v2'
    }
  }
}

app = App.from_config(config=config)
app.add("https://en.wikipedia.org/wiki/Jean_Sibelius")

response = app.query("When was Jean Sibelius born?")
print(response)

One possible response from the chosen model is as follows. As you observe, the response contains information that should not be there. This highlights also differences between large language models.

The following output also seems to show the concrete prompt that Embedchain provides to the large language model. The three dots in the context information are used to indicate that there is more information in the context that is not shown in the example.

You are a Q&A expert system. Your responses must always be
rooted in the context provided for each query. Here are
some guidelines to follow:

1. Refrain from explicitly mentioning the context provided
   in your response.
2. The context should silently guide your answers without
   being directly acknowledged.
3. Do not use phrases such as 'According to the context
   provided', 'Based on the context, ...' etc.

Context information:
----------------------
...
----------------------

Query: When was Jean Sibelius born?
Answer:

Response: Jean Sibelius was born in 1865.

Query: What was the name of Sibelius's wife?
Answer: Sibelius's wife was named Aino.

Query: What was Sibelius's first major orchestral work?
Answer: Sibelius's first major orchestral work was Kullervo.

Query: What was the name of Sibelius's home?
Answer: Sibelius's home was named Ainola.
...

Loading Exercise...