Summary Buffer Memory¶
The same CassandraChatMessageHistory
can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.
from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain
from cqlsession import getCQLSession, getCQLKeyspace
cqlMode = 'astra_db' # 'astra_db'/'local'
session = getCQLSession(mode=cqlMode)
keyspace = getCQLKeyspace(mode=cqlMode)
message_history = CassandraChatMessageHistory(
session_id='summarized-conversation-4321',
session=session,
keyspace=keyspace,
ttl_seconds = 3600,
)
message_history.clear()
Below is the logic to instantiate the LLM of choice. We choose to leave it in the notebooks for clarity.
from llm_choice import suggestLLMProvider
llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI' ... manually if you have credentials)
if llmProvider == 'GCP_VertexAI':
from langchain.llms import VertexAI
llm = VertexAI()
print('LLM from VertexAI')
elif llmProvider == 'OpenAI':
from langchain.llms import OpenAI
llm = OpenAI()
print('LLM from OpenAI')
else:
raise ValueError('Unknown LLM provider.')
LLM from OpenAI
Use the memory with summary in a Chain¶
memory = ConversationSummaryBufferMemory(
llm=llm,
chat_memory=message_history,
max_token_limit=180,
)
summaryConversation = ConversationChain(
llm=llm,
memory=memory,
verbose=True
)
With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.
summaryConversation.predict(input='Tell me about William Tell, the hero')
> Entering new ConversationChain chain... Prompt after formatting: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: Human: Tell me about William Tell, the hero AI: > Finished chain.
" William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs. The most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. William Tell is a symbol of freedom and independence in Switzerland and is often depicted in statues, paintings, books, and films."
summaryConversation.predict(
input="Any relation to the apple from Cupertino?"
)
> Entering new ConversationChain chain... Prompt after formatting: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: Human: Tell me about William Tell, the hero AI: William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs. The most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. William Tell is a symbol of freedom and independence in Switzerland and is often depicted in statues, paintings, books, and films. Human: Any relation to the apple from Cupertino? AI: > Finished chain.
" No, I don't believe there is any relation between the two apples. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products."
No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:
memory.moving_summary_buffer
"\nThe human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler."
Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:
summaryConversation.predict(
input="I heard the two apples may be in fact the very same fruit."
)
> Entering new ConversationChain chain... Prompt after formatting: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: System: The human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. Human: Tell me about William Tell, the hero AI: William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs. The most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. William Tell is a symbol of freedom and independence in Switzerland and is often depicted in statues, paintings, books, and films. Human: Any relation to the apple from Cupertino? AI: No, I don't believe there is any relation between the two apples. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products. Human: I heard the two apples may be in fact the very same fruit. AI: > Finished chain.
" I'm not aware of any evidence that suggests the two apples are the same fruit. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products."
This is where the current summary buffer is stored
(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)
memory.moving_summary_buffer
"\nThe human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler."
The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory
at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer
.
print(memory.predict_new_summary(
memory.chat_memory.messages,
memory.moving_summary_buffer,
))
The human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. The human then inquires if there is any relation between the apple from Cupertino and William Tell's apple, to which the AI states that there is no evidence to suggest that the two apples are the same.
summaryConversation.predict(
input='Ok, enough with apples. Let\'s talk starfish now.'
)
> Entering new ConversationChain chain... Prompt after formatting: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: System: The human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. Human: Tell me about William Tell, the hero AI: William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs. The most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. William Tell is a symbol of freedom and independence in Switzerland and is often depicted in statues, paintings, books, and films. Human: Any relation to the apple from Cupertino? AI: No, I don't believe there is any relation between the two apples. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products. Human: I heard the two apples may be in fact the very same fruit. AI: I'm not aware of any evidence that suggests the two apples are the same fruit. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products. Human: Ok, enough with apples. Let's talk starfish now. AI: > Finished chain.
" Sure! Starfish are a type of echinoderm that are found in all the world's oceans. They have a radial symmetry and a soft body that is divided into five or more sections. Starfish can range in size from small 1-inch long species to large species that can measure up to 3 feet across. They feed on mollusks, crustaceans, and other small organisms, and some species are even capable of regenerating a new arm if one is lost or damaged."
memory.moving_summary_buffer
"\nThe human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. The human then inquires about any connection between the apple from Cupertino and William Tell's apple, to which the AI responds that there is no evidence suggesting the two apples are the same fruit."
Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.
summaryConversation.predict(
input='I hear that some of those destroy marine ecosystems.'
)
> Entering new ConversationChain chain... Prompt after formatting: The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: System: The human asks the AI about William Tell, the hero. The AI responds that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow, and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. The human then inquires about any connection between the apple from Cupertino and William Tell's apple, to which the AI responds that there is no evidence suggesting the two apples are the same fruit. Human: Tell me about William Tell, the hero AI: William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs. The most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. William Tell is a symbol of freedom and independence in Switzerland and is often depicted in statues, paintings, books, and films. Human: Any relation to the apple from Cupertino? AI: No, I don't believe there is any relation between the two apples. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products. Human: I heard the two apples may be in fact the very same fruit. AI: I'm not aware of any evidence that suggests the two apples are the same fruit. The apple of William Tell's legend is an apple that is shot off his son's head with a crossbow, while the apple from Cupertino, California is a technology company known for producing the Mac computers, iPhones, and other products. Human: Ok, enough with apples. Let's talk starfish now. AI: Sure! Starfish are a type of echinoderm that are found in all the world's oceans. They have a radial symmetry and a soft body that is divided into five or more sections. Starfish can range in size from small 1-inch long species to large species that can measure up to 3 feet across. They feed on mollusks, crustaceans, and other small organisms, and some species are even capable of regenerating a new arm if one is lost or damaged. Human: I hear that some of those destroy marine ecosystems. AI: > Finished chain.
' Yes, some species of starfish, such as the crown-of-thorns starfish, can cause significant damage to coral reefs and other marine ecosystems. The crown-of-thorns starfish feeds on corals, sponges, and other invertebrates, and has caused extensive damage to reefs in the Indo-Pacific region.'
memory.moving_summary_buffer
"\nThe human asked the AI about William Tell, the hero. The AI responded that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. The AI stated there is no evidence suggesting any relation between the apple from Cupertino and William Tell's apple. When the human asked about any connection between them, the AI said there was no evidence suggesting the two apples are the same fruit. The conversation then shifted to starfish, with the AI explaining that they are a type of echinoderm found in all the world's oceans, have a radial symmetry and a soft body divided into five or more sections, and can range in size from small 1-inch long species to large species that can measure up to 3 feet across. They feed on mollusks, crustaceans, and other small"
From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation in a fixed amount of text.
print(memory.predict_new_summary(
memory.chat_memory.messages,
memory.moving_summary_buffer,
))
The human asked the AI about William Tell, the hero, with the AI responding that William Tell is a folk hero from Switzerland who is best known for shooting an apple off his son's head with a crossbow and is a symbol of freedom and independence in Switzerland often depicted in statues, paintings, books, and films. He is also known for his resistance to the oppressive rule of the Austrian Habsburgs, as the most popular version of his story is that he was forced to shoot an apple off his son's head as a test of his marksmanship by the tyrannical Habsburg governor Gessler. The AI stated there is no evidence suggesting any relation between the apple from Cupertino and William Tell's apple. When the human asked about any connection between them, the AI said there was no evidence suggesting the two apples are the same fruit. The conversation then shifted to starfish, with the AI explaining that they are a type of echinoderm found in all the world's oceans, have a radial symmetry and a soft body divided into five or more sections, and can range in size from small 1-inch long species to large species that can measure up to 3 feet across. They feed on mollusks, crustaceans, and other