I remember way back when I was coding web applications that ran on a single server that as the consumption and usage of that web application grew, we ran out of CPU and memory.  This required us to add another server and setup a, what we call now, a web farm.  But what happened was that when users bounced between the servers running the same website, they had to login again and any knowledge of their previous data entries were lost.  That’s because we would store session information in cookies or in session objects (server-side memory), and when the user was routed to another server, the session didn’t exist there.

That web based scenario is very similar to this Agentic one, in that there is a linked which needs to be maintained between the Agents in a multi-agent solution and/or between the human and the Agent.  Maintaining session and context is very important so that the Agents which consume inference from LLMs continue to share grounding information to keep previous decisions, conclusions, and knowledge accessible as the conversation and actions progress forward.  I have seen that over time as the context window grows, grounding content seeps away, same like it does for human minds.  It is wise to track and manage grounding knowledge often, as it is not intuitively obvious when the expected content is no longer present in the session and inference is performed without it.  As we know, LLMs are very good at sounding correct, and if we assume the inference is being executed within a given context but does not contain the expected grounding, then we might get a great written answer, but its wrong or not as correct as it should be.

NOTE: The azure.ai.agents library is a new and evolving SDK from Microsoft.  v1 makes a thread synonymous with a session, while v2 defines it to a conversation.

State persistence

There are solutions for managing session and state.  Maintaining the connection between Agents and the Humans/Agents to perform short bits of work is fine, so long as the context is not needed after the work is done.  In this scenario, no retrievable history or grounding knowledge is accessible after the conversation has ended. Which leads to a need of a persistence layer.  Firstly, I recommend always to use a Microsoft solution if one exists, for example Memory in Microsoft Foundry Agent Service will persist session through an API hosted on Microsoft Foundry.  Another solution, if you need some custom feature not from Memory, you can use an Azure Cosmos DB.  Azure Cosmos DB is a very durable, query able state storage for long-lived AI workflows.  Another possible solution to manage the state is Azure Table Storage.  In both cases you associate the unique thread.id (v1) or the conversation.id (v2) with the session context, and retrieve it, again via API prior to each request to the Agent and the LLM for processing and inference.

• With each prompt/instruction execution you would store the session.id, conversation.id, message.id, role, user prompt, response, and a timestamp.
• With each subsequent prompt/instruction will have it stored for retrieval, in case it gets emptied or lost.

Each of these requires some code and formatting for storage.  For retrieval, you would first want to check if history exists and that it is the amount of session history you want/need, only retrieve if it does not match your expected.

Context management

For long running workflows the context history can get large.  As the number of tokens increase so does the cost, therefore, the context history must be managed.  Not only cost is impacted by large context windows, latency and overflow can also occur.  Here are a few techniques for managing context growth.

NOTE: 4 characters is roughly equal to 1 token.  You can use that to estimate token size and potentially the cost.

Sliding window retention – only include the most recent messages and responses.  Continue to store them all, but when forming the next response, remove older messages.

Selective retention with summarization – summarizing responses does reduce token size, consider prompting specifically for summaries or important details and store only those.

Importance-weighted pruning – as an additional prompt instruction, specifically receive a priority score that identifies the importance of the response and its usefulness for context.  If the score meets or exceeds the threshold, to be determined by you, store it.

NOTE: When you attach a file for analysis or grounding, consider storing it on OneDrive or Azure Storage for later grounding support used with conversations, if found relevant.

Here is some great additional information on this topic.