Integrate custom tools into an AI Agent

By benperk · · , , , ,

Before learning what I discovered while integrating tools into an AI Agent, I was a bit perplexed regarding how the LLM was able to determine which tool to use and even which tools are available for use.  Well, how it works is not a mystery, it is clearly described in the code.  In this post I will walk you through how to integrate a custom tool into an AI Agent and describe in some detail how it works.  The steps are:

  • Program the tools with functional code
  • Define the tool name, description, and parameters
  • Create an AI client, add the tool list and instructions
  • Analyze the response
  • Execute the specified tools
  • Send to AI endpoint again for Natural Language Processing (NLP)

Program the tools with functional code

This was an unknown before I dug into it but is more obvious after completing this activity.  You see, I was thinking that the “AI” was somehow dynamically discovering tools to respond to the user_prompt and also dynamically building and executing the logic for those tools.  Which is not the case. Every tool which you need AI to use must be coded.  Take for example I needed my AI Agent to find upcoming events, then I would code a function like the following.


def next_event(location: str) -> str:
    """Returns the next event for a location."""
    today = int(datetime.now().strftime("%m%d"))
    loc = location.lower().replace(" ", "_")

    # Retrieve the next event from the location
    for name, event_type, date, date_str, locs in EVENTS:
        if loc in locs and date >= today:
            return json.dumps({"event": name, "type": event_type, "date": 
		date_str, "visible_from": sorted(locs)})

    return json.dumps({"message": f"No upcoming events found for {location}."})
Listing 1: custom AI Agent tool

Make an assumption that the data stored in EVENTS is pulled from a data source containing all relevant upcoming events prior to the execution of this function.

Define the tool name, description, and parameters

In this example, I am using the Azure AI Projects client library for Python – version 2.5.0 which is important the FunctionTool class contained within it is utilized to define my event tool.  The tool is given a name, a description, parameters, and a required attribute that lets you make any of the properties required.  In the example there is only a single property named, location, however there could be many, and they may or may not be required.  To add an additional property you would add a new attribute with a name, type, and description.  If it is required then add the name to the list of required properties, if not required do not add it to the list.


event_tool = FunctionTool(
	name="next_event",
        description="Get the next event in a given location.",
        parameters={
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "continent to find the next event in 
			(e.g. 'north_america', 'south_america', 'australia')",
                },
            },
            "required": ["location"],
            "additionalProperties": False,
        },
        strict=True,
)
Listing 2: custom AI Agent tool bound to FunctionTool

It is common to have multiple custom tools available to your agents, each one must be bound to the FunctionTool class as described previously.

Create an AI client, add the tool list and instructions

This is where the magic starts to happen.  This is where you create the AI Agent and intentionally identify the tools available to it.  There is no magic quit yet, because the AI Agent is not simply going to infer the user_prompt and discover tools, they are specifically defined and provided to it, as seen in the previous 2 code lists.  The following code illustrates how the tools are bound to the AI Agent.  One important item to call out is the instruction, which I’d like to refer to as the system prompt.  This short set of instructions explains the role the AI Agent will play and that there are some provided custom tools for helping render a response.


from azure.ai.projects import AIProjectClient
with (
        DefaultAzureCredential() as credential,
        AIProjectClient(endpoint=project_endpoint, credential=credential) 
		as project_client,
        project_client.get_openai_client() as openai_client,
    ):

agent = project_client.agents.create_version(
    agent_name="event-agent",
    definition=PromptAgentDefinition(
        model=model_deployment,
        instructions=
            """You are an event assistant that helps users find 
            information about events. Use the available tools to 
	    assist users with their inquiries.""",
        tools=[event_tool],
    ),
)
Listing 3: custom AI Agent tool, creating the agent

Notice that the AI Agent has been given a name, event-agent, there is a model_deployment which defines the model, in this case it’s gpt-5-mini, and the instructions.  The highlight of the AI Agents instantiation is the array of tools it can use to help render a response to the user_prompt.    Recognize that you cannot simply program a single AI Agent, ask it anything, and expect a perfect response.  Additionally, the user_prompt must contain the required attributes, in this case a location, as shown in List 2.  Here is an example of user_prompt that a human would enter that this event-agent could answer.


Find me the next event I can see from Europe.

This user_prompt will be captured and sent to gpt-5-mini along with the tools and the instructions.

Analyze the response

Between sending the response and analyzing the response is actually where the magic happens.  Some of it happens in the Azure AI Projects client library code, but most happens in the Large Language Model (LLM), gpt-5-mini.  The result of the response is:


ResponseFunctionToolCall(
    arguments='{
        "location":"europe"
        }', 
        name='next_event', 
        type='function_call', 
        status='completed', 
        agent_reference={
            'type': 'agent_reference', 
            'name': 'event-agent', 
            'version': '1'
        })
Listing 4: custom AI Agent tool, LLM response #1

You notice that the LLM was able to discover the provided location delivered with the user_prompt, in this case is Europe.  It also identified which custom tool should be used to determine the next upcoming event, next_event, defined in Listing 1.  Two things.  First, if the LLM was unable to identify a location in the user_prompt the code that comes next must handle that and respond appropriately.  Second, in this example there is only a single tool, event_tool named next_event, however, had there been more than 1 tool and the required properties provided in the prompt, then multiple ResponseFunctionToolCall() properties would have been returned in the response.  The code you write next must handle every scenario as they exist.  “AI” does not magically do this.

Execute the specified tools

To retrieve the single function_call type details from the LLM response, use something like the following.


function_call = 
    next((item for item in response.output if item.type == "function_call"), None)

if function_call:
    function_name = function_call.name

    if function_name == "next_visible_event":
        result = next_visible_event(**json.loads(function_call.arguments))

    input_list.append(FunctionCallOutput(
            type="function_call_output",
            call_id=function_call.call_id,
            output=result,
        )
    )
Listing 5: custom AI Agent tool, parse JSON, get the function_name, execute function

In this case let’s assume that function_call.name does equal next_event.  In this case the next_event function from Listing 1 is executed, passing the location argument shown in Listing 4, which is Europe.  Using this event data stored in EVENTS from Listing 1, accept that the discovered event is the following.


AI Event|AI|05-01|north_america;south_america;europe;asia;africa;australia

That upcoming event is stored into the input_list.

Send to AI endpoint again for Natural Language Processing (NLP)

Now pulling it all together I realized that everything was not performed in a single call to the LLM endpoint.  Rather, multiple round trips are required.  This means that the LLM is not executing code, it’s not writing code dynamically, and it is not discovering available tools on the go.  Instead, the LLM is used for:

  • Identifying function arguments contained within a user_prompt
  • Linking the existing arguments to the provided tools list
  • Providing a JSON response with the function name and associated arguments
  • 2nd call: Render a NLP response using the output of the executed tool functions

That is a lot and magical.  Consider having to code for those outcomes which would be very complex and unlikely to ever be successful with code. That is the sweet spot and what makes this AI / LLM capability so innovative.


if input_list:
  response = openai_client.responses.create(
    input=input_list,
    previous_response_id=response.id,
    extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},)
print(f"AGENT: {response.output_text}")
Listing 6: custom AI Agent tool, call LLM again with response id and result of tool function call

Notice that the session context is maintained by passing the response.id with the 2nd inference to the LLM.  The response.id is included in the response shown in Listing 4 but removed for simplicity.  The output from this call to the LLM would render something similar to the following.


AGENT: The next AI event you can attend in Europe is ‘The best AI event in the world’, 
taking place on May 1st.