Tools
Tool functions in Braintrust allow you to define general-purpose code that can be invoked by LLMs to add complex logic or external operations to your workflows. Tools are reusable and composable, making it easy to iterate on assistant-style agents and more advanced applications. You can create tools in TypeScript or Python, and deploy them across the UI and API via prompts.
Creating a tool
Currently, you can only create tools via code. To create a tool, use project.tool.create
and pick a name and unique slug for your tool:
Using tools
Most models support tool calling, which allows you to define tools with well-defined input and output types. They are commonly used for two purposes:
- To enable models to "call" tools that perform external tasks, and then use those results to produce a final response
- To coerce a model into production structured outputs that match a given JSON schema
Braintrust supports both use cases.
Calling external tools
Braintrust allows you to define custom tools that can be called securely during prompt execution. You can use tools to create simple and composable agents that perform tasks like web-scraping, retrieval augmented generation (RAG), API execution, and much more.
Custom tools use the OpenAI tool calling format which means they are automatically supported by most models including OpenAI, Anthropic, and modern open-source LLMs, while following well-established industry standards.
Let's walk through an example. The following tool looks up information about the most recent commit in a GitHub repository:
If you save this file locally to github.ts
, you can run
to push the function to Braintrust. Once the command completes, you should see the function listed in the Library's Tools tab.
Currently, tools must be defined in TypeScript, but we are working on adding Python support.
To use a tool, simply select it in the Tools dropdown in your Prompt window. Braintrust will automatically:
- Include it in the list of available tools to the model
- Invoke the tool if the model calls it, and append the result to the message history
- Call the model again with the tool's result as context
- Continue for up to (default) 5 iterations or until the model produces a non-tool result
Structured outputs
To define a set of tools available to a model, expand the Tools dropdown and select the Raw tab. You can enter an array of tool definitions, following the OpenAI tool format.
By default, if a tool is called, Braintrust will return the arguments of the first tool call as a JSON object. If you use the invoke
API,
you'll receive a JSON object as the result.
If you specify parallel
as the mode, then instead of the first tool call's arguments, you'll receive an array of all tool calls including
both function names and arguments.