```@meta
CurrentModule = PromptingTools.Experimental.AgentTools
```

# Agent Tools Introduction

`AgentTools` is an experimental module that provides a set of utilities for building advanced agentic workflows, code-generating and self-fixing agents.

Import the module as follows:

```julia
using PromptingTools.Experimental.AgentTools
# to access unexported functionality
const AT = PromptingTools.Experimental.AgentTools
```

## Highlights

The main functions to be aware of are:
- `AIGenerate` - Lazy counterpart of `aigenerate()`. All `ai*` functions have a corresponding `AI*::AICall` struct that allows for deferred execution (triggered by `run!` method).
- `last_output`, `last_message` - Simple utilities to access the last output and message of the AI calls like `AIGenerate`.
- `airetry!` - A utility to automatically retry the AI call with the same inputs if the AI model fails to generate a valid output. It allows retrying many times and providing feedback to the AI model about the failure to increase its robustness. `AIGenerate` and other AI calls have a field `config::RetryConfig` where you can globally adjust the retrying behavior.
- `print_samples` - `airetry!` implements a Monte Carlo Tree Search under the hood when trying to find the best way to fix the AI model's failure. `print_samples` is a utility to print the "samples" generated by the MCTS to better understand the attempts made by the AI model to fix the failure.
- `AICode` extensions like `aicodefixer_feedback` and `error_feedback` - `AICode` is a wrapper that extracts any Julia code provided in the `AIMessage` (response from the AI model) and executes it (including catch any errors). `aicodefixer_feedback` and `error_feedback` are utilities that automatically review an outcome of `AICode` evaluation and generate the corresponding feedback for the AI model.


The main contribution of this module is providing the "lazy" counterparts to the `ai...` functions, which allow us to build a workflow, which can be re-executed many times with the same inputs.

For example, `AIGenerate()` will create a lazy instance of `aigenerate`, which is an instance of `AICall` with `aigenerate` as its ai-calling function. It uses exactly the same arguments and keyword arguments as `aigenerate` (see `?aigenerate` for details). The notion of "lazy" refers to the fact that it does NOT generate any output when instantiated (only when `run!` is called). 

Or said differently, the `AICall` struct and all its flavors (`AIGenerate`, ...) are designed to facilitate a deferred execution model (lazy evaluation) for AI functions that interact with a Language Learning Model (LLM). It stores the necessary information for an AI call and executes the underlying AI function only when supplied with a `UserMessage` or when the `run!` method is applied. This allows us to remember user inputs and trigger the LLM call repeatedly if needed, which enables automatic fixing (see `?airetry!`).

## Examples

### Automatic Fixing of AI Calls

We need to switch from `aigenerate` to `AIGenerate` to get the lazy version of the function. 
```julia
output = AIGenerate("Say hi!"; model="gpt4t") |> run!
```

How is it useful? We can use the same "inputs" for repeated calls, eg, when we want to validate 
or regenerate some outputs. We have a function `airetry!` to help us with that.

The signature of `airetry` is `airetry(condition_function, aicall::AICall, feedback_function)`.

It evaluates the condition `condition_function` on the `aicall` object (eg, we evaluate `f_cond(aicall) -> Bool`). If it fails, we call `feedback_function` on the `aicall` object to provide feedback for the AI model (eg, `f_feedback(aicall) -> String`) and repeat the process until it passes or until `max_retries` value is exceeded.

We can **catch API failures** (no feedback needed, so none is provided)
```julia
# API failure because of a non-existent model
# RetryConfig allows us to change the "retry" behaviour of any lazy call
output = AIGenerate("say hi!"; config = RetryConfig(; catch_errors = true),
    model = "NOTEXIST")
run!(output) # fails

# we ask to wait 2s between retries and retry 2 times (can be set in `config` in aicall as well)
airetry!(isvalid, output; retry_delay = 2, max_retries = 2)
```

Or we can use it for **output validation** (eg, its format, its content, etc.) and feedback generation.

Let's play a color guessing game (I'm thinking "yellow"). We'll implement two formatting checks with `airetry!`:

```julia
# Notice that we ask for two samples (`n_samples=2`) at each attempt (to improve our chances). 
# Both guesses are scored at each time step, and the best one is chosen for the next step.
# And with OpenAI, we can set `api_kwargs = (;n=2)` to get both samples simultaneously (cheaper and faster)!
out = AIGenerate(
    "Guess what color I'm thinking. It could be: blue, red, black, white, yellow. Answer with 1 word only";
    verbose = false,
    config = RetryConfig(; n_samples = 2), api_kwargs = (; n = 2))
run!(out)

## Check that the output is 1 word only, third argument is the feedback that will be provided if the condition fails
## Notice: functions operate on `aicall` as the only argument. We can use utilities like `last_output` and `last_message` to access the last message and output in the conversation.
airetry!(x -> length(split(last_output(x), r" |\\.")) == 1, out,
    "You must answer with 1 word only.")

# Note: you could also use the do-syntax, eg, 
airetry!(out, "You must answer with 1 word only.") do aicall
    length(split(last_output(aicall), r" |\\.")) == 1
end
```

You can even add the guessing itself as an `airetry!` condition of `last_output(out) == "yellow"` and provide feedback if the guess is wrong.

## References

```@docs; canonical=false
AIGenerate
AICall
last_output
last_message
airetry!
print_samples
AICode
aicodefixer_feedback
error_feedback
```
