ToolCalling actions are no longer being recognised from model response

[sysradium]

@aymeric-roucher

I think it has been another ToolCallingAgent related regression coming from:

e432d41#diff-0319e64c7d0f9e302a7c7a0dd60f04dedb7d4dcf7dbe9e046543335654a9efa1R893

Looks like you made your mind to remove ToolCallingAgent support :)

Basically if a model returns Action:\n{} it does not get mapped into tool_calls here https://github.com/huggingface/smolagents/blob/main/src/smolagents/models.py#L987-L987

And hence it keeps indefinitely failing here: https://github.com/huggingface/smolagents/blob/main/src/smolagents/agents.py#L1066-L1066

I came across this problem by playing with:

from smolagents import (
    LiteLLMModel,
    ToolCallingAgent,
    VisitWebpageTool,
)


model = LiteLLMModel(model_id="lm_studio/meta-llama-3.1-8b-instruct", api_base="http://127.0.0.1:1234/v1", api_key="foo")

search_agent = ToolCallingAgent(
    tools=[VisitWebpageTool()],
    model=model,
    name="search_agent",
    description="This is an agent that can do web search.",
)

manager_agent = ToolCallingAgent(
    tools=[],
    model=model,
    managed_agents=[search_agent],
)
manager_agent.run(
    "What is the current record of the New York Knicks NBA team?  Feel free to search the web for the answer."
)

[aymeric-roucher]

@sysradium e432d41#diff-0319e64c7d0f9e302a7c7a0dd60f04dedb7d4dcf7dbe9e046543335654a9efa1R893 did not change the LiteLLM class, so the issue must date back before: it seems to affect all models where the underlying API does not properly return tool calls, where tool calls must be "manually" parsed.
Thus this issue should be fixed by #991. Let's wait for review and I'll do a patch release!

[sysradium]

@aymeric-roucher largely the PR became redundant. However I decided to update a few things still.

The model quite often jumps between arguments and parameters when returning a json for tool calling. Currently it is a fixed key, but I made tool_arguments_keys a list.

Also before json parsing used to fail if the remainder of the output not only contained some garbage, but hat an invalid json payload, like:

Action:
{
"name": "final_answer",
"parameters": {
    "answer": "The current record of the New York Knicks NBA team is not provided in the given functions."
    }
}

Action
}looks like valid, but invalid{

My version fixes that.

[sysradium]

Sorry, rebased once again instead of merging the main in, as asked by @albertvillanova before.

[aymeric-roucher]

if I understood well the last argument should betool_arguments_keys instead of tool_arguments_key

[sysradium]

I had to keep the name not to break anything. As you can see it's either a list or a single value

[aymeric-roucher]

Interesting to use parity for this! It seems the only case where this message would appear i when the brace count went up and did never go down to zero, i.e. more braces were opened than closed: maybe mention it, as in "caution: not all opened braces were closed?"

[sysradium]

I improved the error reporting as well as the algorithm a bit and added a few more test-cases.