A few weird runtime errors

[leonj1]

Not sure how to set up a reproducible setup.

Here is the command I ran:

ra-aid -m "Run 'make test' and fix errors"  --provider anthropic --model claude-3-5-haiku-20241022 --cowboy-mode

Here are the errors:

Error: 1 validation error for emit_key_snippets snippets Field required [type=missing, input_value={}, input_type=dict] For further information visit    │
│ https://errors.pydantic.dev/2.10/v/missing Please fix your mistakes.

another similar error which unclear if they are related

Error: 1 validation error for run_shell_command command Field required [type=missing, input_value={}, input_type=dict] For further information visit     │
│ https://errors.pydantic.dev/2.10/v/missing Please fix your mistakes.

[ai-christianson]

Is this on master or a release?

[leonj1]

This is from the release.

[ai-christianson]

Ah, actually I have seen these errors. I believe this is actually the AI not correctly formatting the tool call.

It will often resolve itself, but often get into a doom loop.

It's something we should study/gather more data on. For one, we have next to no logging, just the UI outputs.

We should open a new issue/feature request to add a logging system which would essentially use standard Python logging facilities to output much more detailed logging to a log file.

[ai-christianson]

Added logging feature issue here: #8

We can leave this issue open bc it's definitely a real issue and hopefully once #8 is implemented, we can get some logs to look into this issue more.

In the meantime, it'd be great to report any additional info or anecdotes here.

[leonj1]

Makes sense. I'll give it a try.

[ai-christianson]

Something else I wanted to note about this:

I briefly changed the default model temperature to 0 in order to have more deterministic behavior of the agent, but realized that was a mistake because temp 0 makes it impossible for the agent to have some randomness and work its way out of issues like this.

I have seen cases where it makes the wrong function parameters 1-5 times but recovers. I think that having non zero temp is crucial for this recovery to be possible.

My current working theory for this issue in general is that it may be part of the normal course of operations that some models will fail to properly make function calls some percentage of the time. It may therefore end up being an expected operating condition. If so, we could address this by having a better UI representation of what is going on, e.g. not make it look like a major error occurred and represent it more as "the agent made a bad function call and is trying to correct itself."

[leonj1]

Makes sense. Curious how aider handles this since I've not seen these during usage. It likely does happen but gets handled internally.

[ai-christianson]

Is aider using function calling? I'm not sure how it works internally.

But I have seen models in aider fail to format the edit correctly and it retries multiple times. Aider has a default limit of 3 retries then it will abort.

[ariel-frischer]

Is there a way to integrate it with with_structured_output? Also with_retry. I've used this in another langgraph system and it improved on the output I specified, haven't tried it for tool calls though. Maybe this helps: https://python.langchain.com/docs/concepts/tool_calling/#best-practices

[ariel-frischer]

For reference here are some of the error messages I was getting:

Tool Error: Error executing code: invalid syntax (, line 4)
Tool Error: Error executing code: unterminated string literal (detected at line 1) (, line 1)
Tool Error: Error executing code: /tmp/tmplwzqokro/sympy/sympy
Tool Error: Error executing code: unmatched ')' (, line 1) │