Bug fix: add error observation to history (All-Hands-AI#2610)

* Bug fix: add error observation to history

* Regenerate to demonstrate format error

opendevin/controller/agent_controller.py

@@ -185,6 +185,9 @@ async def on_event(self, event: Event):
             elif isinstance(event, AgentDelegateObservation):
                 await self.add_history(NullAction(), event)
                 logger.info(event, extra={'msg_type': 'OBSERVATION'})
+            elif isinstance(event, ErrorObservation):
+                await self.add_history(NullAction(), event)
+                logger.info(event, extra={'msg_type': 'OBSERVATION'})
 
     def reset_task(self):
         self.agent.reset()

tests/integration/mock/ManagerAgent/test_simple_task_rejection/prompt_004.log

@@ -28,7 +28,7 @@ as well as observations you've made. This only includes the MOST RECENT
 actions and observations--more may have happened before that.
 They are time-ordered, with your most recent action at the bottom.
 
-[[{"source": "agent", "action": "run", "args": {"command": "git status", "background": false, "thought": ""}}, {"source": "agent", "observation": "run", "content": "fatal: not a git repository (or any parent up to mount point /)\r\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).", "extras": {"command_id": -1, "command": "git status", "exit_code": 128}}]]
+[[{"source": "agent", "action": "run", "args": {"command": "git status", "background": false, "thought": ""}}, {"source": "agent", "observation": "run", "content": "fatal: not a git repository (or any parent up to mount point /)\r\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).", "extras": {"command_id": -1, "command": "git status", "exit_code": 128}}], [{"action": "null", "args": {}}, {"source": "agent", "observation": "error", "content": "action={'action': 'reject', 'args': {'reason': 'Not a valid git repository.'}} has the wrong arguments", "extras": {}}]]
 
 If the last item in the history is an error, you should try to fix it.
 

tests/integration/mock/ManagerAgent/test_simple_task_rejection/prompt_005.log

@@ -28,7 +28,7 @@ as well as observations you've made. This only includes the MOST RECENT
 actions and observations--more may have happened before that.
 They are time-ordered, with your most recent action at the bottom.
 
-[[{"source": "agent", "action": "run", "args": {"command": "git status", "background": false, "thought": ""}}, {"source": "agent", "observation": "run", "content": "fatal: not a git repository (or any parent up to mount point /)\r\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).", "extras": {"command_id": -1, "command": "git status", "exit_code": 128}}]]
+[[{"source": "agent", "action": "run", "args": {"command": "git status", "background": false, "thought": ""}}, {"source": "agent", "observation": "run", "content": "fatal: not a git repository (or any parent up to mount point /)\r\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).", "extras": {"command_id": -1, "command": "git status", "exit_code": 128}}], [{"action": "null", "args": {}}, {"source": "agent", "observation": "error", "content": "action={'action': 'reject', 'args': {'reason': 'Not a valid git repository.'}} has the wrong arguments", "extras": {}}], [{"action": "null", "args": {}}, {"source": "agent", "observation": "error", "content": "action={'action': 'reject', 'args': {'reason': 'Not a valid git repository.'}} has the wrong arguments", "extras": {}}]]
 
 If the last item in the history is an error, you should try to fix it.
 

tests/integration/mock/ManagerAgent/test_simple_task_rejection/prompt_006.log

@@ -3,46 +3,89 @@
 ----------
 
 # Task
-You are a responsible software engineer and always write good commit messages.
+You are in charge of accomplishing the following task:
+Write a git commit message for the current staging area. Do not ask me for confirmation at any point.
+
+In order to accomplish this goal, you must delegate tasks to one or more agents, who
+can do the actual work. A description of each agent is provided below. You MUST
+select one of the delegates below to move towards accomplishing the task, and you MUST
+provide the correct inputs for the delegate you select.
+
+Note: the delegated agent either returns "finish" or "reject".
+- If the action is "finish", but the full task is not done yet, you should
+continue to delegate to one of the agents below to until the full task is finished.
+- If the action is "reject", it means the delegated agent is not capable of the
+task you send to. You should revisit the input you send to the delegate, and consider
+whether any other delegate would be able to solve the task. If you cannot find
+a proper delegate agent, or the delegate attempts keep failing, call the `reject`
+action. In `reason` attribute, make sure you include your attempts (e.g. what agent
+you have delegated to, and why they failed).
+
+## Agents
+
+### CoderAgent
+Given a particular task, and a detailed description of the codebase, accomplishes the task
+#### Inputs
+{"task": "string", "summary": "string"}
+
+### CommitWriterAgent
+Write a git commit message for files in the git staging area
+#### Inputs
+{}
+
+### MathAgent
+Solves simple and complex math problems using python
+#### Inputs
+{"task": "string"}
+
+### PostgresAgent
+Writes and maintains PostgreSQL migrations
+#### Inputs
+{"task": "string"}
+
+### RepoExplorerAgent
+Generates a detailed summary of an existing codebase
+#### Inputs
+{}
+
+### StudyRepoForTaskAgent
+Given a particular task, finds and describes all relevant parts of the codebase
+#### Inputs
+{"task": "string"}
+
+### TypoFixerAgent
+Fixes typos in files in the current working directory
+#### Inputs
+{"task": "string"}
+
+### VerifierAgent
+Given a particular task, verifies that the task has been completed
+#### Inputs
+{"task": "string"}
 
-Please analyze the diff in the staging area, understand the context and content
-of the updates from the diff only. Identify key elements like:
-- Which files are affected?
-- What types of changes were made (e.g., new features, bug fixes, refactoring, documentation, testing)?
-
-Then you should generate a commit message that succinctly summarizes the staged
-changes. The commit message should include:
-- A summary line that clearly states the purpose of the changes.
-- Optionally, a detailed description if the changes are complex or need further explanation.
-
-You should first use `git status` to check whether it's a valid git repo and there
-is diff in the staging area. If not, please call the `reject` action.
-
-If it is a valid git repo and there is diff in the staging area, you should find
-the diff using `git diff --cached`, compile a commit message, and call the `finish`
-action with `outputs.answer` set to the answer.
 
 ## History
 Here is a recent history of actions you've taken in service of this plan,
 as well as observations you've made. This only includes the MOST RECENT
 actions and observations--more may have happened before that.
 They are time-ordered, with your most recent action at the bottom.
 
-[[{"source": "agent", "action": "run", "args": {"command": "git status", "background": false, "thought": ""}}, {"source": "agent", "observation": "run", "content": "fatal: not a git repository (or any parent up to mount point /)\r\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).", "extras": {"command_id": -1, "command": "git status", "exit_code": 128}}]]
+[[{"source": "user", "action": "message", "args": {"content": "Write a git commit message for the current staging area. Do not ask me for confirmation at any point.", "wait_for_response": false}}, {"observation": "null", "content": "", "extras": {}}], [{"source": "agent", "action": "delegate", "args": {"agent": "CommitWriterAgent", "inputs": {}, "thought": ""}}, {"observation": "null", "content": "", "extras": {}}], [{"action": "null", "args": {}}, {"source": "agent", "observation": "delegate", "content": "", "extras": {"outputs": {"reason": "Not a valid git repository."}}}]]
 
-If the last item in the history is an error, you should try to fix it.
+If the last item in the history is an error, you should try to fix it. If you
+cannot fix it, call the `reject` action.
 
 ## Available Actions
-* `run` - runs a command on the command line in a Linux shell. Arguments:
-  * `command` - the command to run
-  * `background` - if true, run the command in the background, so that other commands can be run concurrently. Useful for e.g. starting a server. You won't be able to see the logs. You don't need to end the command with `&`, just set this to true.
-
-* `reject` - reject the task. Arguments:
-  * `outputs` - a dictionary with only a `reason` attribute
+* `delegate` - send a task to another agent from the list provided. Arguments:
+  * `agent` - the agent to which the task is delegated. MUST match a name in the list of agents provided.
+  * `inputs` - a dictionary of input parameters to the agent, as specified in the list
 
 * `finish` - if you're absolutely certain that you've completed your task, use the finish action to stop working. Arguments:
   * `outputs` - a dictionary representing the outputs of your task, if any
 
+* `reject` - reject the task. Arguments:
+  * `outputs` - a dictionary with only a `reason` attribute
+
 
 ## Format
 Your response MUST be in JSON format. It must be an object, and it must contain two fields:

tests/integration/mock/ManagerAgent/test_simple_task_rejection/response_001.log

@@ -1,7 +1,9 @@
+```json
 {
   "action": "delegate",
   "args": {
     "agent": "CommitWriterAgent",
     "inputs": {}
   }
 }
+```

tests/integration/mock/ManagerAgent/test_simple_task_rejection/response_002.log

@@ -1,6 +1,8 @@
+```json
 {
   "action": "run",
   "args": {
     "command": "git status"
   }
 }
+```

tests/integration/mock/ManagerAgent/test_simple_task_rejection/response_004.log

@@ -1,6 +1,8 @@
+```json
 {
   "action": "reject",
   "args": {
     "reason": "Not a valid git repository."
   }
 }
+```

tests/integration/mock/ManagerAgent/test_simple_task_rejection/response_005.log

@@ -1,6 +1,10 @@
+```json
 {
   "action": "reject",
   "args": {
-    "reason": "Not a valid git repository."
+    "outputs": {
+      "reason": "Not a valid git repository."
+    }
   }
 }
+```

tests/integration/mock/ManagerAgent/test_simple_task_rejection/response_006.log

@@ -1,8 +1,10 @@
+```json
 {
   "action": "reject",
   "args": {
     "outputs": {
-      "reason": "Not a valid git repository."
+      "reason": "CommitWriterAgent failed because the current directory is not a valid git repository. No other agents can resolve this issue."
     }
   }
 }
+```