RFC: inspecting actor messages and states for testing and debuging

[contrun]

Is your feature request related to a problem? Please describe.
When developing complex actor systems using Ractor, I find it challenging to debug actor behavior, verify correct message handling, and ensure proper state transitions. Take the PingPong actor from current README file as an example, I frequently need to verify the actor received and handled the message, most often, I need to have previous state, message to process and the next state to verify if the state transition is functioning as expected.

Describe the solution you'd like
We propose adding built-in support for actor inspection and debugging in Ractor. This could involve:

Adding a pre_start, post_handle, post_stop hook to the original actor implementation, and create an interface for the downstream users to mock current actor behavior but also to allow them to pass a data structure against which some callbacks will be executed.

To illustrate, I created a Inspector data structure in https://github.com/contrun/ractor-inspector, which accepts an underlying Actor and a InspectorPlugin. The Inspector implements the Actor trait and forwards all the messages it received to the underlying actor, moreover, all the myself references of the underlying actor are changed to this Inspector (have to be done with some code change in the actor implementation). Whenever the underlying actor stops handle messages, we call the message_handled function below (with message and state passed) of the InspectorPlugin. The InspectorPlugins can do anything like dumping the messages or states, or even have its own state to assure the state transition of the underlying is as expected.

pub trait InspectorPlugin {
    type ActorState: ractor::State;
    type ActorMessage: ractor::Message;

    fn actor_started(&mut self, actor_state: &mut Self::ActorState);
    fn message_handled(&mut self, actor_state: &mut Self::ActorState, message: Self::ActorMessage);
    fn actor_stopped(&mut self, actor_state: &mut Self::ActorState);
}

I imagine this would be particularly useful for the developers to debug and test their code. I implemented two plugins to debug print the messages and assert state transition is functioning in https://github.com/contrun/ractor-inspector/blob/main/src/tests.rs .

Describe alternatives you've considered
As I have mentioned, we've implemented a workaround called Ractor Inspector https://github.com/contrun/ractor-inspector , which acts as a mediator between the actor and its environment. It intercepts messages and provides hooks for custom plugins. However, this approach requires actor modifications and uses unsafe Rust code (multiple references to the mutable state).

Additional context
https://github.com/contrun/ractor-inspector for my current workaround to achieve the desired functionality.

[slawlor]

So normally for something like this, my approach is the following.

1. Add message types which are gated by #[cfg(test)] to probe the actor's state as needed to verify proper transitions.
2. Mock all dependent actors, with "fake" actors, that simply have the same message type.
3. If needed, thread in an atomic or mutex-guarded state object to the actor under test to verify a transition of the right type occurred (see factory which uses this pattern a lot to verify the right number of messages were processed at various points, etc).

The great part about actors in Ractor, is that as long as your message types are consistent, you can "mock" any actor quite trivially. For example, assume you have two actors A and B. As part of As normal operation, it's going to query some state from B and do a state transition accordingly.

Instead of having to create the "production" versions of A and B, I can simply create a "mock" B which just has the same message type. A doesn't know it's not talking to a real instance of B, just that the queries support the right format (which is the goal). This lets me artificially modify Bs replies as I see fit in order to test the functionality of A.

I still need something to "inspect" A which is a problem you elude to, and in this case, a common enough pattern that doesn't impact production code is to do the following

enum AsMessage {
   Something(u64),
   SomethingElse(u128),
   #[cfg(test)]
  InspectState(RpcReplyPort<AsState>),
}

which will only be compiled into the test-target, and I can use that in my test to query the actor's state (and verify whatever I need). I don't need to read all the state, but perhaps only some of it, or whatever I need to verify. That's up to the designer.

I don't know how I feel yet about what you've proposed, as I'm worried this kind of logic might leak into more production, risky code, but let me stew on it a bit.

[slawlor]

If you want to put up a PR (even a partial/broken one to show the idea) I'm more than happy to take a look!