Anjin

This is an autopilot tool for games made with Unity. It consists of the following two elements.

1. A dispatcher that launches the corresponding Agent according to the Scene loaded in the game
2. An Agent that realizes auto-execution closed to the Scene

Agents are small, isolated C# scripts that perform specific operations, such as playback of UI operations or monkey tests. In addition to those provided built-in, game title-specific ones can be implemented and used.

Click 日本語 for the Japanese page if you need.

Installation

You can choose from two typical installation methods.

Install via Package Manager window

1. Open the Package Manager tab in Project Settings window (Editor > Project Settings)
2. Click + button under the Scoped Registries and enter the following settings (figure 1.):
   1. Name: package.openupm.com
   2. URL: https://package.openupm.com
   3. Scope(s): com.dena, com.cysharp, and com.nowsprinting
3. Open the Package Manager window (Window > Package Manager) and select My Registries in registries drop-down list (figure 2.)
4. Click Install button on the com.dena.anjin package

Figure 1. Package Manager tab in Project Settings window.

Figure 2. Select registries drop-down list in Package Manager window.

Note

Do not forget to add com.cysharp and com.nowsprinting into scopes. These are used within Anjin.

Note

Required install Unity Test Framework package v1.3 or later for running tests (when adding to the testables in package.json).

Install via OpenUPM-CLI

If you installed openupm-cli, run the command below:

openupm add com.dena.anjin

UNITY_INCLUDE_TESTS || DENA_AUTOPILOT_ENABLE is set in the Define Constraints of the Assembly Definition File, So excluded from the release build in principle.

Recommended .gitignore

The following files are automatically generated when you start Anjin. There is no need to track it, so we recommend adding it to your project's .gitignore file.

/Assets/AutopilotState.asset*

Settings for game title

After installing the UPM package in the game title project, configure and implement the following.

• Generate and configure the AutopilotSettings.asset file
• Generate and configure the .asset file for the Agent to be used
• Implement a custom Agent if necessary
• Implement initialization process as needed

Note

Set UNITY_INCLUDE_TESTS || DENA_AUTOPILOT_ENABLE in the Define Constraints of the Assembly Definition File to which they belong to exclude them from release builds.

Generate and configure the AutopilotSettings.asset file

Right-click in the Unity Editor's Project window to open the context menu, then select Create > Anjin > Autopilot Settings to create the file. The file name is arbitrary, and multiple files can be created and used within a project.

There are three main settings.

Agent Assignment

Scene Agent Mapping and Fallback Agent

Set up a mapping of Agent configuration files (.asset) to be automatically executed for each Scene.

Set the combination of Scene and Agent. The order does not affect the operation. If a Scene does not exist in the list, the Agent set in Fallback Agent will be used.

For example, if you want UGUIMonkeyAgent to work for all Scenes, set Scene Agent Maps to empty, and set Set the Fallback Agent to an UGUIMonkeyAgent's .asset file.

Observer Agent

Set the Agents to always be launched in parallel, independent of Scene Agent Maps and Fallback Agent. As of v1.0.0, EmergencyExitAgent is assumed to be used. If you need to launch multiple Agents in parallel, use ParallelCompositeAgent.

Note that the Agents set here will be destroyed and created each time a new Scene is loaded. It is NOT set to DontDestroyOnLoad.

Autopilot Run Settings

This item can also be overridden from the commandline (see below).

Lifespan

Specifies the execution time limit in seconds. Defaults to 300 seconds, 0 specifies unlimited operation

Random Seed

Specify when you want to fix the seed given to the pseudo-random number generator (optional). This is a setting related to the pseudo-random number generator used by the autopilot. To fix the seed of the pseudo-random number generator in the game itself, it is necessary to implement this setting on the game title side.

Time Scale

Time.timeScale. Default is 1.0

JUnit Report Path

Specifies the JUnit format report file output path (optional). If there are zero errors and zero failures, the autopilot run is considered to have completed successfully.

Logger

Logger used for this autopilot settings. If omitted, Debug.unityLogger will be used as default.

Reporter

Reporter that called when some errors occurred in target application

Error Handling Settings

Set up a filter to catch abnormal log messages and notify using Reporter.

Handle Exception

Report when exception is detected in log

Handle Error

Report when error message is detected in log

Handle Assert

Report when assert message is detected in log

Handle Warning

Report when warning message is detected in log

Ignore Messages

Log messages containing this string will not be report

Generate and configure the Agent setting file (.asset)

Whether you use the built-in Agent or implement custom Agent, you must create an instance (.asset file) of it in the Unity editor.

Instances are created by right-clicking in the Project window of the Unity editor to open the context menu, then selecting Create > Anjin > Agent type name.

Select the generated file, Agent-specific settings are displayed in the inspector and can be customized. You can prepare multiple Agents with different settings for the same Agent type and use them in different Scenes.

Generate and configure the Logger setting file (.asset)

Logger instances are created by right-clicking in the Project window of the Unity editor to open the context menu, then selecting Create > Anjin > Logger type name.

Select the generated file, Logger-specific settings are displayed in the inspector and can be customized. You can prepare multiple Loggers with different settings for the same Logger type.

Generate and configure the Reporter setting file (.asset)

Reporter instances are created by right-clicking in the Project window of the Unity editor to open the context menu, then selecting Create > Anjin > Reporter type name.

Select the generated file, Reporter-specific settings are displayed in the inspector and can be customized. You can prepare multiple Reporters with different settings for the same Reporter type.

Run autopilot

Autopilot can be executed in two ways.

1. Run on play mode in Unity editor (GUI)

Open the AutopilotSettings file you wish to run in the inspector and click the Run button to enter play mode with Autopilot enabled. After the set run time has elapsed, or as in normal play mode, clicking the Play button will stop the program.

2. Run from Play Mode test

Autopilot works within your test code using the async method LauncherFromTest.AutopilotAsync(string). Specify the AutopilotSettings file path as the argument.

[Test]
public async Task LaunchAutopilotFromTest()
{
  await LauncherFromTest.AutopilotAsync("Assets/Path/To/AutopilotSettings.asset");
}

Note

If an error is detected in running, it will be output to LogError and the test will fail.

Warning

The default timeout for tests is 3 minutes. If the autopilot execution time exceeds 3 minutes, please specify the timeout time with the Timeout attribute.

3. Run from commandline

To execute from the commandline, specify the following arguments.

$(UNITY) \
  -projectPath $(PROJECT_HOME) \
  -batchmode \
  -executeMethod DeNA.Anjin.Editor.Commandline.Bootstrap \
  -AUTOPILOT_SETTINGS Assets/Path/To/AutopilotSettings.asset

• UNITY is the path to the Unity editor, and PROJECT_HOME is the root of the project to be autorun.
• -AUTOPILOT_SETTINGS is the path to the settings file (AutopilotSettings) you want to run.
• Do not specify -quit (it will exit without entering play mode).
• Do not specify -nographics (do not show GameView window).

In addition, some settings can be overridden by adding the following arguments. For details on each argument, see the entry of the same name in the "Generate and configure the AutopilotSettings.asset file" mentioned above.

LIFESPAN_SEC

Specifies the execution time limit in seconds

RANDOM_SEED

Specifies when you want to fix the seed given to the pseudo-random number generator

TIME_SCALE

Specifies the Time.timeScale. Default is 1.0

JUNIT_REPORT_PATH

Specifies the JUnit-style report file output path

HANDLE_EXCEPTION

Overwrites whether to report when an exception occurs with TRUE/FALSE

HANDLE_ERROR

Overwrites whether to report when an error message is detected with TRUE/FALSE

HANDLE_ASSERT

Overwrites whether to report when an assert message is detected with TRUE/FALSE

HANDLE_WARNING

Overwrites whether to report when an warning message is detected with TRUE/FALSE

In both cases, the key should be prefixed with - and specified as -LIFESPAN_SEC 60.

Built-in Agents

The following Agent types are provided. These can be used as they are, or project-specific Agents can be implemented and used.

UGUIMonkeyAgent

This is an Agent that randomly manipulates uGUI components. This agent implementation uses open source test-helper.monkey package.

An instance of this Agent (.asset file) can contain the following.

Lifespan Sec

Duration of random operation execution time in secounds. If 0 is specified, the operation is almost unlimited (TimeSpan.MaxValue). With this setting, neither Autopilot nor the app itself will exit when the Agent exits. It will not do anything until the next Scene switch

Delay Millis

Wait interval [milliseconds] between random operations

Secs Searching Components

Seconds to determine that an error has occurred when an object that can be interacted with does not exist

Touch and Hold Millis

Delay time for touch-and-hold [ms]

Enable Gizmos

Show Gizmos on GameView during running monkey test if true

Screenshot Options:

Enabled

Whether screenshot is enabled or not

Directory

Use Default: Whether using a default directory path to save screenshots or specifying it manually. Default value is specified by command line argument "-testHelperScreenshotDirectory". If the command line argument is also omitted, Application.persistentDataPath + "/TestHelper/Screenshots/" is used.
Path: Directory path to save screenshots

Filename

Use Default: Whether using a default prefix of screenshots filename or specifying it manually. Default value is agent name
Prefix: Prefix of screenshots filename

Super Size

The factor to increase resolution with. Neither this nor Stereo Capture Mode can be specified

Stereo Capture Mode

The eye texture to capture when stereo rendering is enabled. Neither this nor Resolution Factor can be specified

If you have a GameObject that you want to avoid manipulation by the UGUIMonkeyAgent, attach the IgnoreAnnotation component in the TestHelper.Monkey.Annotations assembly. See Anjin Annotations below for more information.

UGUIPlaybackAgent

This is an Agent that playback uGUI operations with the Recorded Playback feature of the Automated QA package.

Note

The Automated QA package is in the preview stage. Please note that destructive changes may occur, and the package itself may be discontinued or withdrawn.

The following can be set in an instance (.asset file) of this Agent.

Recorded Json

Specify the recording file (.json) to play

Use the Recorded Playback window for recording operations with Automated QA. The window is opened via the Unity editor menu Automated QA > Automated QA Hub > Recorded Playback. The recording file (.json) is saved under the Assets/Recordings/ folder and can be freely moved or renamed.

Note that the Recorded Playback function in Automated QA can record operations across Scene transitions, but in Anjin, when the Scene is switched, the Agent is also forcibly switched, so playback is also interrupted. Therefore, please be careful to record in units of Scenes.

A screenshot of the operation by Automated QA is stored under Application.persistentDataPath/Anjin. The Application.persistentDataPath for each platform can be found in the Unity manual at Application.persistentDataPath.

DoNothingAgent

An Agent that does nothing.

The following settings can be configured for this Agent instance (.asset file).

Lifespan Sec

Specifies the do nothing time in seconds. 0 means unlimited time to do nothing. If 0 is specified, an unlimited amount of time for no action is taken. It will not do anything until the next Scene is switched.

ParallelCompositeAgent

An Agent that executes multiple Agents in parallel.

The instance of this Agent (.asset file) can have the following settings.

Agents

A list of Agents to be executed in parallel, which can be nested by specifying a CompositeAgent

SerialCompositeAgent

An Agent that executes multiple Agents in series.

The instance of this Agent (.asset file) can have the following settings.

Agents

A list of Agents to be executed in series, which can be nested by specifying a CompositeAgent

OneTimeAgent

An Agent that can register one child Agent and execute it only once throughout the Autopilot execution period. The second time executions will be skipped.

For example, in a game where the title scene leads to a different path only for the first time or where the user receives a login bonus only for the first time on the home scene, it is possible to construct a scenario in which the title screen is executed as is even if a communication error or the like causes a return to the title scene.

The Agent instance (.asset file) can contain the following settings.

Agent

Agent that can be executed only once, which can be nested by specifying a CompositeAgent

RepeatAgent

An Agent that executes one child Agent indefinitely and repeatedly.

Combined with SerialCompositeAgent, it can be used to repeat a scenario many times or to repeat only the last Agent in a scenario. Note that finite iterations are not supported (to use SerialCompositeAgent).

This Agent instance (.asset file) can contain the following.

Agent

Agent to be executed repeatedly, which can be nested by specifying a CompositeAgent

TimeBombAgent

An Agent that will fail if a defuse message is not received before the inner agent exits.

For example, pass the out-game tutorial (Things that can be completed with tap operations on uGUI, such as smartphone games).

1. Set UGUIMonkeyAgent as the Agent. It should not be able to operate except for the advancing button. Set the Lifespan Sec with a little margin.
2. Set the log message to be output when the tutorial is completed as the Defuse Message.

This Agent instance (.asset file) can contain the following.

Agent

Working Agent. If this Agent exits first, the TimeBombAgent will fail.

Defuse Message

Defuse the time bomb when this message comes to the log first. Can specify regex.

EmergencyExitAgent

An Agent that monitors the appearance of the EmergencyExit component in the DeNA.Anjin.Annotations assembly and clicks on it as soon as it appears. This can be used in games that contain behavior that is irregular in the execution of the test scenario, for example, communication errors or "return to title screen" buttons that are triggered by a daybreak.

It should always be started at the same time as other Agents (that actually perform game operations). This can be accomplished with ParallelCompositeAgent, but it is easier to set it up as an ObserverAgent in AutopilotSettings.

Built-in Logger

The following Logger types are provided. These can be used as they are, or project-specific Loggers can be implemented and used.

Composite Logger

A Logger that delegates to multiple loggers.

The instance of this Logger (.asset file) can have the following settings.

Loggers

A list of Logger to delegates

Console Logger

A Logger that outputs to a console.

The instance of this Logger (.asset file) can have the following settings.

Filter LogType

To selective enable debug log message

File Logger

A Logger that outputs to a specified file.

The instance of this Logger (.asset file) can have the following settings.

Output File Path

Log output file path. Specify relative path from project root or absolute path. When run on player, it will be the Application.persistentDataPath. This setting can be overwritten with the command line argument -FILE_LOGGER_OUTPUT_PATH, but if multiple File Loggers are defined, they will all be overwritten with the same path.

Filter LogType

To selective enable debug log message

Timestamp

Output timestamp to log entities

Built-in Reporter

The following Reporter types are provided. These can be used as they are, or project-specific Reporters can be implemented and used.

Composite Reporter

A Reporter that delegates to multiple Reporters.

The instance of this Reporter (.asset file) can have the following settings.

Reporters

A list of Reporter to delegates

Slack Reporter

A Reporter that post report to Slack.

The instance of this Reporter (.asset file) can have the following settings.

Slack Token

Web API token used for Slack notifications. If omitted, no notifications will be sent. This setting can be overwritten with the command line argument -SLACK_TOKEN.

Slack Channels

Channels to send Slack notifications. If omitted, not notified. Multiple channels can be specified by separating them with commas. This setting can be overwritten with the command line argument -SLACK_CHANNELS.

Mention Sub Team IDs

Comma Separated Team IDs to Mention in Slack Notification Message

Add Here In Slack Message

Add @here to Slack notification message. Default is off

Implementation of game title-specific code

Game title specific Agents and initialization code must be avoided in the release build. Create a unique assembly and turn off "Auto Referenced", and set the "Define Constraints" to UNITY_INCLUDE_TESTS || DENA_AUTOPILOT_ENABLE.

This asmdef and its storage folder can be created by opening the context menu anywhere in the Project window and selecting Create > Anjin > Title Own Assembly Folder.

Custom Agent

A custom Agent is created by inheriting from Anjin.Agents.AbstractAgent. Simply implement the process to be executed by the Agent in the method UniTask Run(CancellationToken).

The following fields defined in AbstractAgent are available. Each instance is set before calling the Run method.

• Logger: implementation of UnityEngine.ILogger. Use this for log output. Currently, it is a console output, but there are plans to replace it with a file logger.
• Random: Implementation of Anjin.Utilities.IRandom. Which is created from a seed specified in a configuration file or a startup argument.

Note that it is convenient to set the [CreateAssetMenu] attribute to create an instance from the context menu.

Game title-specific pre-processing

If your title requires its own initialization process, add the InitializeOnLaunchAutopilot attribute to the static method that does the initialization. An added method is called from the autopilot launch process.

Note that the autopilot launch process is performed with RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterSceneLoad) (default for RuntimeInitializeOnLoadMethod). Also, RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.SubsystemRegistration) implements the initialization process for Configurable Enter Play Mode.

Anjin Annotations

Annotations are defined to control Anjin operations. Use the DeNA.Anjin.Annotations assembly by adding it to the Assembly Definition References. Please note that this will be included in the release build due to the way it works.

Note

Even if the annotations assembly is removed from the release build, the link to the annotation component will remain Scenes and Prefabs in the asset bundle built. Therefore, a warning log will be output during instantiate. To avoid this, annotations assembly are included in release builds.

IgnoreAnnotation

The GameObject to which this component is attached avoids manipulation by the UGUIMonkeyAgent.

EmergencyExit

When a Button to which this component is attached appears, the EmergencyExitAgent will immediately attempt to click on it. It is intended to be attached to buttons that are irregular in the execution of the test scenario, such as the "return to title screen" button due to a communication error or a daybreak.

Troubleshooting

Autopilot runs on its own in play mode.

The AutopilotState.asset that persists in Anjin's execution state may be in an incorrect state. Open it in the inspector and click the Reset button.

If this does not solve the problem, try deleting AutopilotState.asset.

[CompilerError] Argument 1: Cannot convert to 'System.Threading.CancellationToken'

The following compilation error has been reported in some cases:

[CompilerError] Argument 1:.
Cannot convert from 'DeNA.Anjin.Reporters.SlackReporter.CoroutineRunner' to 'System.Threading.CancellationToken'
Compiler Error at Library\PackageCache\[email protected]\Runtime\Reporters\SlackReporter.cs:66 column 53

This compile error occurs when using UniTask v2.3.0 or older. The UniTask.WaitForEndOfFrame(MonoBehaviour) API was implemented in UniTask v2.3.1

Usually, the correct version of UniTask is installed if you follow the installation instructions. However, if UniTask is placed as an embedded package in the project, it will be prioritized. Find out why you have an old UniTask installed. e.g., Check the contents of Packages/packages-lock.json generated by the Unity editor; use your IDE's code definition jump function to check where the source file of UniTask.WaitForEndOfFrame() is.

License

MIT License

How to contribute

Open an issue or create a pull request.

Be grateful if you could label the pull request as enhancement, bug, chore, and documentation. See PR Labeler settings for automatically labeling from the branch name.

How to development

Add this repository as a submodule to the Packages/ directory in your project.

Run the command below:

git submodule add https://github.com/dena/Anjin.git Packages/com.dena.anjin

Warning

Required install packages for running tests (when adding to the testables in package.json), as follows:

• Unity Test Framework package v1.3.4 or later
• Test Helper package v0.4.2 or later

Generate a temporary project and run tests on each Unity version from the command line.

make create_project
UNITY_VERSION=2019.4.40f1 make -k test

Release workflow

Run Actions > Create release pull request > Run workflow and merge created pull request. (Or bump version in package.json on default branch)

Then, Will do the release process automatically by Release workflow. And after tagged, OpenUPM retrieves the tag and updates it.

Do NOT manually operation the following operations:

• Create release tag
• Publish draft releases