cg_referee

Introduction

Generic referee system for Codingame bots. The system is as disconnected from a single game as possible, so you have to code an additional game binary that will manage the evolution of the game and tell the referee when a game is over.

The referee is written in Python. It should work on any machine having Python 2.5+ but has been only tested on Linux. The referee does not work yet on Python 3, although it could be adapted. Unfortunately, all attempts have been ending in failure because of the peculiar serialization used for multiprocessing. If anyone wants the challenge, please feel free to fork and add a pull request :)

Important note : This is a work in progress. Some features are missing. For instance, it is not possible to test position swapping in games yet.

Changes :

• 08/03/17 : Changed the GITC bot to be Python2 so people don't have to install 2 different version of Python on their machines, one for the referee, one for the engine (unless you already have Python 3 ...).

Dependencies :

The referee requires a working version of Python 2.5+. Other than that, everything is supposed to be standard in its code, so you should not have to install any additional requirements. You will also need numpy (a Python package). But, in any case you should get numpy, 'cause numpy is awesome.

List of Engines included in the package :

• Ghost in the Cell (Python 3)
• Tron (C++)

If you feel like contributing, feel free to send your engines to me, I'll include them with pleasure to the repo.

Todo list for the next versions :

• Adding a parameter to swap the positions of a run. This will be necessarily mixed with the inputs of the game binary.
• Checking the time of response of the bots to kill them if they are too long to respond.

Getting started

The repository provides you with an example file tron.json for the codingame Tron. This configuration file details all the referee needs to know to work. Aside from this configuration file, a simple "game evolution" code has been provided for Tron as an example in the file tron_eval.cpp.

To get started with your own tron ai local battles, start by compiling the evaluation binary for tron :

g++ -std=c++11 -O3 -o tron tron_eval

This should create a tron binary file in your folder. Now go and edit the configuration file tron.json. The overall details of the configuration file are explained in the appropriate section below. For now, lets just see that the file is divided in three sections Game, Bots and Settings.

The Game section describes the game we are playing. For now, everything is set up correctly and there is no need to modify anything. Just check that your binary corresponds to the value of Game bin. You will also see that we can provide to the binary arguments on the command line. These are stored in the list Arguments. In that case, the binary only expects the number of players on the command line and is set by default to 2.

The Bots section is a list where every entry describes a single bot. This is the section where you have to enter your bots names and binaries. These are not provided in this example so you will have to code a little something to make it work. Be careful if the bots you are coding are not in the form of a binary please refer to the configuration section to properly start the bots using an interpreter (such as python, ruby, perl, etc.).

Finally, the Settings section takes care of the game sessions. You will see in the example file that we log everything generated by the game and the bots to files, and we will play 4 games in a row using 2 threads.

Considering you have a working bot (or two) and want to test it so you have filled in the Bots section of the configuration file, it is time to start the game.

./referee tron.json

Normally you should get a result showing this after a few minutes of computations :

==============================================
================= CG Referee =================


Initializing referee :
 - Reading parameter file : tron.json
 - Refering for game : Tron
 - Creating bot bot1. Command line : ./bot_1 
 - Creating bot bot2. Command line : ./bot_2 

Running the game ...
 - Playing run 1
 - Playing run 2
 - Playing run 3
 - Playing run 4
Statistics (in percents) :
Bot name	Rank 1	Rank 2	
bot1		50.00	50.00	
bot2		50.00	50.00	

After such a run, you will find all the logs of the games in the folder runs. This folder is organised in subfolders. The first level will correspond to the name of the game (as indicated in the configuration file), then the number of the run (eg run_001). In this subfolder you will find a stderr log per bot, and the stderr log of the evolution binary. Finally in the game folder (eg. runs/Tron), you will find the order of winner bots in the file scores.log. The log indicates the rates of victory for each bot. Each line corresponds to a bot and each column corresponds to a ranking. Thus each entry (line i, column j) indicates the rate of matches for which bot #i finished at position #j.

0.5 0.5
0.5 0.5

Configuration file

The configuration file is a simple JSON file. Note that for the moment the referee does not have default value so all the fields must be present or the referee won't work. This is something that might appear in future versions but in the meantime, try not to remove any line from the configuration file example. This section details the effect of every element of the configuration file.

Game section

This section of the JSON file tells the referee what game we are playing :

Name

The name given to the game. This parameter has two meanings, first of all it serves a reminder to you of the name of the session. Secondly, this name is used when you log the results. They will be stored in runs/<Name>.

Game bin

The path to the game binary. Note that this binary will be started by the safe Popen process mechanism of Python. Meaning that there should be NO space in the name of the binary or only the part before the first space will be ran.

In the case you need to run your game code as an interpreted code, use the binary to call the interpreter and the first argument as your script. For instance, if the game binary is a python script called tron.py, you can use python as Game bin and add tron.py as the first argument in the following list.

Thus, I would have a game section looking like this :

"Game": {
    "Name": "Tron",
    "Game bin": "python",
    "Arguments": [
        "tron.py",
        "2"
    ]
}

Arguments

The list of arguments that will be passed on the command line after Game bin.

Bot section

The Bot section is very similar to the Game one. The object is an array where every sub-object is a bot.

Name

As for the game name, this name identifies the bot for you and for the logs. The logs of the bot will be stored under the name of the bot so be sure you use different name for every bot or your logs will be overwritten.

Bin

The binary of the bot. As for the game, it can be an interpreter (then providing your bot script as an argument). It is not necessary to have different binaries for every bot, you can use the same since different processes will be ran.

Arguments

The command line arguments given to the bot if necessary

Settings section

This object stores all the global variables of the session.

Log stderr

A boolean. States if the standard error stream of the bots and the game binaries should be logged on the disk. If not, nothing will be displayed on the screen. If the logs are stored you will find all the logs of run #i in the folder runs/<Game name>/run_i/. There you will find the logs of the game for that run and the respective logs of the bots.

Log scores

A boolean. As before, the scores will be logged to a file only if this is set to true. The logs will be stored in the file runs/<Game name>/scores.log.

Runs

The number of games to play in that session.

Theads

The number of threads to compute the runs. Each run will be assigned to a single thread.

Seed

It is now possible to indicate a seed for the Referee. The referee will then provide a series of seeds to the game so that the initialisation will persist in-between runs. Note that for the seed to be used, you will have to provide it to the game engine. The seed is read by the engine as a command line argument, and, if you want to use it, it should be included in the parameters of the engine as a parameter calle $seed. For an example, see the GITC parameter file and engine. Please also note that using the same seed as the ones given by the Codingame IDE will not give you the same initialization ...

Coding a game binary

Last but not least, the game binary must be coded in a specific way to be able to speak with the referee program. The way the programs talk is made as simple as possible.

First of all, your bots will be the same as the ones you push on codingame. No modifications required here (except to provide a binary for the compiled versions. All information between the referee, the bots and the game binary will be communicated through stdin and stdout. The bots never directly communicate with the game binary, the referee is here to make them talk together.

The game binary in itself will always have the same structure in pseudo-code :

while (true):
  If game is finished:
    Write -1 to stdout
    Write ranking
  Else If current_player is dead:
    Write 0 to stdout
  Else
    Write N to stdout, where N is the number of lines the game will feed to the bot
    Then write N lines to stdout
    Read line L from stdin
    Apply the effect of "output" L to current_player
    
  current_player = next_player

Basically there are two important things to remember :

First, at the beginning of every turn, you need to send to the referee the state of the game. If the game is finished, send -1, the server will interpret this as the end of the game. If the game is not finished, the referee is expecting a number of lines to send to the player. If this number is "0" it means the player is dead.

If we take the example of Tron, if we have 3 players in game, and the current player is alive, then the bot will be expecting 4 lines (one with it's id and number of players, and three indicating the position of each bots). So we will send to the referee the number 4.

The second important thing is that once the inputs of a bot send, the game binary must expect the answer of the bot for this turn. Thus, reading a the result of its action on stdin. This input will be exactly the same as the output provided by the bot. Then, the game binary will apply the effect of this action to the game state and skip to the next player.

Finally if the game is finished, the engine must return the ranking to the referee. The ranking is the order of the players, starting from 0, and separated by spaces. If the game ends up in a tie, you can just send tied, and every players will be considered as first place.

If things are getting confused, read the tron_eval.cpp or the gitc.py files provided as an example to work for Tron and Ghost in the Cell.