Keepaway Framework Tutorial

This tutorial page explains step-by-step how to install the keepaway benchmark players, run a simulation, and generate a graph of the learning curve.

The keepaway framework described below has been used extensively in our previous research.

1) The RoboCup Soccer Server

To run the keepaway players, you must first install the RoboCup Soccer Server. If you already have the server installed, you may skip to step 2. If you have not installed the server yet, we highly recommend that you use version 9.4.5. You will need the following files: (Note that these are links to download pages and not the files themselves.)

Note that for this version of the server, GCC version 3 is required. GCC4 is known to cause compilation errors on some systems. Later versions of the server may compile with GCC4.

Choose the installation directory:
First, you must choose where you want to install the soccer server files. For system-wide installation, you will probably want to choose a directory such as /usr/local. If you do not have root access, you will probably want to choose something like a subdirectory of your home directory. For instance, /home/username/rcss. You need to set the RCSSBASEDIR environment variable to this directory:

export RCSSBASEDIR=/home/username/rcss

Now you need, to add the bin sub-directory to your PATH. For system-wide installation, you will probably not need to do this because the sub-directory (e.g. /usr/local/bin) is already in your PATH. For local installation:

export PATH=$RCSSBASEDIR/bin:$PATH

For this change to take effect permanently, you will need to add this line to a file that is sourced when you open a shell, such as ~/.profile.

Install the base files:

tar xjvf rcssbase-9.4.5.tar.bz2
pushd rcssbase-9.4.5
./configure --prefix=$RCSSBASEDIR
make
make install
popd

Install the server:

tar xjvf rcssserver-9.4.5.tar.bz2
pushd rcssserver-9.4.5

The following three steps are necessary for local installation only: (Note that it is unnecessary to set these variables permanently as they are used only during installation.)

export LDFLAGS="-L$RCSSBASEDIR/lib"
export CXXFLAGS="-I$RCSSBASEDIR/include"
export LLIMPORTER_PATH="$RCSSBASEDIR/bin"

./configure --prefix=$RCSSBASEDIR
make
make install
popd

Install the monitor:

tar xjvf rcssmonitor-9.3.7.tar.bz2
pushd rcssmonitor-9.3.7
./configure --prefix=$RCSSBASEDIR
make
make install
popd

2) Downloading and Compiling the Players

First, download the keepaway framework package:

Download Version 0.6

Next, open the package in the directory in which you would like it installed:

tar xzvf keepaway-0.6.tar.gz

Now compile the players:

cd keepaway-0.6/player
make depend
make

Next, compile and install the tools:

cd ../tools
make

Copy the binaries to a directory in your PATH. For example:

cp hist killserver kunzip kwyzipper kzip monitor winsum /usr/local/bin

3) Running the players

To start the simulation:

./keepaway.sh

The startup script will finish after about 10 seconds. The simulation is now running. To watch the players, execute the monitor script:

monitor

To end the simulation:

killserver

4) Generating a learning curve

During the simulation, information about the episodes was being recorded to a text file in the logs. This section describes how to use that information to generate a learning curve. First, enter the logs directory to find the log file that was generated:

cd logs

There should be a file with a .kwy extension. To generate a learning curve, this file must contain at least 900 episodes. Since it probably doesn't, you can use an included example file to generate the curve. We'll create a learning curve for the example.kwy file:

cd ../tools
cat example.kwy | winsum 900 0.01 30 > 1.out

This creates a sliding window average of the episode durations in the .kwy file. The window size is 900 episodes. 0.01 is the alpha value used in the low-pass filter to smooth the curve, and only every 30th data point is output to cut down on file size.

Finally, we create the curve with gnuplot:

gnuplot graph.gnuplot

There should now be a file called graph.eps. After running:

gv graph.eps

You should see a graph something like this:
graph

5) Where to go from here...

Here are some hints and possible next steps after getting the keepaway players to run on your system.

Changing the policy:
Most likely, the first thing that you will want to do is change the keeper behavior. The keepaway player source code comes with the three fixed policies described in our papers:

hold - always hold the ball
rand - choose random action uniformly
hand - hand-coded policy

Initially, the policy is set to rand. To try a different policy, open keepaway.sh in a text-editor and change the value of keeper_policy. For example:
keeper_policy="hold"

All fixed policies are implemented in HandCodedAgent.{h,cc}. To create your own policies, you may want to start by modifying this file. In addition, there is another value available to the keeper_policy option in the startup script: "learned". This value is intended to be used for your own learned policy. To create your own learned policy, see the Learning Agent Tutorial.

Speeding up with synch mode:
The simulation can be made to run at faster than real-time using an option called synch_mode. Messages are passed between the clients and players as quickly as possible. Be aware that running in this mode typically puts a very high load on the system. To use, modify keepaway.sh in a text-editor and set:
synch_mode=1

Using unrestricted vision:
By default the players can see objects in a 90 degree view cone in front of them. To make the players omniscient, you can change the view angle to 360 by making the following change in keepaway.sh:
unrestricted_vision=1

Please email questions or comments to the mailing list.