Evaluations

The Teamwork Challenge scenario described above has been idealized by several AI researchers, at least in the planning and multiagent communities. RoboCup, both in its simulated and real leagues, provides a synergistic framework to develop and/or test dynamic planning multiagent algorithms.

Specifically, we are planning to evaluate the architecture and teams in the following evaluation scheme:

Basic Performance:

The team must be able to play reasonably well against both the best hand-coded teams, which has no planning, and against other planning-based systems. Relative performance of the team can be measured by actually playing a series of games against other unknown teams. Thus, basic performance will be measured by:

• Performance against hand-coded teams.
• Performance against other teams.

Robustness:

The robustness in teamwork means that the team, as a whole, can continue to carry out the mission even if unexpected changes, such as accidental removal of the players in the team, sudden change of team conposition, or changes in operation environment. For example, if one of players in the team was disabled, the team should be able to cope with such accidents, by taking over the role of disabled players, or reformulating their team strategy. Thus, this evalution represents a set of unexpected incidents during the game, such as:

• Some players will be disabled, or their capability will be significantly undermined by these accidents. Also, some disabled players may be enabled later in the game.
• Opponent switch their strategy, and the team must cope with their new strategy in real time.
• Some of opponent's players will be disabled, or their performance will be significantly undermined. These disabled players may come back to the game later.
• Teammate changes during the game.
• Weather factor changes.

The RoboCup Teamwork Challenge therefore is to define a general set of teamwork capabilities to be integrated with agent architectures to facilitate flexible, reusable teamwork. The following then establish the general evaluation criteria:

General Performace:

General performance of the team, thus the underlying algorithms, can be measured by a series of games against various teams. This can be divided into two classes (1) normal compeitions where no accidental factors involved, and (2) contigency evaluaiton where accidental factors are introduced.

Real-Time Operations:

The real-time execution, monotoring, and replanning of the contingency plan is an important factor of the evaluaiton. For any team to be successful in the RoboCup server, it must be able to react in real time: sensory information arrives between 2 and 8 times a second and agents can act up to 10 times a second.

Generality:

Reuse of architecture in other applications: Illustrate the reuse of teamwork capabilities in other applications, including applications for information integration on the internet, entertainment, training, etc.

Conformity with Learning:

Finally, given the premises above and the complexity of the issues, we argue and challenge that a real-time multiagent planning system needs to have the ability to be well integrated with a learning approach, i.e., it needs to refine and dynamically adapt and refine its complete behavior (individual and team) based on its past experience.

Other issues such as reuse of teamwork architecture within the RoboCup community, and planning for team players that are not yet active in order to increase their probability of being useful in future moves, such as role playing and positioning of the team players that do not have the ball, will be considered, too.