Each page in these instructions has a "Next" button and a "Previous" button. For each page, perform the operations described and press "Next" when you are ready for the next task. You can press Previous to go back in the instructions (Note: it does not undo what you did in the underlying NERO simulation). You can exit this or any mission by pressing the ESC key. Pressing "F1" will bring up an in-depth help guide (but you should not be needing it for this exercise).
(2/30)
To move around the world, use the W, A, S, and D keys. To rotate the camera view horizontally, press Q or E. To rotate the camera more freely with the mouse, press the M key (in combination with W and S, you can then move up and down). Pressing M again will return you to the normal "RTS" camera mode. Pressing the O key will get you to the "Options" menu where you can e.g. make the NERO window larger (which is recommended---the default window is rather small), adjust audio, or remap the control keys.
(3/30)
The display on the bottom allows you to control the fitness function (and a few other simulation parameters). Right mouse click on the ground and on various objects pops up radial menus that allow you to change the environment (another mouse click gets rid of the menu again). The "Esc" key brings up a menu that allows you to save or reset the simulation and exit.
(4/30)
We will begin by evolving a most basic skill: Approaching an enemy. NERO robots start with simple, random-weight neural networks, i.e. they have no built-in skills. For this task, you will begin by placing a single static enemy (i.e. an enemy that does not fire back or run around) on the field as a target, as follows.
(5/30)
You will place this static enemy approximately in the center of the field. To do so, hover the mouse pointer over the center of the field, right-click, and choose the "Spawn," "Enemy," and then "Static Enemy" option from the radial menu. A robot should appear at the location in which the initial right-click occured. All spawned enemies are invincible and will never die, unless you remove them from the field (by right-clicking on them and selecting "Delete"). Enemies do not work together, unless you spawn a pretrained team that is trained to do so.
(6/30)
Before launching a new training team, we need to specify its training objectives. These objectives are specified through the sliders in the right panel. If you hover your mouse pointer over the slider icons, an info box appears next to each icon explaining its meaning.
(7/30)
The training goal is to teach the robots to approach the enemy. To begin, put your mouse below the "Approach Enemy" icon and move the slider up and down. We want to reward this behavior, so complete this step by leaving the slider at its maximal setting above the line. (Placing the slider below the line would indicate a desire to punish the behavior.)
(8/30)
In the future you will use the other slides to set various combinations of rewards and punishments, but for now the goal is to evolve a single behavior---from scratch. You get to see the entire evolutionary process on the field: the entire population of neural networks controlling the agents, and how this population changes over time as old agents are removed and new ones are created.
(9/30)
The enemy is now on the field and the objectives are set. We are ready to launch a new team. Choose a spot to the left of the enemy (e.g. 1/4 of the sandbox width away), right-click, choose "Spawn," and then "Team" from the radial menu. A untrained team will appear falling from the sky. Each individual is controlled with a different randomly weighted neural network with a simple feedforward topology (inputs connected directly to the outputs). Their inputs sense walls, other agents, the enemy, and enemy fire, and the networks respond by moving left, right, back, forth, and firing. As a result they exhibit a wide variety of behavior patterns.
(10/30)
Each agent will be tested in the field for a prespecified amount of time. Once their lifetime expires, they are removed from the field, and replaced by offspring agents that again fall from the sky. You should see some of this replacement going on by now.
(11/30)
By chance, some of the agents respond to the enemy sensors by moving towards the enemy. They will achieve high fitness, and will thus be chosen for reproduction. New neural networks are generated by crossing over and mutating these parent networks (through the NEAT neuroevolution method). The new offspring agents have these new networks as their brains. As you should see the offspring should be better at moving towards the enemy than the earlier agents.
(12/30)
Wait between 120 and 200 seconds and the team should learn to approach the target successfully. If they do not learn, try moving the target around by right-clicking the target and choosing "Move" from the radial menu. (A left-click will then release the object being moved.) Moving the enemey closer or farther may improve training. In fact, the robots learn while you move the enemy, so you can move it around in any direction and they should improve at following it. If you have trouble clicking on an enemy, you can pause the simulation by pressing "p". Every unit will stop moving, and your troops will not evolve. Pressing "p" again will unpause the simulation.
(13/30)
You can press the "Evolution" button on the panel to turn evolution on and off, and the Collision button to allow or prevent agents from occupying the same location. The bar under the clock changes the lifetime of the individuals, and the bar to the left of it defines what distance is deemed "close". We won't need them for this exercise but feel free to explore them later (their effect is discussed in more detail in the Advanced tutorial).
(14/30)
Note that it is possible to move the camera around and watch the action from different views. It is also possible to inhabit the enemy (by right-clicking on it and choosing "Inhabit") and move it around with the WASDQE keys and the mouse, and see how well the agents follow (mouseclick terminates the inhabit). The point is that the neural networks have evolved to maximise fitness: Given their sensor inputs, their actions lead them close to the enemy, no matter what the enemy is doing. Thus, this result illustrates the first main point of this exercise: how evolution conducts a parallel search for solutions that maximize fitness.
(15/30)
Once the robots are skilled at approaching the enemy, we are ready to elaborate on this behavior, by evolving the agents to approach around an obstacle. We are following an important general principle in neuroevolution (and learning in general): Start with basic simple skills and elaborate on them. In general, trying to train for numerous complex skills simultaneously does not work well. Like natural evolution and much of animal learning, NEAT neuroevolution progresses best through increasingly difficult challenges.
(16/30)
Our team has never seen a wall before. Let's place a wall between the spawn area and the enemy target so that the agents have to learn to go around it. Find a point halfway between the team and the enemy, right-click, and choose "Spawn," then "Wall" from the radial menu.
(17/30)
We want the wall to be oriented properly so that it is directly blocking the team from the enemy. Try rotating and moving it into an ideal position by right-clicking the wall and choosing "Rotate" or "Move" and using the mouse to position it correctly. (Note that you can also scale the size of the wall in this way.)
(18/30)
Once the wall is positioned, the team is forced to go around it in order to fulfill the current objectives. Within three minutes, the team should begin to figure out how to navigate the wall and find the enemy. Note that no further slider changes are necessary for this task---this elaboration of behavior results from an environment change only.
(19/30)
Next, let's elaborate the behavior by adding a second fitness objective. Go to the second slider called "Hit Target" and pull it up to full reward. This objective will reward the agents for shooting at the enemy (while they are still rewarded for approaching it also). As a result, in a few minutes, they should start shooting at the enemy after they clear the wall. Since shooting is controlled by a separate output unit in the neural network, shooting does not interfere with the approach behavior, and it is easy for evolution to discover this elaboration even though it involves a different objective. These last two results illustrate the second main point of the exercise: how evolution elaborates on existing behaviors to meet new challenges, whether they originate from the environment or the objectives.
(20/30)
At this point, let's save our team so we can come back to it if we wish. Press ESC key and choose "Save Army" option in the menu that appears. The team is now saved and can be loaded back into training, or battle (it can also be loaded into the team creator utility that allows you to compose teams from individuals of different evolutionary runs---we won't be using it in this exercise but you can explore it later).
(21/30)
Next, let's see how evolution adapts to changes that do interfere with what it has learned before. It is likely that most of your agents always go around the wall on the same side. For example, assume that it is the right side: Add another wall on the right side that blocks their path (i.e. extends away from the first wall towards the spawn side, creating a convex corner). Those agents try to skirt the extended wall but it will take a long time to get to the enemy, and they receive low fitness.
(22/30)
However, the agents that were not very good at going right end up spending more time near the enemy, and thus they achieve a higher fitness. Thus, after a few minutes, these agents should evolve away from going right, and eventually instead evolve to go around the left side of the first wall. Thus, evolution capitalizes on the diversity in the population in order to adapt to the changing environment. This result illustrates the third main point: Diversity is essential for successful evolutionary adaptation.
(23/30)
Another way to interfere is to give the agents conflicting goals, requiring them to find innovative solutions to satisfy both. First, take out the second wall (by right-clicking and choosing "Delete"). The agents should still run to the enemy as before. Now place an enemy that shoots just behind the corner of the side they like to use (by right clicking on that spot on the ground, and then selecting "Spawn", "Enemy", "Normal Turret"). Then rotate that turret so that it is shooting at the agents just as they come around that corner (that is, move above the turret, right-click on it, choose rotate, and move the mouse until the turret shoots approximately at the right direction---which is a bit tricky to do).
(24/30)
Then go to the third slider, "Avoid Fire", and pull it up to full reward. This action creates a situation with conflicting goals: The agents want to attack the enemy (by getting close and firing), but when they do, they get hit a lot, resulting in low fitness. Assuming that there is enough diversity in the population, evolution discovers a way to satisfy both goals. After a few minutes, evolution should discover that if agents go around the other side of the wall, they can approach the enemy and the turret (which is also an enemy and attracts the agents) without getting hit. It is this kind of discovery that makes evolution creative. This result thus illustrates the fourth main point: evolution can discover solutions that are difficult to develop by hand.
(25/30)
This concludes the basic exercise. The steps you took illustrated the basic mechanisms of neuroevolution: parallel search for solutions that maximize fitness, elaborating learned behaviors to meet new challenges in the environment and the objectives, the importance of diversity, and discovering novel solutions.
(26/30)
From this point on, the exercise becomes open ended, and you can continue it as long as you have time. First, you should save your team so that you can come back to it if you want. Then, you can go in one of two directions: testing your team in battle, and training more complex behaviors.
(27/30)
Testing your team in battle: You have already created a team that has the basic behaviors that it needs for battle: moving around obstacles, approaching and firing at the enemy, and avoiding fire. In order to try it in battle against other teams, do the following: After saving your team, exit training, and go to Sandbox Battle. Choose your saved team and some other team (for instance the various aggressors and grovers), and an arena. The Virtual Sandbox is the simplest arena, but you may want to try some others as well (such as Shadow Grove and Mountain Pass). You can also battle against someone else in the class (those teams need to be copied to the nero/data/saves/brains directory first).
(28/30)
You could also train your team towards more complex behaviors. For instance, try expanding the size of the wall. Try adding more walls to create a maze like the "Virtual Navigation Arena". The team "Dispersers" is an example of a team that was trained to navigate the Dispersion level successfully. Try further training your team to go around the huge dividing wall such as the one in the "Shadow Grove" level. Try adding multiple enemies and see if your team can handle them. Walk through the Advanced Tutorial to learn about the other fitness components and parameters that affect training and performance. You may want to go to battle every now and then to see if your more complex teams perform better than the simpler ones, both against the pre-existing teams and your earlier teams.
(29/30)
In the end, you will hopefully develop an understanding of how neuroevolution works and what its potential is in solving design and optimization problems. Thanks for participating!
(30/30)