Training an artificial neural network for the micro of marines

[Leckofunny]

[krasi0]

Hey! I have been interested in the same topic for a while (training ANNs to command the micro behaviour of individual units) and I like it how you've laid out the architecture and explanations of your approach so far.

IMO, the current list of output actions that you make use of is sufficient. For the time being, I'd put more attention to the inputs to the first layer though, as those might have a crucial impact on any efficient training.
I haven't yet had the time to take a look at your source code but do you consider stuff like the HPs of the current unit, distance to each enemy unit, whether positioned on high / low ground, etc.?

Wrt the expirations of each command, I think you're on the right track when considering the duration of each command. So If I were you, I'd just not touch the unit for X frames after an issued command, where X varies per command type, e.g. 4 frames for AttackEnemyUnitInRange, 12 frames for retreat, etc.

Also I am not sure if you already do this, but perhaps it makes sense to only train the network once the whole battle has finished (or a timeout of Y frames has occurred) and for the fitness function then consider something like total frames passed (a lower count is better if our side has won; more is better if the enemy side has won), own units' remaining HPs; enemy units' remaining HPs and so on. Perhaps, you already do this?

Keep us updated on any progress. I'll be very interested to follow the project (you've already got the star on GitHub )

[Leckofunny]

Thanks for your reply.

Right now I'm planning on simplifying the current scenario. I had to debug and trace down some issues regarding the fitness function and some other stuff related to that. So far my fitness function wasn't able to generate proper training examples. After training, most output vectors would be like [1,1,1,1,1], meaning that any action would be great. So for some reason most of the actions taken are considered to be good.

Regarding Supreme Commander 2, all the sources, authored by Michael Robbins, are somewhat inconsistent. Since decrypting these information is pretty challenging, I believe that getting a simple case to work is a good base for increasing complexity.

These are my references about Supreme Commander 2:





By simplification, I'm thinking about to scratch stimpack usage and maybe to use a 1 vs 1 combat situation. In the blog post, Robbins mentioned that during training the individual units would only know about itself and the enemies, whether in execution mode it would gather information about the friendly units as well. It is also confusing that Robbins talks about delta values, because he used to write about ratios in his chapter, which is the most detailed source. Well, ratios make the most sense, since overall hp or a single units hp should work always as percentage (value inbetween 0 - 1). This is also a good approach for data normalization. So in the end I could simplify the information to local information. This makes somewhat sense about my earlier explained issue, there one unit would consider a stupid action to be great, because its mates went all out and dealt tons of damage, while that one particular unit was useless. I guess this tends to guide the training to that [1,1,1,1,1] output.


There is another confusing information given by Robbins. He first states that data is generated by comparing the state pre and post an action. Further down on a comment he talks about encounters instead.


And at last concerning my current training flow, at the end of each math, all the gathered training examples are backprogated just once.

[Leckofunny]

I simplified the setup like this:

2 inputs: Local hit points, Enemy overall hit points
8 hidden nodes
3 outputs: Attack Closest, Move Towards, Move Back

I also modified the action Attack. As soon as one shot has been fired, the attack is completed. So moving into range belongs to that action.


The real issue should be within my , because the computed output keeps going towards [1,1,1]. It could be some stupid logical flaw. Or the whole approach is missing out some crucial point.

[krasi0]

Always converging towards [1, 1, 1...] sounds like the NNs might not be getting enough important data (hence a random action is considered as good as any other) or the training process is somehow broken...

How about feeding it a bit more data like:
1. current unit HPs
2. the HPs of the enemy unit with the lowest HPs in range
3. total number of enemy units that have the current unit in range
4. total damage that those units could cause to the current unit
5. total number of enemy units that do NOT yet have the current unit in range, but are just one move away (say their attack radius + 2 * 32 pixels)
6. total HPs of allied units in range of the enemy units from point 3)

In Broodwar it doesn't seem logical to attack the strongest (having the most HPs) enemy unit, but it rather makes sense to eliminate enemy units as quickly as possible, so I'd say have the AttackWeakest instead of AttackClosest or have them both.


Besides, have you tested your ANN implementation on a much simpler problem like XOR or tic-tac-toe just to verify that training works in general?

Kinda off topic, but since I won't be able to compile your project any time soon, how about you streaming the training process? I'd be interested in watching the game play evolve?

BTW, you could message me at krasi000 at gmail or on FB messenger for faster communication if you like

[Leckofunny]

I use the framework for the neural nets. I did some basic stuff with the frame work like classifying xor and that flower example.

The strongest unit could be a smart target if more types of units would get involved.

The training process is not really exciting to watch since actions are randomly chosen.


I'll consider more detailed inputs. In order to get some immediate insights, I'd like to keep things simple first.

[Leckofunny]

[krasi0]

Regarding the noise while training, I would still re-introduce an idea that I mentioned in my first post on this topic: "perhaps it makes sense to only train the network once the whole battle has finished (or a timeout of Y frames has occurred) and for the fitness function then consider something like total frames passed (a lower count is better if our side has won...". So basically this is a form of *offline* learning, i.e. during the battle, all units react using the same ANN; the battle finishes, then the ANN trains are adjusted accordingly to the outcome of the battle. Then a new battle is launched with the adjusted ANN weights and so on.
I think it's worth giving a shot.
Please report back if you do!

[Leckofunny]

The neural net doesn't make the decisions during training. The output action is selected by random. Otherwise a fresh neural net would most likely classify any input to one output. Still before the random action is selected, the net has to be fed forward in order to generate training examples. Making random decisions makes sure that a broad range of training examples are covered.

All the gathered training examples of one match are iterated once for training on the conclusion of the match.

With noise I mean conflicting data. So there are data pairs which have the same inputs but are totally different on outputs. So there is always a huge training error.


The current state of my project is that I'm getting close to be done with implementing my feedback logic. So next week the training procedure should generate clean training examples.

[krasi0]

Cool! Keep us updated on how it goes next week