Training an artificial neural network for the micro of marines

Anything related on how to build / improve your AI using BWAPI, BWTA, BWSAL, etc.
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Training an artificial neural network for the micro of marines

Postby Leckofunny » Tue May 03, 2016 12:49 pm

Attachments
flowchart.png
flowchart.png (62.6 KiB) Viewed 12812 times
krasi0
Posts: 254
Joined: Thu Dec 18, 2008 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby krasi0 » Thu May 05, 2016 8:48 pm

Hey! I have been interested in the same topic for a while (training ANNs to command the micro behaviour of individual units) and I like it how you've laid out the architecture and explanations of your approach so far. :)

IMO, the current list of output actions that you make use of is sufficient. For the time being, I'd put more attention to the inputs to the first layer though, as those might have a crucial impact on any efficient training.
I haven't yet had the time to take a look at your source code but do you consider stuff like the HPs of the current unit, distance to each enemy unit, whether positioned on high / low ground, etc.?

Wrt the expirations of each command, I think you're on the right track when considering the duration of each command. So If I were you, I'd just not touch the unit for X frames after an issued command, where X varies per command type, e.g. 4 frames for AttackEnemyUnitInRange, 12 frames for retreat, etc.

Also I am not sure if you already do this, but perhaps it makes sense to only train the network once the whole battle has finished (or a timeout of Y frames has occurred) and for the fitness function then consider something like total frames passed (a lower count is better if our side has won; more is better if the enemy side has won), own units' remaining HPs; enemy units' remaining HPs and so on. Perhaps, you already do this?

Keep us updated on any progress. I'll be very interested to follow the project (you've already got the star on GitHub :P)
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby Leckofunny » Thu May 05, 2016 9:12 pm

Thanks for your reply.

Right now I'm planning on simplifying the current scenario. I had to debug and trace down some issues regarding the fitness function and some other stuff related to that. So far my fitness function wasn't able to generate proper training examples. After training, most output vectors would be like [1,1,1,1,1], meaning that any action would be great. So for some reason most of the actions taken are considered to be good.

Regarding Supreme Commander 2, all the sources, authored by Michael Robbins, are somewhat inconsistent. Since decrypting these information is pretty challenging, I believe that getting a simple case to work is a good base for increasing complexity.

These are my references about Supreme Commander 2:





By simplification, I'm thinking about to scratch stimpack usage and maybe to use a 1 vs 1 combat situation. In the blog post, Robbins mentioned that during training the individual units would only know about itself and the enemies, whether in execution mode it would gather information about the friendly units as well. It is also confusing that Robbins talks about delta values, because he used to write about ratios in his chapter, which is the most detailed source. Well, ratios make the most sense, since overall hp or a single units hp should work always as percentage (value inbetween 0 - 1). This is also a good approach for data normalization. So in the end I could simplify the information to local information. This makes somewhat sense about my earlier explained issue, there one unit would consider a stupid action to be great, because its mates went all out and dealt tons of damage, while that one particular unit was useless. I guess this tends to guide the training to that [1,1,1,1,1] output.


There is another confusing information given by Robbins. He first states that data is generated by comparing the state pre and post an action. Further down on a comment he talks about encounters instead.


And at last concerning my current training flow, at the end of each math, all the gathered training examples are backprogated just once.
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby Leckofunny » Fri May 06, 2016 3:02 pm

I simplified the setup like this:

2 inputs: Local hit points, Enemy overall hit points
8 hidden nodes
3 outputs: Attack Closest, Move Towards, Move Back

I also modified the action Attack. As soon as one shot has been fired, the attack is completed. So moving into range belongs to that action.


The real issue should be within my , because the computed output keeps going towards [1,1,1]. It could be some stupid logical flaw. Or the whole approach is missing out some crucial point.
krasi0
Posts: 254
Joined: Thu Dec 18, 2008 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby krasi0 » Fri May 06, 2016 3:56 pm

Always converging towards [1, 1, 1...] sounds like the NNs might not be getting enough important data (hence a random action is considered as good as any other) or the training process is somehow broken...

How about feeding it a bit more data like:
1. current unit HPs
2. the HPs of the enemy unit with the lowest HPs in range
3. total number of enemy units that have the current unit in range
4. total damage that those units could cause to the current unit
5. total number of enemy units that do NOT yet have the current unit in range, but are just one move away (say their attack radius + 2 * 32 pixels)
6. total HPs of allied units in range of the enemy units from point 3)

In Broodwar it doesn't seem logical to attack the strongest (having the most HPs) enemy unit, but it rather makes sense to eliminate enemy units as quickly as possible, so I'd say have the AttackWeakest instead of AttackClosest or have them both.


Besides, have you tested your ANN implementation on a much simpler problem like XOR or tic-tac-toe just to verify that training works in general?

Kinda off topic, but since I won't be able to compile your project any time soon, how about you streaming the training process? I'd be interested in watching the game play evolve? :)

BTW, you could message me at krasi000 at gmail or on FB messenger for faster communication if you like
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby Leckofunny » Fri May 06, 2016 4:05 pm

I use the framework for the neural nets. I did some basic stuff with the frame work like classifying xor and that flower example.

The strongest unit could be a smart target if more types of units would get involved.

The training process is not really exciting to watch since actions are randomly chosen.


I'll consider more detailed inputs. In order to get some immediate insights, I'd like to keep things simple first.
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby Leckofunny » Sat May 07, 2016 5:25 pm

Attachments
logs.rar
FItness function logs.
(160.72 KiB) Downloaded 576 times
krasi0
Posts: 254
Joined: Thu Dec 18, 2008 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby krasi0 » Thu May 12, 2016 9:39 am

Regarding the noise while training, I would still re-introduce an idea that I mentioned in my first post on this topic: "perhaps it makes sense to only train the network once the whole battle has finished (or a timeout of Y frames has occurred) and for the fitness function then consider something like total frames passed (a lower count is better if our side has won...". So basically this is a form of *offline* learning, i.e. during the battle, all units react using the same ANN; the battle finishes, then the ANN trains are adjusted accordingly to the outcome of the battle. Then a new battle is launched with the adjusted ANN weights and so on.
I think it's worth giving a shot.
Please report back if you do! ;)
User avatar
Leckofunny
Posts: 27
Joined: Mon Apr 04, 2016 7:41 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby Leckofunny » Thu May 12, 2016 10:20 am

The neural net doesn't make the decisions during training. The output action is selected by random. Otherwise a fresh neural net would most likely classify any input to one output. Still before the random action is selected, the net has to be fed forward in order to generate training examples. Making random decisions makes sure that a broad range of training examples are covered.

All the gathered training examples of one match are iterated once for training on the conclusion of the match.

With noise I mean conflicting data. So there are data pairs which have the same inputs but are totally different on outputs. So there is always a huge training error.


The current state of my project is that I'm getting close to be done with implementing my feedback logic. So next week the training procedure should generate clean training examples.
krasi0
Posts: 254
Joined: Thu Dec 18, 2008 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: Training an artificial neural network for the micro of marines

Postby krasi0 » Thu May 12, 2016 1:22 pm

Cool! Keep us updated on how it goes next week :)

[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Return to “AI development (BWAPI Bots)”

Who is online

Users browsing this forum: No registered users and 1 guest