What speed (rpm) did your spindle run at? The power of the spindle sounds more than enough, but most of these motors originally aimed at the woodworker market are too slow to move such a tiny bit through aluminium. A minimum of 15,000, netter 20,000 rpm is necessary. The speed of the cutting edge biting into the material to be cut away is significant. So, a larger mill diameter may be helpful for you.
Above that, did you remove the chippings? They can build up a real wall and get compressed, leading to broken tools. Compressed air and / or a good vacuum cleaner are a must!
Don’t use coolants for VHM tools – they get micro fissures and will crack VHM may get very hot without problems.