Some bits are not meant for plunging, you can also peck plunge. It goes in and out at increasing increments, or ramp in.
Stick with a short 1/8″ bit single or dual flute. 3 and 4 flute is no good. Have a look into chip load. You could very well just be work hardening the material.
I have a video showing my typical aluminum set up before the machine updates so it should be even more rigid/faster now.