Something isn’t right. Can you put up a picture of your middle assembly.
Last night I cut oak ply almost 6mm thick. I cut all the way through + 2mm into HD foam. I cut at 15mm/s full depth dewalt at 50% speed, 2flute down cut bit. Cut perfectly within in easily .2mm, but I was literally holding it down by hand. there was very little force on the wood, like butter. holes and outer paths.
Are you using a proper milling bit?