Your middle should be rigid enough to cut much deeper than .1mm in wood per pass. My machine can’t handle a plunge rate of 14mm/s because of tool strength not just machine speed.
How far is your material from the gantry?
I cut that Brazilian walnut at 4mm (my full tool depth) in a single pass at 20mm/s, at 4″ from the gantry and easily had room to do much more much faster. That wood is super dense so if you are doing pine or something soft you should be able to go much more per pass. Try roughing using a waterline pass much faster with very few z moves and high step over and only use the contour cut for the final pass with a low step over.
You haven’t given any information to determine if your gantry is loose or you are just trying to cut beyond the machines capabilities. Axis speed is not the only way to get a final product faster more efficient tool paths, larger roughing tools and smaller finishing tools help.
The posts that just say it isn’t fast enough or it needs to be more rigid is bad press, so I do my best to help those people out. But not rigid enough for what material, what gantry distance, What spindle, what spindle speed, what cutting bit, what step over, what depth of cut? Your settings might just be off a bit and the problem would be solved or maybe you just expect too much.