The chatter you are hearing is likely from either A, a lack of rigidity or B, your workpiece is resonating. If going to a thinner stock made it worse, then yeah it is probably that it is moving too much during cutting. I see you clamped it quite well on the sides, but try using some wood screws in the center to hold the center down more. Do you find that the chattering goes down a little when you cut closer to a location of a clamp? Also, your z axis is rather large so that doesn’t help much.
Is your bit perpendicular with your workpiece?
Have you faced/leveled your base mdf?