Ah! I think I oriented the z conduit backwards. The tool holder collides with the middle z piece.
The school I work at has a small cnc mill. You are right, it would be easy to face them, but the machine is booked from now till the last day of school for various student projects.