In http://undocumentedmatlab.com/articles/improving-graphics-interactivity there is a suggestion about disabling unwanted interactivity behaviors while preserving zoom and pan (but you can choose what you like...):
hAxes.Interactions = [zoomInteraction regionZoomInteraction rulerPanInteraction];
hAxes.Toolbar = ;
The first attempts to apply the suggestion failed like described by Bruno, because of the reset of the default axes Interaction property at line 53 of clo.m (<MATLABROOT>\toolbox\matlab\graphics\private\clo.m).
By adding lines 54 and 55 in clo.m (see below) everything works like requested.
This is encouraging, but a serious analysis of the side effects was not done yet.
52 obj = handle(obj); % In case of double handle
53 obj.clo(hsave, (do_reset == 1)); % Call clo method on graphics class
54 obj.Interactions = [zoomInteraction regionZoomInteraction rulerPanInteraction];
55 obj.Toolbar = ;