How can I stop WindowButtonDownFcn callback from being activated when the zoom icon is clicked?

I have callback functions working for WindowButtonDownFcn as well as up and move. These work fine, but I discovered that if I click the magnifying glass icon, it behaves as if the button is down and stays down until the magnifying glass icon is disabled. This is for a regular figure, not a GUI. How can I stop this? I want the zoom to work as usual.