Pop up the bezel that the switch is attached to. Look under it and check for a loose connection. The switch itself could be broken from use. Try working the other buttons a few times to get a "feel" for them, then the button in question. If it acts different ( ex. loose), then that could be the problem.
If you still have the owner`s manual for your vehicle, it should have a diagram of the fuse box and each fuse`s location and what it protects. If not, try a google search. OR, your power mirror switch may be bad/worn out. Again, pop that switch off the door panel and check for loose connection. Hope this helps.