I've done it both ways personally. If they break, I usually just leave it. If they come out, I put them back in... I've never noticed any problem not having them in.
To remove them I like to use a manual impact tool shown below. When you smack it with a hammer it turns.
Amazon.com: Lisle 29200 3/8" Hand Impact Tool Set: Automotive
