An out of spec caster will not cause a pull concern. Camber and toe being out will. However the only thing you can adjust is the front toe, being that the rear is a dead axle. If you install camber bolts, then you will be able to adjust the front camber slightly, which may be all you need. Have you checked your control arm bushings?