Well, I'm almost a doctor (less than 8 months!), so here is some free info!

-The definition of "infertility" is having unprotected sexual relations with someone consistently for one year (if the female is under 35) and not getting pregnant. Consistently means having relations every other day.
-The chance of getting pregnant in any one month that a couple tries to conceive (having sex every other day) is about 20%, which at the end of 1 year leaves about 7% of "normal" people still not pregnant (ie people with truly nothing wrong with them)
-Around 25% of couples never find a reason for their infertility
-Around 40% of the time, the cause for infertility is the male (which means the other 60% of the time, it is the female)
-Most insurances don't cover infertility workup- but it is a lot easier to workup male infertility than female (anywhere from $50-250 for a semen analysis).

Depending on how comfortable you are with your doctor, it wouldn't hurt to ask about having a semen analysis performed. Your regular PCP should be able request the test if you really want to know(don't necessarily have to go to a urologist), but it doesn't sound like you're ready right now. Most doctors I know don't get embarassed by talking about things like this ie infertility, sex, erectile dysfunction, etc. So it's really up to you to work up the nerve to ask at your next appointment (again, if you want).