I drove the car today for about 12 miles round trip. Much to my dismay, the problem happened again with the a/c OFF. Here are my observations:

It seems to happen mostly when you start out going up hill. I am starting to think this is a shifting problem. There are only ~46000 miles on the car.

Also, it didn't start again until the engine was warmed up. The temp guage was a little bit below half way. (not very accurate)

What do you think now?