I know that driving race cars is pretty physical - more physical then the layman would think. Here's a couple interesting threads about the topic:

http://www.ekartingnews.com/viewtopic.php?t=81028
http://www.teamjuicyracing.com/forums/showthread.php?t=2270

The conclusion from the limited research I've done on the topic seems to be that the arms, neck, shoulders, upper back and calves are important muscles used for driving race cars. Conventional resistance training covers most of these muscle groups with the exception of the neck. The threads above mention some ways to build up neck strength.

My workouts are honestly pretty light and often few and far between. I usually do 20-30 minutes of cardio - usually running or an elliptical machine, and few resistance exercises - mostly the normal bench, rows, pull ups, leg press, etc. The most unconventional exercise I do is the "neck curl" described in the karting thread. Basically I lay on my back and/or sides with my head hanging off a bench or my bed with a weight on top of my head, and pull my head up and down or side to side.

After my first 3 days in a formula car, I found my back and shoulders were pretty sore, my arms weren't really sore at all, and my neck was just barely sore. The rest of my body was not sore at all, besides a few bruises I had on various parts of my body form banging into unpadded parts of the car. Surprisingly, my neck was not as sore as i expected it to be, although I don't think an older formula car on street tires pulls that many g's in the corners. I'm sure a modern F1 or Indy car would be a different story. Also surprisingly, my most sore muscles were my right hand fingers, from my death grip on the shifter! I could barely make a fist with my right hand a few days later.

In terms of fitness, I wasn't physically tired at all after any of the sessions in the car. Then again, I was only in the car 15 minutes at a time, with long breaks in between. Mentally, I knew that it would be very strenuous, but it was even more so then I expected. Although I did not feel mentally fatigued or spent like I used to in college or sometimes at work - at least not immediately after getting out of the car, I think it was just adrenaline keeping me going during the day. I was passing out pretty early back at the hotel. Driving the car - I don't think I've ever been more focused on anything else in my life, and it is draining.

So overall, it was less physically demanding and more mentally demanding then I expected.