I will put in my 'two cents' worth. I visited Yosemite for the first time the week of July 4th 2005 and agree that two days is appropriate. My boyfriend had his cast off from a broken foot for only three weeks when we visited so backpacking was not really in the picture and we drove most of the time. We did walk to the giant trees, one mile downhill on the way there, one mile uphill on the way back. That pretty much took a toll on his foot so we drove most of the time (both in our early 40's and in good shape, normally we would have done a lot more backpacking).
If you are going to do more driving than hiking, two days is plenty. Yosemite is not as large as some of the other National Parks (Yellowstone/Grand Teton, Grand Canyon) and you will cover most of it. You will definately need two full days (at least 8 hours each) if you want to due some hiking/backpacking to the really cool spots.