from downtown Seattle you just take the
goes direct to downtown Victoria and you can buy a direct ticket right to the gardens. Its passenger only, but you wouldn't need a car anyway as everything is right there, I do this a lot and the time is about the same even from Vancouver when you factor in the driving, wait for the ferry then ferry ride over plus drive to gardens and into downtown Victoria. Sort of depends on where your starting from.
If you take the Clipper from downtown Seattle to downtown Victoria, then you can either take public transportation out there (about 1 hour one way), take a tour bus, or a taxi (believe that would cost about $50 one way).
If you come over from Vancouver you can sometimes take a public bus directly to the gardens. The ferry is a lot closer to the gardens than the Clipper, so a taxi may cost about $25.
This all depends on the time of day and month that you are travelling.
It is a bit of a challenge getting from Vancouver to Vancouver Island by public transportation. (Is that what you meant by "by ourselves"?) You would take the ferry from Tsawwassen (about 40 minutes south of Vancouver) to Swartz Bay. If you are staying in downtown Vancouver, you would take the Canada Line (rapid transit, partly underground) to Bridgeport Station and then catch the #620 bus to Tsawwassen. The ferry ride, itself, is lovely, but about 90 minutes long. You would then take the #81 bus from Swartz Bay to Butchart Gardens (or taxi as suggested by a previous poster) -
All in all, allow 4 hours each way.