Assuming you mean Ashford in Kent, probably the fasted way would be to get a train to London and then another from Waterloo that stops at Brockenhurst. There you catch a third train that goes to Lymington Harbour anf the ferry to Yarmouth. Another option, shorter in distance but with a lot of intermediate stops, would be train to Brighton and change there for one to Portsmouth Harbour where you could get a catamaran to Cowes and then a bus. Check with http://www.nationalrail.co.uk
Going via Portsmouth requires you to get down the pier at Ryde (walk or train) and two buses (9 to Newport and 7 to Yarmouth), at least an hour and a half of additional travel, with the additional expence, besides the route via Lymington is far more scenic (the New Forest and the Solent Crossing). Journey times, via Yarmouth just under 3 hours, via Portsmouth about 4 hours (including buses). Living near Yarmouth, I know the Yarmouth route is far easier, ferries run every 40 minutes and trains every half hour.