Picton Bay - swimming

We are looking at booking a cottage in this area. We normally rent north ... Muskoka, Huntsville, Haliburton area but thought we would like to try PEC this year. Swimming and sitting lakeside is important to us (beach not do much, although we will do a day visit to Sandbanks) but Ive read some pretty negative reviews about swimming in this area. So my question is about lake swimming in the Bay of Picton. Will I be able to enjoy clean deep water swims or will I be caught up in the weeds??