We were there on a Thursday, in term time. It was not crowded and I don't imagine it is ever unpleasantly so. There is plenty of room, both inside and out. The restaurant however is very popular with local people as well and many tables were full when we were there, so we actually chose to eat at one of the outside tables.
We went on a Monday and it was fairly quiet.
According to Google Tuesday and Wednesday, with Monday close behind.
Get quick answers from Palace House staff and past visitors.