When I visited Caus in 2001, I passed through Westbury, took the right hand road, following the castle (on the left) until you come to a very tiny gate entrance where there is a footpath sign, and parked there (on the left).
Caus is a fantastic site and yes, you have to veer off the footpath to experience the evening view from it.
There is a little of the outer bailey gate visible and the path goes up through this and round to the right, into the main part of the woodland. The deep dry moat cut out of the rock here is very impressive. But not more so than a steep climb up into the inner bailey where you will find yourself surrounded by the visible foundations of the great hall and possible round tower, a well (fenced off) and then a steep climb up the motte to the top will reveal the quite substantial remains of the stone tower.
The locals ignore the footpath signs, appreciate they will be sharing the site with sheep or cattle and use it for weekend walks and summer picnics. However, I would stress that the private signs are respected and that the only other choice would be to park in Westbury and stomach the walk to Caus!
Something is not right. This message is just to keep things from messing up down the road