Hi Ben,
I pay close attention to this myself, and notice a definite difference when I do.
Here are some areas that I target and would recommend to others:
1. Abdominal muscles. This will help you hold up your body, instead of strangling the bars.
2. Groin muscles.
3. Calf and quads.
Both 2 and 3 are for holding onto the bike as well. More strength and flexibility in these areas will allow you to hold on better while hanging-off, and while bracing under heavy braking. These are the areas that feel the weakest or are sore after that first rusty school of the year, and no prior work-out.
Of course cardio is good for overall endurance, but personally, strong legs really pays off.
Hope this helps.