My grandson is 5, almost every time he has come out so far has involved swinging. At Castle it has been easy to find a steep section, Pinns a bit harder at some spots. Trees near climbs work. We did crack one of Yosomighty's twins' head fairly hard on one swing though, when he twisted around.
I can't think of any thing better than your idea. Good luck.