Like most any physically demanding sport, physical fitness (nutrition, hydration, strength, flexibility, etc.) is a factor in your ability to perform, and so are training, understanding, and practice. But, in my opinion, personalized coaching, and willingness to BE coached, are extremely important.
I'll give you my perspective: I started riding quite late, in my mid thirties. I was very slow and very nervous and I don't think anyone expected me to have the potential to ride fast, let alone race (least of all me!), but I got really interested in the sport, got lots of coaching, and devoted a lot of time to really understanding the material, and my understanding of the material ABSOLUTELY changed and evolved as I rode faster. Going back and reading Twist II, I found lots of information struck me differently as my pace increased and I found techniques that were a bit vague to me at first became much more important, much more useful to me, because I NEEDED them more. For example, I could get away with slow body transitions at slower speeds but as my laptimes came down, speed of moving across the bike in a chicane became a limiting factor, I couldn't get through a particular section any quicker without moving over faster. Suddenly hip flick, which didn't seem very useful to me before, became a critical skill. That is just one example, but I have had, over the years, a BUNCH of breakthroughs like that, and have found that as I progress in my riding, becoming proficient in certain techniques and riding faster overall, new barriers crop up and as I address each one I get quicker again - and then encounter something else. How do I overcome the barriers? Through coming to school and getting coaching, mostly. Sometimes study of the material helps, sometimes analyzing data (laptimes, braking zones, lines, etc. from my lap timer or data logger) help, but coaching is what always makes the biggest difference - very often what I THOUGHT was my barrier turned out to be something different entirely, and it required the eye of a coach to discover that.
Of course, my mindset while being coached is a huge factor in my ability to improve. If, for example, I came to school fiercely determined that I already knew "what my problem was", then I did not get nearly as much benefit from coaching because I was resistant to allowing the coach to help me. After I figured that out, I got even more improvements on my school days.
I said physical fitness is important, but I am a lot older than many of the riders I race against, and not as fit as most of them, either, but my CSS training allows me to ride with fewer SR's and a lot less wasted effort so I can go faster and be calmer overall. I thought I had reached my riding peak years ago but I am actively racing this year and I am riding faster than ever before. I still get coaching as often as I can, generally I come to school as a student at least 4 days a year, if not more, and that makes a huge difference for me. It is not because I am in better shape, because I am not. It is because I understand and apply the riding tech better than I could before.
I used to think, at the end of any school day as a student, that I was riding as fast as I ever would, because I figured I would just get older and slower.... but I'm not getting slower, I'm getting faster. Every time I come to school I get some new piece of information or tackle a new skill that adds something to my riding, and I get quicker. And what a thrill that is! And it is even better when some twenty year old comes over to your pit to ask you how you do it.