The reason for this is pretty simple. Most companies work in a Waterfall approach, and they know it doesn't work. They know this intuitively, and they can prove it with case studies, both their own and from online sources.
From time to time, I encounter people who say Waterfall is OK for them. What's interesting here is they are willing to try Agile. If it's better great, if not they can go back, so whats the problem?
Convincing people to leave Waterfall is not that hard. Honestly, more people than you think are willing to at least give it a go.
I don't think the challenge is persuading people to try Agile. It's persuading them to try Agile again, after they have tried Fragile.
Fragile is the evil twin of Agile. It is far worse than Waterfall. It's chaotic! Managers feel out of control, they say "I don't like this! I have no idea what the developers are wasting time on". Nothing gets finished, nothing is tested, and nothing at all gets delivered. There is not even the illusion of progress that you get with Waterfall.
The Agile process becomes fragile for a number of reasons. Typically the team (including the business) is new to Agile. They like the idea, but have no Agile experience or skills. They don't understand why any of the practices are important. More importantly, when the early warning signs start to show, they don't recognise them, and so take no corrective action.
Things go wrong, typically by having iterations that deliver nothing, and this is bad. It's very bad. Someone in management stood up for Agile. They promised amazing productivity, weekly delivery, and all the things Agile is supposed to give. The acceptance of Agile was a tentative arrangement, and early failures cause it to become rolled back.
I think most teams fail the first time they try Agile if they do it alone. I have no hard numbers to back this up, but I suspect it's true. They just don't have the skills yet. Agile hasn't yet produced the team.
This last point is the key one. Agile has two main deliverables. It produces continuously improving software AND it produces a continuously improving team.
The first try will almost certainly go wrong. But the team will learn from this process. They will understanding Agile more as they see what happens when it fails. Failing is how we learn things. As children, and as adults.
But once they have seen it fail, they will want to try again. This time with the knowledge of what went wrong last time. The second time a team tries an Agile project, I'm guessing it normally works. It won't be perfect, and almost certainly gets lots of individual things wrong, but the important thing is it works. The continuous improvement process has been bootstrapped. With any luck the idea of continuous improvement was one of the things they learned, so future improvements will happen faster.
This is the message that we, as the agile community, need to spread. It's no good telling people that Agile is better than Waterfall. Instead we need to tell people how Agile teaches teams, and makes them better, and to find a way to persuade the business to give it a long enough chance for the tam to develop into a better team.