That's making an assumption that they will send back the A/AAAA records. Empirically, you might be right, but it's a recommendation, not a requirement, in RFC2782. (Not sure if there's an RFC that supercedes that particular point.)

So, either you hope the server responds with the A/AAAA records in the additional section, or you have to query for all records on an RR, or further still, do multiple queries. What happens when your SRV records point to CNAMEs? Do most DNS servers that support sending back the A/AAAA records in the ADDITIONAL section also support resolving the CNAMEs before populating the additional section?

There's a few other things, too,, like having to make interesting tradeoffs on TTLs: if you have low enough TTLs to support using DNS as a near-real-time configuration of what web servers to use, what happens when DNS itself breaks? There's some operational pain there, to be sure.

This is all to say: there's clearly a lot of angles to something as simple as using SRV records in lieu of A/AAAA/CNAME records, and we're here, right now, talking about this, all because of the rushed design of HTTP/2.0, which is a protocol unto itself. It's not surprising that a standard that went through so quickly managed to not include something, like SRV records, which have been in a weird state of existence since their inception. To think it would be so simple, so easy, seems incredibly overoptimistic.

If you have this problem, you are in control of what DNS server you use, and can make sure it sends the appropriate records. We are talking about a “problem” which affects very few people, and those it affects have the budget to make sure this is the case.

Also, there is, by now, a lot of operational experience with both MX records and SRV records, and they are well understood. They are not the wild unknown you make them out to be.