fix: update how swaggerUI fetches initial documentation for v2

[pleary]

• Changes the way swagger documentation is loaded from giving it a schema directly, to having it fetch the schema from a URL
  • This causes swaggerUI to automatically include the schema URL in the documentation, which is why this PR removes that bit of the V2 API description
• Allows bots to access favicon.ico

[caseyhofford]

Looks good, thanks!

[jwidness]

Can I ask a related question while you're looking at bot access to public docs -- the www doc pages (/pages/api+reference, /pages/developers, /pages/api+recommended+practices, /pages/api+deprecated) are still behind cloudflare, is that intentional?

[pleary]

We intentionally applied CloudFlare protections to just about everything, and have made exceptions for certain pages. It was not intentional to not make exceptions for these pages, it just hasn't been identified as a problem. Since those API reference pages are not likely to be vectors for abuse, they don't necessary need CloudFlare protections. But, is there a specific reason to remove them? CloudFlare allows different rules for different requestors, so if the consideration is indexing in search engines, it's not safe to assume search engine bots are hitting the same checks users are

[jwidness]

Thanks for the response! I think as more and more people are coding with AI agents, it would be useful for those agents to be able to directly read the docs pages. I have had to repeatedly tell my agents not to hit those pages (because they get 403'd), and instead I manually feed in the relevant information. I suspect many people are unaware of these pages and their agents may be missing information that would help them stay compliant with recommendations.