As part of the new securedrop.org website logic, we’re trying to integrate Discourse content in the “search” functionality, so that folks searching will find relevant results on this here forum. We’ve got a Discourse API token set up for that, and already have app code logic that queries the Discourse API to update the search index and store it in the website database.
There’s a problem, however: the Discourse server is blocking our requests due to a ratelimit trigger. It looks like the ratelimit is triggered at around 60 or 61 requests. Unfortunately it doesn’t look like we can reduce the number of GETs via smart pagination in the API client calls—the docs imply by omission that this isn’t possible.
Question for @dachary and @fpoulain: Would you be amenable to raising the ratelimit to support the search index updates? We’ll try to isolate certain URL paths if that’s useful to you—it depends on where the ratelimit is occurring. I assume the ratelimit is implemented at the webserver level, rather than the firewall, but it could also be Discourse itself (i.e. application-level). Here’s an example error message from local testing:
429 Client Error: Too Many Requests for url: https://forum.securedrop.club/t/579.json
The “update search indices” requests will likely run on a daily schedule, to strike a balance between keeping the search results current. Happy to work with you on a solution that works for all involved.