As part of the new securedrop.org website logic, we’re trying to integrate Discourse content in the “search” functionality, so that folks searching will find relevant results on this here forum. We’ve got a Discourse API token set up for that, and already have app code logic that queries the Discourse API to update the search index and store it in the website database.
There’s a problem, however: the Discourse server is blocking our requests due to a ratelimit trigger. It looks like the ratelimit is triggered at around 60 or 61 requests. Unfortunately it doesn’t look like we can reduce the number of GETs via smart pagination in the API client calls—the docs imply by omission that this isn’t possible.
Question for @dachary and @fpoulain: Would you be amenable to raising the ratelimit to support the search index updates? We’ll try to isolate certain URL paths if that’s useful to you—it depends on where the ratelimit is occurring. I assume the ratelimit is implemented at the webserver level, rather than the firewall, but it could also be Discourse itself (i.e. application-level). Here’s an example error message from local testing:
429 Client Error: Too Many Requests for url: https://forum.securedrop.org/t/579.json
The “update search indices” requests will likely run on a daily schedule, to strike a balance between keeping the search results current. Happy to work with you on a solution that works for all involved.
PS: Currently, the forum has been manually deployed from docker upstream image. Migration to the ansible repo/deployment should occurs in next weeks/months.
@conorsch Would it make sense to whitelist FPF’s VPN IPs for dev as well as the new securedrop.org server?
I’m not 100% sure how many requests are required for us to index the posts, but we reindex them completely every time and Discourse’s API requires a lot of pagination, so this will grow with the size of the forum. I’d estimate at the current size ~12 requests just to get a list of topics and then at least one request per topic (more if the topic has a lot of comments) so my guess is that we’ll currently clock in somewhere below 200 requests right now. If this is or becomes a major issue, we might look into a different approach (like maybe using discourse’s webhooks to update the index).
There’s no bursts behavior right now. I am fairly sure that the script fires the requests consecutively. I can add in a delay if that would be helpful.
@fpoulain I’d prefer not to use IP whitelisting here, since the systems are managed separately, and any changes on the securedrop.org webserver side, e.g. due to rebuild, would break the whitelisting, and require that we correspond with you about the change so you can implement a fix again.
Would you be amenable to loosening the ratelimit a bit on the URL routes the API will hit? We can generate a list of routes, e.g. /t/*.json or similar, and then only those routes would have the ratelimit adjusted, and you can preserve the existing ratelimit site-wide.
Will work on generating a list of the expected GET requests from the API consumer, to confirm that the routes are predictable and appropriate for the selective ratelimiting strategy proposed above, and share them here.
Is there an issue in securedrop.org somewhere that explains how the import from the forum is designed? I’m not sure I understand why it would take over 60 requests to just get the latest messages from the forum. But (disclaimer) I’m ignorant of the API
It feels like we should not be having this problem at all because the activity of the forum is way below the threshold.
@dachary The Discourse API, sadly, doesn’t provide any way of identifying or retrieving latest posts, just latest topics, which doesn’t let us know if there’s new content to index under an old topic. As a result, we’re doing a complete reindexing of all topics every time we run the script, hence the large number of requests.
The script does these things:
Paginates over the entirety of latest.json to generate a complete list of public topics (the URL for this endpoint is sort of a misnomer—it makes available all topics sorted by date created)
Gets details for each topic at /t/{topic_id}.json
If a topic has posts, paginate through those posts at /t/{topic_id}/posts.json
Using Discourse’s webhooks to get new partial content updates immediately might be a superior alternative to this sometime down the line, but, unfortunately, I don’t think I have time to build that out before our planned launch for securedrop.org, so it’d be great to get this working, even if as a stogap.
# we bypass limits for json
location ~ ^/(latest|t/).*\.json {
add_header Referrer-Policy 'no-referrer-when-downgrade';
add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $thescheme;
proxy_pass http://discourse;
break;
}
@fpoulain I’m still getting 429 after 61 requests. I’m not sure why. I’m going to experiment with building in a delay before retrying after getting that error, but I’m still confused.
I managed to resolve this by sleeping the script for 30 seconds and retrying every time it encounters a 429 response, so I think we’re actually good here regardless, @fpoulain. Thanks for your help. Having successfully run it, I can now state with certainty that with 380 forum topics, the script requires 410 requests to index the entire thing! I don’t think we’re gonna knock over the server or anything, but still Yikes! We’ll be looking into alternative options for this process sometime after launch.
Thanks, @fpoulain, that’s quite helpful. Will check our logs and compare, I’m optimistic we can iron something out on the request rate that’ll work for all involved.
Looks like we’ve got successful index updates via Discourse! The state of the config at present works well for our needs. Thanks again for your collaboration here, @fpoulain!
