Weblate down for maintenance

Translations
1

Perhaps we could ask La Quadrature, on IRC ?.. They’re used to crowdsource translations…
Btw, I wanted to correct some mispellings, but when I tried to connect through Github => 502 Bad Gateway
https://weblate.securedrop.club/accounts/login/?next=/translate/securedrop/securedrop/fr/

2

re 502 Bad Gateway it occured to me once as well but I’m no longer able to reproduce it. Does it consistently happen to you ?

3

Actually, and after registrating, 1/3 of the corrections I try to add => 502 Bad Gateway; but when I reload the page, the correction seems to have been registered.

4

This is not good, I’ll suspend the machine and verify/diagnose what’s going on.

5

fsck -f -y on tonight backup is clean. The machine is rebuilt to rule out memory / local storage issues.

6

I’ve been able to reproduce the 502 Gateway on a brand new machine. It is neither disk corruption nor bad memory. Simplest explanation would be that it’s a configuration mistake on my part when I setup the letsencrypt nginx front a few days ago.

7

Now checking the logs for clues as to why 502 happens.

8

$ sudo docker-compose logs --tail=20 -f weblate
…
weblate_1 | [2017-07-31 14:46:56,764] - Broken pipe from (‘172.18.0.5’, 42212)
…
$ sudo docker-compose logs --tail=20 -f web
…
web_1 | nginx.1 | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:14:49:29 +0000] “GET /accounts/login/?next=/ HTTP/2.0” 200 7789 “https://weblate.securedrop.club/” "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1 | nginx.1 | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:14:49:29 +0000] “GET /js/i18n/ HTTP/2.0” 200 3217 “https://weblate.securedrop.club/” "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1 | nginx.1 | 2017/07/31 14:49:31 [error] 3003#3003: *732 upstream prematurely closed connection while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: “POST /accounts/login/github/?next=/ HTTP/2.0”, upstream: “http://172.18.0.3:5432/accounts/login/github/?next=/”, host: “weblate.securedrop.club”, referrer: "https://weblate.securedrop.club/"
web_1 | nginx.1 | 2017/07/31 14:49:31 [warn] 3003#3003: *732 upstream server temporarily disabled while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: “POST /accounts/login/github/?next=/ HTTP/2.0”, upstream: “http://172.18.0.3:5432/accounts/login/github/?next=/”, host: “weblate.securedrop.club”, referrer: "https://weblate.securedrop.club/"
web_1 | nginx.1 | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:14:49:31 +0000] “POST /accounts/login/github/?next=/ HTTP/2.0” 502 173 “https://weblate.securedrop.club/” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0”
…

9

It definitely is a misconfiguration on my part when setting up letsencrypt. Somewhere http is used instead of https and that creates occasional 502 because nginx fails on those.

10

GitHub OAuth is setup with http://weblate.securedrop.club and upon successful login the web container log looks like this:

web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:07:50 +0000] "POST /accounts/login/github/?next=/ HTTP/2.0" 302 0 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:07:59 +0000] "GET /accounts/complete/github/?redirect_state=58F79DwFIfbqZyVqiQ3HhbyOT03gcMV1&code=27308edd0b89628766cf&state=58F79DwFIfbqZyVqiQ3HhbyOT03gcMV1 HTTP/2.0" 302 0 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:08:02 +0000] "GET / HTTP/2.0" 200 41490 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1          | nginx.1    | 2017/07/31 15:08:03 [error] 3027#3027: *907 upstream prematurely closed connection while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: "GET /avatar/32/dachary.png HTTP/2.0", upstream: "http://172.18.0.3:5432/avatar/32/dachary.png", host: "weblate.securedrop.club", referrer: "https://weblate.securedrop.club/"
web_1          | nginx.1    | 2017/07/31 15:08:03 [warn] 3027#3027: *907 upstream server temporarily disabled while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: "GET /avatar/32/dachary.png HTTP/2.0", upstream: "http://172.18.0.3:5432/avatar/32/dachary.png", host: "weblate.securedrop.club", referrer: "https://weblate.securedrop.club/"
web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:08:03 +0000] "GET /js/i18n/ HTTP/2.0" 200 3217 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:08:03 +0000] "GET /avatar/32/dachary.png HTTP/2.0" 200 2970 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
11

When a 502 occurs after clicking on the octocat the web logs show

web_1          | nginx.1    | 2017/07/31 15:06:25 [error] 3027#3027: *907 upstream prematurely closed connection while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: "POST /accounts/login/github/?next=/ HTTP/2.0", upstream: "http://172.18.0.3:5432/accounts/login/github/?next=/", host: "weblate.securedrop.club", referrer: "https://weblate.securedrop.club/"
web_1          | nginx.1    | 2017/07/31 15:06:25 [warn] 3027#3027: *907 upstream server temporarily disabled while reading response header from upstream, client: 80.12.38.252, server: weblate.securedrop.club, request: "POST /accounts/login/github/?next=/ HTTP/2.0", upstream: "http://172.18.0.3:5432/accounts/login/github/?next=/", host: "weblate.securedrop.club", referrer: "https://weblate.securedrop.club/"
web_1          | nginx.1    | weblate.securedrop.club 80.12.38.252 - - [31/Jul/2017:15:06:25 +0000] "POST /accounts/login/github/?next=/ HTTP/2.0" 502 173 "https://weblate.securedrop.club/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
12

For the record the stupid mistake I did was to add

VIRTUAL_HOST=weblate.securedrop.club
LETSENCRYPT_HOST=weblate.securedrop.club
LETSENCRYPT_EMAIL=admin@securedrop.club

to the environment file used by both weblate and the database containers. As a consequence some requests reaching the ssl nginx reverse proxy got redirected to weblate (good) and others to the database (502). Since it’s round robin it explains why fixing the problem was just a matter of "reload*.

13

@manhack it should work reliably now. And I should learn to doubt myself before thinking the RAM is corrupted or other interesting theories based on the idea that the universe is crumbling around me :slight_smile: