Weblate outage since 7am CEST (fixed)

DevOps
1

Bonjour,

Weblate is unreachable as of 7am CEST this morning. This is an electric outage of the hosting facility we’re using.

Fun fact: the coworking space I went to this morning uses server on OVH for membership. Since they are unreachable they reverted to writing membership cards on post-its.

a.jpg800×600 154 KB

Cheers

1 Like
2

Fun comments: http://downdetector.co.uk/problems/ovh/map/
OVH datacenter updates: https://twitter.com/olesovhcom

3

https://horizon.cloud.ovh.net/ is back, connecting to assert the situation

lab.securedrop.club and weblate.securedrop.club are reachable again but seem to have DNS problems

$ ansible --private-key infrastructure_key --user debian -i inventory -a '/bin/hostname' all
icinga-host | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: connect to host 54.36.100.195 port 22: No route to host\r\n",
    "unreachable": true
}
weblate-host | SUCCESS | rc=0 >>
weblate-host.securedrop.club
postfix-host | SUCCESS | rc=0 >>
postfix-host.securedrop.club
bind-host | SUCCESS | rc=0 >>
bind-host.securedrop.club
gitlab-host | SUCCESS | rc=0 >>
gitlab-host.securedrop.club
4

The icinga host did not reboot properly openstack console log show icinga-host, rebooting once more. Assume that’s because of the network.

5

All machines are available now, let’s wait a bit for icinga to assert all systems are ok.

$ ansible --private-key infrastructure_key --user debian -i inventory -a 'uptime' all
gitlab-host | SUCCESS | rc=0 >>
 11:53:25 up 27 min,  1 user,  load average: 0.20, 0.30, 0.17
postfix-host | SUCCESS | rc=0 >>
 11:53:25 up  1:05,  1 user,  load average: 0.00, 0.00, 0.00
bind-host | SUCCESS | rc=0 >>
 11:53:25 up 23 min,  1 user,  load average: 0.00, 0.00, 0.00
icinga-host | SUCCESS | rc=0 >>
 11:53:25 up 0 min,  2 users,  load average: 0.03, 0.01, 0.00
weblate-host | SUCCESS | rc=0 >>
 11:53:25 up 32 min,  1 user,  load average: 0.00, 0.00, 0.00