Server upgrades

DevOps
1

I manually upgraded servers and applied secu upgrades, since we don’t have yet a playbook for this (It is probably straightforward but I would like to test it…). We should reboot when possible.

for vm in packages weblate postfix bind icinga; do ssh debian@"$vm"-host.securedrop.club sudo apt update; done
for vm in packages weblate postfix bind icinga; do ssh debian@"$vm"-host.securedrop.club sudo apt upgrade; done
# verify diffs and keep installed config files
for vm in packages weblate postfix bind icinga; do ssh debian@"$vm"-host.securedrop.club sudo etckeeper commit 'debian upgrade'; done

For gitlab:

ssh debian@gitlab-host.securedrop.club -p 2222

sudo apt update
sudo apt upgrade
sudo etckeeper commit 'debian upgrade'
2

My current workflow on this topic:

  1. If not already, subscribe to debian security announces.
  2. current configuration of unattemded upgrades apply automatically security pugrades when no deps are involved.
  3. define in your ~/.ssh/config (to be updated when needed; personally I don’t like wildcards in ssh Hosts because it impedes completion):
Host gitlab-host
  Hostname %h".securedrop.club"
  User debian
  SendEnv GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL
  Port 2222

Host ansible runner-host bind-host icinga-host packages-host postfix-host gitlab-host weblate-host website-host demo-host bots-host chat-host cloud-host
  Hostname %h".securedrop.club"
  User debian
  SendEnv GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL
  1. update/upgrade in a for loop:
for host in gitlab-host ansible runner-host bind-host icinga-host packages-host postfix-host gitlab-host weblate-host website-host demo-host bots-host chat-host cloud-host ; do ssh $host sudo apt update ; done
for host in gitlab-host ansible runner-host bind-host icinga-host packages-host postfix-host gitlab-host weblate-host website-host demo-host bots-host chat-host cloud-host ; do ssh $host sudo apt upgrade ; done
3

From the ansible machine we could do

$ ansible --private-key infrastructure_key --become -i inventory -a 'apt-get update' all
chat-host | SUCCESS | rc=0 >>
Hit:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch Release
Hit:4 https://download.docker.com/linux/debian stretch InRelease
Reading package lists...
bind-host | SUCCESS | rc=0 >>
...
Fetched 628 kB in 0s (1,030 kB/s)
Reading package lists...
(virtualenv) debian@ansible-host:/srv/checkout$

Is there a reason why that would not work as well?

4

Yes that would. Maybe we have to limit parallelism and verify that apt interactions (questions, conf management) are allowed.

1 Like
5

If there is no way to use the information from the inventory and we need ssh config to run commands interactively, maybe we could create it at the same time we create the inventory? So that we do not need to maintain it separately. But that would still not allow us to write ssh all hosts dothis…

6

I use it for personal convenience (I prefer short explicit host names in ~/.ssh/config rather than wildcard because it allows efficient completion). But it could be a good idea to generate it from the inventory.