Ansible repository organisation

DevOps
1

Bonjour,

I propose we organize the ansible securedrop-club by following the following conventions:

  • all the infrastructure is deployed with ansible-playbook playbook.yml at the top level
  • playbook.yml at the top level only contains includes to playbooks in molecule. In the example below, it means the molecule/postfix/playbook.yml must be split in two. postfix-playbook.yml which is used for deployment and test-postfix-client-playbook.yml which is only relevant for testing.
  • roles contains home made roles
  • roles-external contains third party roles
  • in each molecule/*/roles directory only roles that are strictly necessary are symlinked to the top level roles and roles-external directories. This hopefully helps figure out the role dependencies.
.
├── ansible.cfg
├── clouds.yml
├── group_vars
│   └── all
│       ├── clouds.yml -> ../../clouds.yml
│       ├── secrets.yml
│       ├── weblate-secrets.yml.example
├── molecule
│   ├── postfix
│   │   ├── create.yml -> ../infrastructure/create.yml
│   │   ├── destroy.yml -> ../infrastructure/destroy.yml
│   │   ├── group_vars
│   │   ├── molecule.yml
│   │   ├── playbook.yml
│   │   ├── postfix-playbook.yml
│   │   ├── pytestdebug.log
│   │   ├── roles
│   │   │   ├── debops.ansible_plugins -> ../../../roles-external/debops.ansible_plugins
│   │   │   ├── debops.etc_aliases -> ../../../roles-external/debops.etc_aliases
│   │   │   ├── debops.ferm -> ../../../roles-external/debops.ferm
│   │   │   ├── debops.postfix -> ../../../roles-external/debops.postfix
│   │   │   └── debops.secret -> ../../../roles-external/debops.secret
│   │   ├── test-postfix-client-playbook.yml
│   │   └── tests
│   │       ├── __pycache__
│   │       │   └── test_postfix.cpython-27-PYTEST.pyc
│   │       ├── test_postfix.py
├── openstack.py
├── playbook.yml
├── README.txt
├── requirements.in
├── requirements.txt
├── roles
│   ├── icinga2
...
│   ├── icinga2_client
...
│   └── weblate
...
├── roles-external
│   ├── ansible-role-docker
...
│   ├── bertvv.bind
...

Cheers

1 Like
2

It looks nice for me.

3

Looking good! Thanks for sharing the strategy you’re using.

Why use another nested roles directory at molecule/postfix/roles? Personally, I prefer to have external roles locked to specific versions via git submodules in the top-level roles-external/ directory. Then the roles will be reusable for all scenarios and playbooks, and guarantee consistent behavior. If you’re running the playbooks via Molecule, then you may need to futz with the roles_path setting in ansible.cfg to ensure Molecule finds the roles.

Nice use of the debops work. :ok_hand: That’s a project I’ve had an eye on for a long time, and while we do not use it internally yet, it seems very attractive. It would likely take a lot of work to migrate our entire infra repository to use debops, and debops feels like an “all-in” configuration strategy, meaning it’s difficult to use just select portions of it, rather than switching to it entirely as an Ansible wrapper. Would like to hear more about how you find its usage together with Molecule.

4

We followed your example regarding the roles and roles-external at the top level. I meant write symlinked to the top level roles and roles-external directories instead of copied, sorry for the confusion. It would indeed be unnecessary to copy the roles in each molecule subdirectory :wink:

P.S. Updated the topic description accordingly

5

That’s exactly the reason why I hesitated to use it for the postfix role. What bothered me most is the requirement on the firewall. But since we’re using firewalls that are outside of the VM control, there won’t be conflicts there. The rest is not too intrusive. It did not cause trouble with molecule testing and did not interfere with the other roles we re-use. So far so good, fingers crossed and all that :wink:

6

I pushed a reorganized repository @fpoulain make sure you get it before starting anything, things moved around :wink: I removed the obsolete molecule/icinga directory and your molecule/monitoring_client has been renamed molecule/icinga, which makes more sense.

7

Ok. Moreover, since deploy_dummy_monitoring_objects is a dummy role (here in a testing context, not going to be reused, although it may provides some snippets) which makes only sense in the context of molecule/icinga, we should move it there.

1 Like
8

I wonder where does monitoring stuff should go. For internal roles, I started to add a task list name monitoring.yml. Example given in roles/weblate/tasks.

For the postfix role, I appended monitoring stuff to main.yml but imho it should moves to an included file.

Some monitoring configurations may be specifics to a given scenario. Then it is easy to add a specific role (like deploy_dummy_monitoring_objects for the icinga scenario).

For external roles, I can’t do this without forking the submodule, and that’s not what we want. Does git allow a kind of overlay over submodules? Or have we to create specific monitoring roles to be called from scenario playbooks?

9

I think it would be cleaner indeed. For firewall rules I did not create a separate file but I should have.

10

I started with specific monitoring role, at least temporarily.

11

I think it would make sense to have a monitoring_client role that some playbooks can use, rather than having a file inside the role. That’s what I did for the firewall for instance. Maybe that’s exactly when you meant by I started with specific monitoring role.

12

For now I did this in http://lab.securedrop.club/main/securedrop-club/merge_requests/6

  • a new role has been done: monitoring_bind, with some needed dependencies;
  • since we do not have handlers for icinga configuration, the icinga host should be provided early in the scenario;
  • I added role in the bind-playbook. I try here to be generic;
  • I test at least on an object that the monitoring configuration has been handled (icinga service (check) existence);
1 Like
13

The proposed organization where all molecules/*/roles directories have symlinks is inconvenient to maintain. The higher level roles (like weblate) have symlinks to the low level roles (like firewal) and that’s a lot of duplications.

I’ll try something else instead, which is what Mike and @conorsch did in SecureDrop: setting

  env:
    ANSIBLE_ROLES_PATH: roles:../../../infrastructure/roles

etc. in the molecule.yml files. And ansible.cfg at the root of the repository has something like:

roles_path = molecule/infrastructure/roles:molecule/bind/roles:molecule/icinga/roles:molecule/postfix/roles:molecule/weblate/roles

We have some redundancy in maintaining the list of directories in which roles are stored. But much less than symlinking each role. I realized how tedious it would be when splitting the firewall role out of the vm role. I had to create a symlink in all molecule/*/roles directory because they all indirectly depend on it.

14

I made the change, and it is indeed better. Hopefully that’s the last major reorganization of the directory tree.