Snake bites its tail problem automating configuration of Certbot with Ansible

Question

I have a public web-site with Nginx in front of it. We use free Letsencrypt certificates to set up https. These are acquired and renewed by Certbot program. I want to implement entire servers configuration with Ansible playbooks. These playbooks have to:

1. Automatically setup everything from scratch on a set of freshly dispatched empty CentOS instances.
2. Be able to setup with a different domain name if necessary
3. Be completely idempotent so I can make small changes and just run them as is.

There are two similar problems I faced trying to achieve it.

The first is I need Nginx already running to fulfill challenges to acquire certificate and then run Nginx with different config to make use of now acquired certificate. When I acquire certificate Nginx has to serve some files for Letsencrypt to check. This incurs several problems:

1. Playbook has to run Nginx twice in one go with different configs. This is tricky to express using Ansible because of descriptive nature of playbooks. Nginx now has different states in just one Playbook execution. I no longer express state but rather instructions.
2. Using several slightly different configs means either a lot of Nginx config code repetitions or complex and confusing ifs in them.

The second problem is that I wanted to put all my Nginx configuration under VCS so I want playbooks to come with already written Nginx configs. However this not the way recommended by Certbot. Certbot wants you to use a command that parses and then generates Nginx configs to add a certificate there. This may seem convenient but I this way I cannot be sure the results are going to be completely reproducible. Suppose I acquire cert and generate config, then change something by hand and then I lose my servers and domain name and try to run them from scratch. Then some new problems may occur which otherwise wouldn't have. My Nginx configs are complex with many services on different ports, conditions and other details. I have to control how it looks like.

I considered making several different playbooks for different cases. But this also leads to code repetitions and makes using them more complex for a newcoming employee. And rarely used playbooks or scripts may become out of sync.

I did achieve a solution that passes all these criteria. But I am not happy with it because it turned out rather complex with Nginx configuration happening twice and many ifs and booleans variables. And it's not how Ansible playbooks are intended to be written. Is there a good way to do all this?

Answer 1

Don't read the text written on Certbot site, the real talk is in the documentation.

In order to make less actions, consider grouping configs by domains. Plays for a playbook in order to achieve what you want:

Deploy certbot and nginx with no config files, then for each domain on host:

1. Deploy an nginx config for default server with location /.well-known/acme-challenge { root /etc/letsencrypt/challenges; }
2. Reload nginx config
3. Check existence of a certificate for the domain in /etc/letsencrypt/live/domain
4. If cert is already presented, end the run
5. certbot certonly -n --web-root=/etc/letsencrypt/challenges --keep -d domain to make a cert request
6. Check for cert in a retry loop
7. Fix permissions for live certificates
8. Deploy normal nginx configs

You may deploy a stub config per domain instead of using the default server. The playbook could be run asynchronously for each domain.