It's still early days for Guix's guix deploy, but it may well be my server deployment tool of the future. I'm quite excited!

guix deploy promises to simply and reliably drop predefined operating systems onto remote machines; including services, configuration, packages and data. In the past I've used Ansible, Salt, Puppet and Fabric for this purpose, but guix deploy has a slightly different approach. Rather than "massaging things until they look right", guix deploy uses Guix's superpowers to declare the whole operating system precisely and drop it into place. It's like functional programming for operating systems. Rather than templated YAML, guix deploy also uses a proper programming language; one of the reasons I still heavily use the Fabric deployment tool at work.

This approach could significantly reduce the maintenance burden on deploying production software. The experience should hopefully be similar to using a platform-as-a-service system but more flexible, using commodity virtual machines rather than proprietary infrastructure and without needing complicated orchestration systems such as Kubernetes.

Overall strategy

The overall strategy here is to:

1. Create a minimal and generic base Guix operating system image using guix system image.
2. Manually provision a cloud server with this base image.
3. Use guix deploy to replace the generic configuration with the specific configuration we want.

Note that we don't run a traditional installer at any point. In the future, guix deploy may also be able to provision the remote server for a range of cloud hosting providers. There is initial support for provisioning on Digital Ocean, but I don't use this provider.

A web server from scratch

1. Preparation: You'll of course need Guix installed on your local machine or a remote virtual machine. For experimentation, it's convenient to generate images and run guix deploy on a temporary remote virtual machine to save uploading lots of large files on a slow internet connection. I'd suggest provisioning a Debian VM and installing Guix. Down the track I intend to deploy directly from my laptop.

   Wherever you're deploying from, create a new directory for this experiment, change into it and generate a new SSH keypair with ssh-keygen.

2. Create a minimal base image: Create a basic a Guix System config in vm.scm. Mine looks like this:

   (use-modules (gnu))
   (use-service-modules networking ssh)
   (use-package-modules bootloaders ssh)
   
   (operating-system
    (host-name "vm")
    (timezone "Etc/UTC")
    (bootloader (bootloader-configuration
                 (bootloader grub-bootloader)
                 (target "/dev/vda")
                 (terminal-outputs '(console))))
    (file-systems (cons (file-system
                         (mount-point "/")
                         (device "/dev/vda1")
                         (type "ext4"))
                        %base-file-systems))
    (services
     (append (list (service dhcp-client-service-type)
                   (service openssh-service-type
                            (openssh-configuration
                             (openssh openssh-sans-x)
                             (permit-root-login #t)
                             (authorized-keys
                              ;; Authorise our SSH key.
                              `(("root" ,(local-file "id_rsa.pub")))))))
             %base-services)))
   

   Generate an image from this config using guix system image --save-provenance vm.scm. This will return the path of the new ~1.5GB image in the default "efi-raw" format. If you're creating the image on a remote VM you can publish it quickly by installing Nginx and building with rm -f /var/www/html/latest.raw && guix system image --save-provenance --root=/var/www/html/latest.raw vm.scm, which will then be available at http://yourvm/latest.raw.

   If your hosting provider supports it, using --image-type=qcow2 is much more space efficient at ~500MB. My provider doesn't support QCOW2, but I know that Digital Ocean does.

3. Provision the server. After logging into my Binary Lane hosting control panel, I select "Add Cloud Server". I learned that I had to select "BYO ISO" here because it turns off their auto-magic storage resizing feature which chokes on the multi-partition Guix image; but that's probably not applicable to other providers eg. Digital Ocean. The machine will be provisioned and assigned a hostname I'm taken to the image upload screen.

4. Upload the minimal base image as a backup and restore it. Rather than running an installer ISO, we'll upload a full disk image. Select "Settings" (cog), "Snapshots, Backups & ISO Management". Select the "Upload" tab. Choose "Create a new temporary image", upload from "Local file" (if building locally) or "HTTP server" if building on a remote VM. Provide your image created in step 2 and select "Start Upload". After that's uploaded, select "Restore this Backup". The machine should reboot into the new image. In the recovery console, I can see the machine has successfully booted.

5. Test the connection to your new server. Run ssh-add id_rsa to load the key we created earlier to your keyring and then SSH to the server using the hostname or IP. In my case the generated hostname is tobacco-rebel.bnr.la, so I use ssh root@tobacco-rebel.bnr.la.

6. Resize the filesystem to use the full storage allocation. Our image was only 1.5G, but my provider initially allocates 20G of storage. To use this, log into the VM and run cfdisk, select the /dev/vda2 root partition, select "Resize", "Write" and type "yes". Then resize the filesystem to match by running resize2fs /dev/vda2.

7. Create a full guix deploy configuration in deploy.scm. My operating-system section is significantly the same as above, with additions for the Nginx web server and to authorise my locally generated packages:

   (use-service-modules networking ssh web)
   (use-package-modules bootloaders ssh)
   
   (define %system
     (operating-system
      (host-name "tobacco-rebel")
      (timezone "Etc/UTC")
      (bootloader (bootloader-configuration
                   (bootloader grub-bootloader)
                   (target "/dev/vda")
                   (terminal-outputs '(console))))
      (file-systems (cons (file-system
                           (mount-point "/")
                           ;; Must be vda2 or you won't be able to reboot after `guix deploy`.
                           ;; This is because our base image makes an EFI partition at vda1.
                           (device "/dev/vda2")
                           (type "ext4"))
                          %base-file-systems))
      (services
       (append (list (service dhcp-client-service-type)
                     (service openssh-service-type
                              (openssh-configuration
                               (openssh openssh-sans-x)
                               (password-authentication? #false)
                               (permit-root-login #t)
                               (authorized-keys
                                ;; Authorise our SSH key.
                                `(("root" ,(local-file "id_rsa.pub"))))))
                     ;; Security updates, yes please!
                     (service unattended-upgrade-service-type)
                     ;; Note that Nginx isn't automatically restarted during
                     ;; `guix deploy`, so run `herd restart nginx`.
                     (service nginx-service-type
                              (nginx-configuration
                               (server-blocks
                                (list (nginx-server-configuration
                                       (server-name '("tobacco-rebel.bnr.la"))
                                       (listen '("80"))
                                       (root "/var/www/")))))))
               (modify-services %base-services
                 ;; The server must trust the Guix packages you build. If you add the signing-key
                 ;; manually it will be overridden on next `guix deploy` giving
                 ;; "error: unauthorized public key". This automatically adds the signing-key.
                 (guix-service-type config =>
                                    (guix-configuration
                                     (inherit config)
                                     (authorized-keys
                                      (append (list (local-file "/etc/guix/signing-key.pub"))
                                              %default-authorized-guix-keys)))))))))
   
   (list (machine
          (operating-system %system)
          (environment managed-host-environment-type)
          (configuration (machine-ssh-configuration
                          ;; Use host name or IP address here.
                          (host-name "tobacco-rebel.bnr.la")
                          (system "x86_64-linux")
                          ;; Update this!
                          (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKp5IsRNi/qU2vrWNaH9MlZnOzN4umiEXkamScuwxF4M")
                          (user "root")
                          ;; Use this key to communicate with the machine.
                          (identity "id_rsa")
                          (port 22)))))
   

8. Deploy the config with guix deploy from your local machine. Guix will initially complain about the host-key, so you'll need to update your configuration with the host key of the new machine:

   $ guix deploy deploy.scm
     tobacco-rebel
   
   guix deploy: deploying to tobacco-rebel...
   guix deploy: error: failed to deploy tobacco-rebel: server at 'tobacco-rebel.bnr.la' returned host key 'AAAAC3NzaC1lZDI1NTE5AAAAIKsuev1SJ3SODqkHsRX4oWib6e6A3MiS9CArvXZhmNbq' of type 'ed25519' instead of 'AAAAC3NzaC1lZDI1NTE5AAAAIKp5IsRNi/qU2vrWNaH9MlZnOzN4umiEXkamScuwxF4M' of type 'ed25519'
   

   After correcting the host key:

   $ guix deploy deploy.scm
   The following 1 machine will be deployed:
     tobacco-rebel
   
   guix deploy: deploying to tobacco-rebel...
   ...
   guix deploy: successfully deployed tobacco-rebel
   

   If you run guix system list-generations on the new server, you'll see that there is now a new generation for each guix deploy, allowing you to potentially guix system switch-generations if needed.

   Now create a test file on your web server:

   $ ssh root@tobacco-rebel.bnr.la "echo 'Hello World!' > /var/www/index.html"
   

   If you got this far you should be able to now access the new Nginx server, eg. For me http://tobacco-rebel.bnr.la/. Yay!

   Did this work for you? Please let me know.

Further work

From here, we have a working server with Nginx, but would need HTTPS and some content to call it a website. I've added some further resources below. Adding Certbot still currently requires a dance of first deploying a config with Certbot and without HTTPS so Nginx doesn't break, obtaining the certificate, then re-deploying with the HTTPS enabled in Nginx. Certbot is run via mcron, so see herd schedule mcron for the command to initially run to obtain the certificate.

Should you upload the website content through guix deploy or separately eg. rsync? That's up to you. Many examples use rsync, but the configuration for Berlin (below) uses static-web-site-configuration which builds a static website from a git repository.

How about adding a Mumble server, or any of the other services available in Guix? They're all only a few lines of config away!

Could the base image be shrunk further? 1.5GB is rather a lot to be moving around from many internet connections. This is not so much of an issue if your provider supports compressed QCOW2 images.

My host also has a Nova API and examples of using python-novaclient and docker-machine to provision VMs. I'd like to see whether I can upload Guix images and provision new machines this way.

Wrong turns I took along the way

Here's a few issues I wasted plenty of time on:

• My host doesn't support loading QCOW2 images. It accepted them, but they would always fail to boot.
• Before I added the signing-key config, guix deploy would fail with error: unauthorized public key because it didn't trust the substitute packages built on my deployment machine.
• I misunderstood the guix system image argument --volatile and ended up with a read-only server the regenerated it's SSH key on every reboot. Volatile in computing means non-persistent.
• Resizing storage for the base image was problematic on my hosting provider due to their auto-magic resizing feature that allocates storage, resizes the partition and the filesystem. Their support advised that selecting "BYO ISO" disables the partition/filesystem resizing and you can just do it manually. Before this I did experiment with creating extra-large efi-raw images eg. 20GB, but it wasn't really feasible.
• The efi-raw image creates a 40MB EFI partition at /dev/vda1 labelled "GNU-ESP", before the root partition at /dev/vda2. Deploying a config with the root partition listed as /dev/vda1 will mean the VM will no longer boot. After discussions with Mathieu (below), he is working on a single partition raw image type that will be available in the near future.

Thanks and further information

Mathieu Othacehe has a great blog post on this topic, and goes further into the web hosting configuration of the server.

Ludovic Courtès and roptat pointed out the configuration for Guix's server Berlin, which uses static-web-site-configuration.

Christine Lemmer-Webber initially encouraged me to look into guix deploy, wrote the Running Guix on a Linode Server cookbook entry and provided some examples from her own use.

Jakob L. Kreuze implemented much of guix deploy and described it in Towards Guix for DevOps and Managing Servers with GNU Guix: A Tutorial. Thanks Jakob and the other Guix contributors!