GitLab CI/CD on OpenShift Online Kubernetes… or not

In short, it appears that you currently can't use OpenShift Online as a Kubernetes cluster for gitlab.com. I'd be happy to find out I'm wrong though.

I don't want to learn how to administer a Kubernetes cluster right now. I want less less infrastracture to manage, not more. With that in mind, Openshift Online seems like the ideal Kubernetes backed to run my GitLab continuous integration/deployment services. I already pay for and use the platform, it's run by a company I trust and respect and provides a great web interface. Let's set it up!

  1. Given an account on gitlab.com, create a new project, for example by forking minimal-ruby-app.

  2. Given either a "Starter" or "Pro" account on OpenShift Online, create a project called gitlab-managed-apps, create a service account for GitLab to use, and copy the service token:

    $ oc login
    $ oc new-project gitlab-managed-apps
    $ oc create serviceaccount gitlab-robot
    $ oc policy add-role-to-user admin system:serviceaccount:gitlab-managed-apps:gitlab-robot
    $ oc serviceaccounts get-token gitlab-robot
    

    If you log in to the OpenShift web console and visit the gitlab-managed-apps project, you'll find our new service account under "Resources, Membership".

    Alternately you can create the project and the service account via the web console, but I don't believe it's possible to view the service account token without the oc command line.

  3. Returning to GitLab, select "Settings, CI / CD" and expand the "Runners settings". Click "Install Runner on Kubernetes". Click "Add Kubernetes cluster", then click "Add an existing Kubernetes cluster". Set the cluster name, API URL and token. For me these are something like:

    GitLab Kubernetes setup

    Note that the "API URL" is the base URL from your OpenShift Online web console, excluding the bit after the "/" and the token is the one we exported above.

  4. So far so good. On the following screen, under "Applications" click the "Install" button next to "Helm Tiller". If it's not clickable, try refreshing the page. If the service account is set up right, the indicator will spin for a bit, then display "Installing" for about a minute, then issue an error:

    Something went wrong while installing Helm Tiller
    /bin/sh: can't create /etc/apk/repositories: Permission denied
    

    If you're fast, you can briefly see a pod being spun up and deleted on the OpenShift web console.

Unfortunately I wasn't able to overcome this issue, that's as far as I got. There's no clear indication of what actual problem is and, as far as I know, no way to dive in and find out what's going on. I suspect the issue may be related to OpenShift Online's default policy of not allowing containers to run as root, but I can't say for sure.

The disappointing thing is that back in 2016 and 2017, there did seem to be some sense of partnership between GitLab and OpenShift. There are a number of official GitLab/OpenShift videos from around this time, demonstrating the manual setup of GitLab Runner on OpenShift (pre-GitLab Kubernetes integration). Unfortunately, there don't seem to be any official and stable GitLab Runner templates for OpenShift. This official-sounding one is marked EXPERIMENTAL and this unofficial one template I came across advises that it is not compatible with OpenShift Online (only self-hosted OpenShift). The Review Apps mention down the bottom that docs for OpenShift/kubernetes are soon to be added.

The approach I have had some success with is using the .gitlab-ci.yml template that GitLab provide for OpenShift via the "Set up CI/CD" button. This uses gitlab.com's shared runners, but deploys to OpenShift. This review-apps-openshift looks promising too. But that's another blog post.

An aside: How is this meant to work?

In contrast, the GitLab CI/CD and Google Kubernetes Engine (GKE) integration appears to be getting significantly more promotion and quite deep-running integration from the GitLab team. While I don't use or recommend Google services to any of my clients due to privacy concerns and lock-in from proprietary network services, I gave this a brief trial to see how the GitLab/Kubernetes integration is meant to work.

GitLab provide a custom setup process and documentation for GKE. This mostly avoids the need to use the GKE web interface at all, aside from creating an account, creating a project and enabling API permissions. That's a good thing, since the GKE web interface seemed to be very buggy. Once Kubernetes was configured, GitLab did the rest; launching a cluster and prompting me to install Helm Tiller, GitLab Runner and create a wildcard DNS entry. Using this setup, I was able to enable the Kubernetes GitLab Runner and use Auto Deploy to successfully deploy "minimal-ruby-app" to the custom domain

The overall user experience with GKE wasn't actually significantly different to that of using the 80 line OpenShift .gitlab-ci.yml mentioned above; just that instead of using gitlab.com shared runners, the build/test/deploy work is done from on Kubernetes. The major downside of the Kubernetes approach though (either GKE or OpenShift if it worked), is that Auto Deploy magic is actually 800 lines of shell script and YAML which, sooner or later, will need troubleshooting. So while it makes for a slick webcast, I'm not sure this is a path I want to take after all.

blogroll

FSF member since 2006 Become a Conservancy Supporter! EFF member

social

Powered by Pelican and Python. Theme by Smashing Magazine.