Build container images in OpenShift using Podman as a GitLab runner

As you start your container journey, it is mandatory that you have image build tools. There are various tools available; the most common ones are Docker, Buildah, Podman, and Kaniko. You need to have image build tools to pull down images from external registry, customize the images, and build new images with libraries or applications. These images then published to container registries and further used in Kubernetes platforms such as Red Hat OpenShift, EKS, AKS, etc.

In this article, we'll be mainly talking about OpenShift, but the contents can be used for other platforms as well with little to no modifications.

In most enterprises, especially where security, compliance, and operational consistency are paramount, image build processes are strictly governed due to control risks. I have seen organizations having a separate build infrastructure just to manage the image build process. On the other side, few organizations allow application teams to manage build infrastructure. In both the cases, the infrastructure needs to have any of the above listed tools available. It is very common that such tools are mostly installed on virtual machines (VMs) reserved just for container build processes. Along with such infrastructure, it also comes with the overhead of maintenance, security, capacity management, and more.

Why we are talking about just container image builds? It is because most of the other pipeline stages such as application compilation or build, packaging, and deployment can be handled within containers on OpenShift. Only image build process needs specific configurations to make it work.

If you are using GitLab as CI/CD tool, you have to manage various GitLab runners (meaning various virtual machines) with approved container image build tools. If your organization has OpenShift, then Podman would be the tool of choice. In such scenarios, if we move these runners on OpenShift, wouldn't that be cost beneficial if we don't have to manage and maintain VMs just for image builds anymore?

Let's look at what is required to run Podman on OpenShift that can enable the image build process in OpenShift itself.

Requirements to run Podman on OpenShift

Note

In order to perform some of the following steps, you should have admin access in OpenShift.

Create new Security Context Constraint (SCC) in OpenShift.

OpenShift 4.11 introduced restricted-v2 SCC; however, prior to that the default SCC was restricted SCC. Create a new custom SCC based on the restricted SCC by adding below SETUID and SETGID capabilities. Below is the complete SCC that should be created:

apiVersion: security.openshift.io/v1
metadata:
  name: podman-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities:
- SETUID
- SETGID
defaultAddCapabilities: null
fsGroup:
  type: MustRunAs
groups: []
kind: SecurityContextConstraints
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret

Create entrypoint.sh to use in the Containerfile:

#!/usr/bin/env bash

if [ ! -d "${HOME}" ]
then
  mkdir -p "${HOME}"
fi

mkdir -p ${HOME}/.config/containers
# Below line is not needed for ocp 4.15+ clusters as fuse-overlayfs will be made available
(echo '[storage]';echo 'driver = "vfs"') > ${HOME}/.config/containers/storage.conf

if ! whoami &> /dev/null
then
  if [ -w /etc/passwd ]
  then
    echo "${USER_NAME:-user}:x:$(id -u):0:${USER_NAME:-user} user:${HOME}:/bin/bash" >> /etc/passwd
    echo "${USER_NAME:-user}:x:$(id -u):" >> /etc/group
  fi
fi
USER=$(whoami)
START_ID=$(( $(id -u)+1 ))
echo "${USER}:${START_ID}:2147483646" > /etc/subuid
echo "${USER}:${START_ID}:2147483646" > /etc/subgid

exec "$@"

Create the container image with Podman and additional configurations to run Podman in rootless mode.

If you have ever tried to run the Podman locally or on a VM in the rootless mode, you should be familiar with some of the configurations below in the Containerfile (Dockerfile). The pod in OpenShift comes up with a default user with a random user ID (UID). To set up the required user permissions to run Podman commands, you need to make the following configurations. Build the image with the below Containerfile and publish it to image repository. The below Containerfile also installs other tools along with Podman but it is not required:

FROM registry.access.redhat.com/ubi9/ubi-minimal

ARG USER_HOME_DIR="/home/user"

ENV HOME=${USER_HOME_DIR}
ENV BUILDAH_ISOLATION=chroot

RUN microdnf --disableplugin=subscription-manager install -y openssl git tar which shadow-utils bash zsh wget jq podman buildah skopeo; \
    microdnf update -y ; \
    microdnf clean all ; \
    mkdir -p ${USER_HOME_DIR} ; \
    chgrp -R 0 /home ; \
    #
    # Setup for root-less podman
    #
    setcap cap_setuid+ep /usr/bin/newuidmap ; \
    setcap cap_setgid+ep /usr/bin/newgidmap ; \
    touch /etc/subgid /etc/subuid ; \
    chmod -R g=u /etc/passwd /etc/group /etc/subuid /etc/subgid /home
WORKDIR ${HOME}
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
ENTRYPOINT [ "./entrypoint.sh" ]

Create a namespace in OpenShift for running the GitLab runner instance, e.g., podman-runner:
```
       oc create ns podman-runner
```

Create service account in the above namespace and associate the SCC with the service account:

       oc -n podman-runner create sa podman-sa
       oc -n podman-runner adm policy add-scc-to-user podman-scc -z podman-sa

Create imagePullSecret to access the image built in step 3 from external registry. This imagePullSecret is referenced in the subsequent step for deployment resource:

kind: Secret
apiVersion: v1
metadata:
  name: gitlab-secret
  namespace: podman-runner
data:
  .dockerconfigjson: eyJhdXRocyI6eIiwicGFzc3dvcmQiOiJAbGVhcm5HaXRsYWIwIiwiYXV0aCI6IllURnFhMkZ1T2tCc1pXRnlia2RwZEd4aFlqQT0iLCJlbWFpbCI6IiJ9fX0=
type: kubernetes.io/dockerconfigjson

Create an example deployment in the namespace to test the Podman image and the functionality:

# Created podman-runner namespace 
kind: Deployment
apiVersion: apps/v1
metadata:
  name: podman-example
  namespace: podman-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: name
  template:
    metadata:
      labels:
        app: name
    spec:
      containers:
        - name: container
          # Below image is built at step 3 and published to gitlab container repository
          image: 'registry.gitlab.com/myrepo/podman-runner:1.0.0'
          # Command with below added so that container doesn't go away by the time we complete test
          command:
            - /bin/sh
            - '-c'
            - ./entrypoint.sh sleep 3600
          imagePullPolicy: Always
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
            add:
            - "SETUID"
            - "SETGID"
      serviceAccountName: podman-sa
      restartPolicy: Always
      imagePullSecrets:
        - name: gitlab-secret

Validate that Podman commands can be executed in the container terminal in OpenShift.

Make sure configurations done in the entrypoint script (step 2) for OpenShift random ID are in effect correctly:

# Below commands are executed in the terminal window of container for validation
# Note that random uid 1000850000 has necessary configurations for this container
sh-5.1$ podman version
Client:       Podman Engine
Version:      4.9.4-rhel
API Version:  4.9.4-rhel
Go Version:   go1.21.7 (Red Hat 1.21.7-1.el9)
Built:        Mon Apr 15 12:33:08 2024
OS/Arch:      linux/amd64
sh-5.1$ cat /etc/sub{u,g}id ; cat /etc/passwd | grep user ; cat /etc/group | grep user ; getcap /usr/bin/newuidmap ; getcap /usr/bin/newgidmap ; podman info | grep -i driver
user:1000850001:2147483646
user:1000850001:2147483646
user:x:1000850000:0:user user:/home/user:/bin/bash
users:x:100:
user:x:1000850000:
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
  logDriver: k8s-file
  graphDriverName: vfs

Set up and configure GitLab runner

Now that the Podman container is validated, let's set up a GitLab runner to use the image that was built in step 3. The steps below provide high-level details on how to set up and configure the GitLab runner.

Install the GitLab Runner operator labeled Certified from OperatorHub in the podman-runner namespace, as shown in Figure 1.

Figure 1: Gitlab runner operator.
In the GitLab project that you intend to build, create a Kubernetes runner and get the runner authentication token.

Create Kubernetes secret in the podman-runner namespace with the runner authentication token:

kind: Secret
apiVersion: v1
metadata:
  name: gitlab-runner-secret
  namespace: podman-runner
data:
  runner-registration-token: Z2xydC1CY1d4UjNvdw==

Create the ConfigMap with config.toml key for GitLab runner configurations:

kind: ConfigMap
apiVersion: v1
metadata:
  name: podman-gitlab-runner-custom-toml
  namespace: podman-runner
data:
  config.toml: |
    [[runners]]
      [runners.kubernetes]
      # Below overwrite configurations are required for podman runner
        pod_annotations_overwrite_allowed = "openshift.io/required-scc=.*"
        service_account_overwrite_allowed = ".*"
        pull_policy = "always"
        # build container
        cpu_limit = "500m"
        memory_limit = "1024Mi"
        # service containers
        service_cpu_limit = "100m"
        service_memory_limit = "200Mi"
        # helper container
        helper_cpu_limit = "100m"
        helper_memory_limit = "200Mi"

Create a GitLab runner instance. Use the ConfigMap and secret created above in the runner instance:

apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: example-runner
  namespace: podman-runner
spec:
  config: podman-gitlab-runner-custom-toml
  gitlabUrl: 'https://gitlab.com'
  imagePullPolicy: Always
  tags: 'podman-runner'
  token: gitlab-runner-secret

Validate that the GitLab operator (runner controller manager) and example-runner pods are running in the namespace. This ensures that all above steps are followed correctly.
Make sure that you have a runner associated with the GitLab project. See Figure 2 indicating that a runner is successfully registered in the GitLab project. Notice the tag used is podman-runner.

Gitlab runner displayed inside Gitlab. — Figure 2: Runner is successfully registered in Gitlab.

Create a Containerfile to validate the image build process using GitLab Pipeline. The example uses a Containerfile to install few required packages and copy source code and few additional commands. The goal is to showcase that we can build a container image with Podman GitLab runner in the same way as it can be done locally or a virtual machine:
```
FROM explore0.jfrog.io/base-repo/redhat.io/ubi8/python-38:1-94

LABEL MAINTAINER developer_name

WORKDIR /app

COPY app.py requirements.txt /app/

RUN pip install --trusted-host pypi.python.org -r requirements.txt

EXPOSE 9000

ENV NAME Hello from Dockerfile

ENV BGCOLOR blue

CMD ["python","app.py"]
```

Create a .gitlab-ci.yml file that uses runner tag podman-runner. Below is an example YML file for reference.

Note that in the build_image stage, KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: podman-sa and KUBERNETES_POD_ANNOTATIONS_1: "openshift.io/required-scc=podman-scc" is required.

Additionally, /home/user/entrypoint.sh need to be executed in the image_build stage because GitLab runner overrides the entrypoint defined in the image:

image: registry.gitlab.com/o1266/podman-runner/podman-runner-cmd:3.0.0

default:
  tags:
   - podman-runner
  
stages:          # List of stages for jobs, and their order of execution
  - build_image
  - other_stage


build_image_job:
  variables:
# Below 2 lines are needed to make sure the runner uses appropriate service     # account and scc.
    KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: podman-sa
    KUBERNETES_POD_ANNOTATIONS_1: "openshift.io/required-scc=podman-scc"
  stage: build_image
  script:
    - /home/user/entrypoint.sh
    - podman build -t testimage:1.0 -f Containerfile .

other_demo_job:
  stage: other_stage
  script:
  - echo "This is not build stage"

Run GitLab pipeline. It should run build_image and other_stage from .gitlab-ci.yml.

Every stage in the GitLab pipeline spins up a new container. For the build_image stage, the container comes up with SCC podman-scc and service account podman-sa. This stage will build the container image. However, the other_stage container comes up with restricted-v2 SCC and default service account.

Sample output:

Running with gitlab-runner 17.2.1 (9882d9c7)
  on example-runner-runner-55dcf49df4-w5rw7 Txbc_BqZq, system ID: r_sK6jnT16Erqv
Preparing the "kubernetes" executor
00:00
"ServiceAccount" overwritten with "podman-sa"
"PodAnnotations" "openshift.io/required-scc" overwritten with "podman-scc"
Using Kubernetes namespace: podman-runner
Using Kubernetes executor with image registry.gitlab.com/o1266/podman-runner/podman-runner-cmd:3.0.0 ...
Using attach strategy to execute scripts...
Preparing environment
00:08
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod podman-runner/runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z to be running, status is Pending
Waiting for pod podman-runner/runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-txbcbqzq-project-58314782-concurrent-0-9fisjq7z via example-runner-runner-55dcf49df4-w5rw7...
Getting source from Git repository
00:04
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/o1266/podman-runner/.git/
Created fresh repository.
Checking out 1b45e11f as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
06:19
$ /home/user/entrypoint.sh
$ podman build -t testimage:1.0 -f Containerfile .
STEP 1/9: FROM registry.access.redhat.com/ubi9/python-312:latest
Trying to pull registry.access.redhat.com/ubi9/python-312:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:95a3c7ae41d820ce47bbbff3848e46c36e6402431600de4457bfdf091901a772
Copying blob sha256:db22e630b1c7cf081461536c489254a8d1b39ceda32f8f3025314f032860d984
Copying blob sha256:cc296d75b61273dcb0db7527435a4c3bd03f7723d89a94d446d3d52849970460
Copying blob sha256:9f2fbf79f82326e925352def0dbdccc800cb9da2fe0125c4d1c33a9cbfc1b629
Copying config sha256:ca17740c85b9615346d509e099062b761a29692040b143af0689dc4f5bd11005
Writing manifest to image destination
Storing signatures
STEP 2/9: LABEL MAINTAINER developer_name
--> 9c1e5c6f60fd
STEP 3/9: WORKDIR /app
--> e4e28c3cca48
STEP 4/9: COPY app.py requirements.txt /app/
--> 69fbbe770a66
STEP 5/9: RUN pip install --trusted-host pypi.python.org -r requirements.txt
Collecting Flask (from -r requirements.txt (line 1))
  Obtaining dependency information for Flask from https://files.pythonhosted.org/packages/61/80/ffe1da13ad9300f87c93af113edd0638c75138c42a0994becfacac078c06/flask-3.0.3-py3-none-any.whl.metadata
  Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Collecting Redis (from -r requirements.txt (line 2))
  Obtaining dependency information for Redis from ........
  .......
  .......
--> 5960727b5065
STEP 6/9: EXPOSE 9000
--> ce4704e8ccc5
STEP 7/9: ENV NAME Hello from Dockerfile
--> f1a2ba32abd7
STEP 8/9: ENV BGCOLOR blue
--> 1e5d528d242d
STEP 9/9: CMD ["python","app.py"]
COMMIT testimage:1.0
--> 81a670dd615e
Successfully tagged localhost/testimage:1.0
81a670dd615e5266143cb19d271c1685cc435b05e0af62511575603f500ab60a
Cleaning up project directory and file based variables
00:00
Job succeeded

Note

The configurations are tested only on OpenShift 4.12.

Conclusion

Running Podman in rootless mode, as demonstrated in this article, offers numerous advantages. It eliminates the need for standalone build machines for runners or agents, enhances cost efficiency, enables ephemeral runners, and optimizes the use of the OpenShift platform for image building while leveraging native Kubernetes features.

While the article primarily focuses on setting up a GitLab runner, the initial steps for enabling Podman containers can also be applied to other CI tools like Jenkins, GitHub, and more. It also opens up possibilities for other integrations and automations.

Limitations

Nested containers are not supported up to OpenShift 4.15. OpenShift 4.16+ may have support for fully nested containers and users can execute commands such as podman run, podman compose, and etc.

Credits: The configurations related to Podman are provided by Charro Gruver. Charro's contribution significantly helped with this implementation.

Build container images in OpenShift using Podman as a GitLab runner

Share:

Requirements to run Podman on OpenShift

Set up and configure GitLab runner

Conclusion

Limitations

Products

Build

Quicklinks

Communicate

RED HAT DEVELOPER

Red Hat legal and privacy links

Red Hat legal and privacy links

Report a website issue