Automate Image signing with Tekton Chains to Private Registry¶

As we hear about more and more (software) Supply Chain attacks, securing our software supply chain becomes increasingly important. One of the ways to do this is to sign our container images.

In this post, we will look at how to automate the signing of container images using Tekton Chains⁶.

We use the Tekton Operator, Kyverno, SecretGen Controller, and Kaniko to automate the signing of the images. Then, verify the image's signature using the cosign tool.

Summary¶

In summay, we install the Tekton Operator¹, and Kyverno² on our Kubernetes cluster.

We set up Kubernetes secrets for signing with Cosign⁵ and uploading the image and the signature to the private registry.

We create a Tekton Pipeline that uses Kaniko⁴ to build the image and then Tekton Chains⁶ automatically signs the image using Cosign⁵.

We then verify the signature using the cosign⁵ tool.

Tools¶

Let's take a look at the tools we will use in this post:

Tekton Operator: the Tekton Operator is a Kubernetes Operator that manages the lifecycle of Tekton components and resources.
Tekton Chains: Tekton Chains is Tekton's Automated Supply Chain Security component, which can sign images and create attestations of our software supply chain.
Kyverno: manages policies in Kubernetes. We use Kyverno to automatically apply patches to the Tekton Chains Controller, to trust our private registry.
Kaniko: as we're in Kubernetes, we need a non-docker-in-docker way to build our images. Kaniko is a tool to build container images from a Dockerfile inside a container or Kubernetes cluster.
Cosign: we use Cosign to sign our container images. Cosign is a tool for signing and verifying container images. It is a part of the sigstore project.
Harbor: the post is about how to set this up for a private registry. My self-hosted private registry of choice is Harbor⁸

Steps¶

Each tool has good documentation on how to install it, so I will not go into detail about how to install them.

Instead, we look at what we need to configure to reach the desired outcome.

Install and configure all the mentioned tools
Create a Kubernetes secret for the private registry for Kaniko
Create a Kubernetes secret for the private registry for Tekton Chains
Create a Kyverno policy to add private registry secret to the Tekton Chains ServiceAccount
Create ConfigMap for the custom CA certificate
Create a Kyverno policy to trust the private registry
Create a Kubernetes secret for the Cosign signing key
Create a Tekton Pipeline that uses Kaniko to build the image and Tekton Chains to sign the image
Trigger the Pipeline with a PipelineRun
Verify the signature using the cosign tool

Configure the tools¶

I assume you've installed and run the tools on your Kubernetes cluster.

Some configuration is left to ensure they do what needs to be done.

The only tool for which we need to do something is Tekton Chains.

This is with the possible exception of the Tekton Operator, which you might want to configure to install all Tekton components.

Tekton Operator¶

For the Tekton Operator, you need to install the Tekton components. As we want to use Tekton Chains and Tekton Pipelines, I recommend using the all profile.

This installs all controllers in the same namespace, tekton-pipelines. Here's an example of how to install the Tekton components:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  profile: all
  targetNamespace: tekton-pipelines
  pruner:
    resources:
    - pipelinerun
    - taskrun
    keep: 100
    schedule: "0 8 * * *"

Tekton Chains¶

Tekton Chains also does attestations, but we need to choose between TaskRun and PipelineRun.

In this post, we use the PipelineRun approach.

We also want to use the latest format that is supported, which is slsa/v2alpha2.

Note

There is already a v2alpha3 version, but that requires versions of Tekton Chains and Tekton Pipelines that are not currently covered by the Tekton Operator.

I would prefer to use the OCI storage for the artifacts but not lose the ability to use the Tekton storage.

So we patch the chains-config ConfigMap to set the format and storage.

kubectl patch configmap chains-config -n tekton-pipelines -p='{"data":{"artifacts.pipelinerun.format": "slsa/v2alpha2"}}'
kubectl patch configmap chains-config -n tekton-pipelines -p='{"data":{"artifacts.pipelinerun.storage": "tekton,oci"}}'
kubectl patch configmap chains-config -n tekton-pipelines -p='{"data":{"artifacts.oci.storage": "tekton,oci"}}'
kubectl patch configmap chains-config -n tekton-pipelines -p='{"data":{"transparency.enabled": "true"}}'

The last patch disables transparency, a feature of Tekton Chains that stores the attestations in a separate storage. By default, this goes to a public registry, which I want to avoid in non-production environments (such as my Homelab).

Create a Private Registry secret for Kaniko¶

We need to create a Kubernetes secret for the private registry so Kaniko can push the image to it.

The secret must be in the same namespace as the Tekton Pipeline that uses Kaniko. I've chosen to use the build namespace for this.

The secret that works the best for the Kaniko task is a Docker config.json secret.

The config looks as follows: ' auth` is base64 encoded username:password.

~/.docker/config.json

{
  "auths": {
    "my-private-registry.domain.com": {
      "username": "username",
      "password": "password",
      "auth": "..." 
    }
  }
}

We encode it with base64:

base64 -w 0 ~/.docker/config.json

I've run into some quirks with the Kaniko task and the Docker config.json secret. So this is a secret that works for me with Harbor.

kaniko-harbor-secret.yaml

apiVersion: v1
data:
  config.json: # base64 encoded Docker config.json
kind: Secret
metadata:
  name: kaniko-harbor
  namespace: build
type: kubernetes.ioconfig.json

Apply the secret to the Kubernetes cluster:

kubectl apply -f kaniko-harbor-secret.yaml

In the Pipeline, we reference the secret in the spec of the Task, so we do not need to add it to the ServiceAccount of the TaskRun.

Create Private Registry secret for Tekton Chains¶

Tekton Chains needs to push the signature to the private registry.

For this, it needs the same secret, but in a different format unfortunately. It needs a docker-registry secret.

Here's an example of how to create the secret:

kubectl create secret docker-registry my-private-registry-secret \
  --docker-server=my-private-registry.domain.com \
  --docker-username=username \
  --docker-password=password \
  --namespace=tekton-pipelines

We use Kyverno next to add this secret to the ServiceAccount of the Tekton Chains Controller.

Kyverno policy to add private registry secret to the Tekton Chains ServiceAccount¶

We use Kyverno to automatically apply patches to the Tekton Chains Controller, to trust our private registry.

Like most Tekton controllers, Tekton Chains automatically picks up secrets from the ServiceAccount it runs under.

The ServiceAccount is managed by the Tekton Operator, so we do not want to manually edit the ServiceAccount.

Hence, we're using Kyverno to keep it up to date.

Here's an example of how to create the Kyverno ClusterPolicy:

policy-add-secrets-to-serviceaccount.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-secrets-to-serviceaccount
spec:
  background: true
  generateExisting: true
  rules:
    - name: add-imagepullsecret-to-serviceaccount
      match:
        resources:
          kinds:
            - ServiceAccount
          names:
            - tekton-chains-controller
          namespaces:
            - tekton-pipelines
      mutate:
        patchStrategicMerge:
          metadata:
            name: tekton-chains-controller
          imagePullSecrets:
            - name: my-private-registry-secret
    - name: add-secret-to-serviceaccount
      match:
        resources:
          kinds:
            - ServiceAccount
          names:
            - tekton-chains-controller
          namespaces:
            - tekton-pipelines
      mutate:
        patchStrategicMerge:
          secrets:
            - name: my-private-registry-secret

And apply the policy to the Kubernetes cluster:

kubectl apply -f policy-add-secrets-to-serviceaccount.yaml

Create ConfigMap for the custom CA certificate¶

It is recommend to use TLS for the private registry.

This means that the private registry uses a certificate, and in the case of a private registry, it is likely a self-signed certificate.

We need to make the the Tekton Chains Controller trust the CA certificate that signed the registry certificate.

Let's create a ConfigMap for the custom CA certificate:

custom-ca-certificate-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-registry-cert
  namespace: tekton-pipelines
data:
  cert: |-
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

And apply the ConfigMap to the Kubernetes cluster:

kubectl apply -f custom-ca-certificate-configmap.yaml

Kyverno policy to trust the private registry¶

Unfortunately, the Tekton Chains Controller does not have a way to configure a custom CA certificate by its own configuration.

So we turn to Kyverno again to patch the Tekton Chains Controller to mount our custom CA certificate ConfigMap.

This policy matches the Tekton Chains Controller and adds the ConfigMap to the Pod via the volumes and volumeMounts properties.

policy-trust-registry.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-configmap-volume-to-specific-deployment
spec:
  background: true
  generateExisting: true
  rules:
    - name: inject-configmap-as-volume-to-tekton-chains-controller
      match:
        resources:
          kinds:
            - Deployment
          names:
            - tekton-chains-controller
          namespaces:
            - tekton-pipelines
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                volumes:
                  - name: reg-certs
                    configMap:
                      name: config-registry-cert
                containers:
                  - (name): "tekton-chains-controller"
                    volumeMounts:
                      - name: reg-certs
                        mountPath: /etc/ssl/certs

And apply the policy to the Kubernetes cluster:

kubectl apply -f policy-trust-registry.yaml

This ensures that the Tekton Chains Controller trusts the certificate of the private registry.

Cosign signing key¶

We need to create a Kubernetes secret for the Cosign signing key.

Cosign has a built-in command to generate a key pair.

It supports storing it in a file or in a Kubernetes secret directly. We use the latter.

cosign generate-key-pair k8s://tekton-pipelines/signing-secrets

To explain, k8s:// is the prefix for the Kubernetes secret store, tekton-pipelines is the namespace, and signing-secrets is the secret's name.

This is what Tekton Chains expects to exist in the Controller's namespace.

The public file (cosign.pub) that is then generated can be used to verify the signature.

Tekton Pipeline¶

We create a Tekton Pipeline that uses Kaniko to build the image and Tekton Chains to sign the image.

We do that as follows:

Install the community Task git-clone¹¹ from the Tekton Catalog⁹
Install the community Task kaniko⁷ from the Tekton Catalog⁹
Compose a Pipeline that uses the git-clone and kaniko Tasks

The git-clone Task clones the source code from a Git repository.

The kaniko Task builds the image and pushes it to the private registry. In addition, the kaniko exports the image's digest and URL as an results output.

Info

Tekton Chains expects the results to have a specific name.

The Kaniko Task is pre-configured with these outputs. By re-using the Task, we don't need to do anything for Tekton Chains to sign the image!

Example format:

results:
- name: IMAGE_DIGEST
  description: Digest of the image just built.
- name: IMAGE_URL
  description: URL of the image just built.

Tekton Chains reads the results output and signs the image using the Cosign secret. It then uploads the signature to the private registry.

The structure of the Pipeline looks as follows:

Define the Parameters to make the Pipeline more flexible
Define the Workspaces to cover what the Tasks need (store the git checkout, Docker config.json)
Define the Tasks to do the work (clone the source code, build the image)

Let's look at a working example:

Example Pipeline

pipeline.yaml

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: signed-image-builds
  namespace: build
spec:
  description: |
    This Pipeline clones a git repo, then echoes the README file to the stdout.
  params:
    - name: repo-url
      type: string
      description: The git repo URL to clone from.
    - name: gitrevision
      description: git revision to checkout
    - name: dockerfile
      description: dockerfile to use for the kaniko task
      type: string
      default: Dockerfile
    - name: registry
      description: registry to push to
      type: string
      default: ghcr.io
    - name: repo
      description: repo to push to
      type: string
      default: joostvdg/gitstafette/server
    - name: tag
      description: tag to push to
      type: string
      default: latest
  workspaces:
    - name: checkout
      description: |
        This workspace contains the cloned repo files, so they can be read by the
        next task.
    - name: registry-credentials
      description: |
        This workspace contains the registry credentials, so they can be read by the
        next task.
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: checkout
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: $(params.gitrevision)
    - name: server-image-build-and-push
      runAfter: ["git-clone"]
      taskRef:
        name: kaniko
      workspaces:
      - name: source
        workspace: checkout
      - name: dockerconfig
        workspace: registry-credentials
      params:
      - name: IMAGE
        value: "$(params.registry)/$(params.repo):$(params.tag)"
      - name: DOCKERFILE
        value: $(params.dockerfile)
      - name: EXTRA_ARGS
        value: [--skip-tls-verify-registry=harbor.home.lab]

And apply the Pipeline to the Kubernetes cluster:

kubectl apply -f pipeline.yaml

Warning

To avoid delving into the depths of Kaniko configuration (which is out of scope), I've set the EXTRA_ARGS to skip TLS verification.

--skip-tls-verify-registry=harbor.home.lab

Trigger the Pipeline with a PipelineRun¶

We can now trigger the Pipeline with a PipelineRun.

You can create a PipelineRun as a Kubernetes resource or use the Tekton CLI.

Here's an example of how to create a PipelineRun:

Tekton PipelineRun

pipelinerun.yaml

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: gitssigned-image-builds-run-
  namespace: build
spec:
  pipelineRef:
    name: signed-image-builds
  workspaces:
  - name: checkout
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  - name: registry-credentials
    secret:
      secretName: kaniko-harbor
  params:
  - name: repo-url
    value: https://github.com/joostvdg/gitstafette.git
  - name: gitrevision
    value: main
  - name: registry
    value: harbor.home.lab
  - name: repo
    value: gsf/server

Some things to note:

We use the kaniko-harbor secret for the registry-credentials workspace
We specify a VolumeClaimTemplate for the checkout workspace
We don't use a name, but a generateName for the PipelineRun

Because a PipelineRun needs to be unique, we use generateName to generate a unique name for the PipelineRun.

And apply the PipelineRun to the Kubernetes cluster via a kubectl create, because we use generateName:

kubectl create -f pipelinerun.yaml

Verify PipelineRun Was Signed¶

To verify the signing happened, we can investigate the PipelineRun. Find your latest TaskRun:

kubectl get PipelineRun -n build

Store it in a variable:

PR=$(kubectl get PipelineRun -n build -l tekton.dev/pipeline=signed-image-builds -o jsonpath='{.items[0].metadata.name}')

And then retrieve the Annotations:

kubectl get PipelineRun $PR -n build -ojson | jq .metadata.annotations

Which, among other things, should contain the following annotation.

{
  "chains.tekton.dev/signed": "true"
}

Verify the signature using Cosign¶

We can verify the signature using the cosign tool.

We can do that with the cosign verify command:

cosign verify --key cosign.pub $REGISTRY/$MY_IMAGE

Because we use a private registry, we might encounter a certificate issue, which can be resolved by adding the flag --allow-insecure-registry=true.

In addition, I disabled Tekton Chains' transparency feature, so the attestations are not stored in a public registry. This means we must also add the flag --insecure-ignore-tlog.

Which results in the following command:

cosign verify \
  --key cosign.pub \
  --allow-insecure-registry=true \
  --insecure-ignore-tlog \
  $REGISTRY/$MY_IMAGE

Tip

Pipe the output to jq to get a more readable output.

cosign verify \
  --key cosign.pub \
  --allow-insecure-registry=true \
  --insecure-ignore-tlog \
  $REGISTRY/$MY_IMAGE | jq .

An example output looks like this:

First, the Bash output:

Verification for harbor.home.lab/gsf/server@sha256:1a6dab8ea26ca9e920dc638a2325d3e6fc3b4b9d3bf61da7b1f152cb1ad276c9 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

And this is the JSON payload.

[
  {
    "critical": {
      "identity": {
        "docker-reference": "harbor.home.lab/gsf/server"
      },
      "image": {
        "docker-manifest-digest": "sha256:1a6dab8ea26ca9e920dc638a2325d3e6fc3b4b9d3bf61da7b1f152cb1ad276c9"
      },
      "type": "cosign container image signature"
    },
    "optional": null
  }
]

Conclusion¶

There is more ground to cover with Tekton Chains, Pipeline Signing, and Attestations.

But this was enough to cover the basics of automatically getting your images signed.