Evaluate policy compliance in CI
Adding Policy Evaluation to your continuous integration pipelines helps you detect and prevent cases where code changes would cause policy compliance to become worse compared to your baseline.
The recommended strategy for Policy Evaluation in a CI setting involves evaluating a local image and comparing the results to a baseline. If the policy compliance for the local image is worse than the specified baseline, the CI run fails with an error. If policy compliance is better or unchanged, the CI run succeeds.
This comparison is relative, meaning that it's only concerned with whether your CI image is better or worse than your baseline. It's not an absolute check to pass or fail all policies. By measuring relative to a baseline that you define, you can quickly see if a change has a positive or negative impact on policy compliance.
How it works
When you do Policy Evaluation in CI, you run a local policy evaluation on the image you build in your CI pipeline. To run a local evaluation, the image that you evaluate must exist in the image store where your CI workflow is being run. Either build or pull the image, and then run the evaluation.
To run policy evaluation and trigger failure if compliance for your local image
is worse than your comparison baseline, you need to specify the image version
to use as a baseline. You can hard-code a specific image reference, but a
better solution is to use
environments
to automatically infer the image version from an environment. The example that
follows uses environments to compare the CI image with the image in the
production
environment.
Example
The following example on how to run policy evaluation in CI uses the
Docker
Scout GitHub Action to
execute the compare
command on an image built in CI. The compare command has
a to-env
input, which will run the comparison against an environment called
production
. The exit-on
input is set to policy
, meaning that the
comparison fails only if policy compliance has worsened.
This example doesn't assume that you're using Docker Hub as your container
registry. As a result, this workflow uses the docker/login-action
twice:
- Once for authenticating to your container registry.
- Once more for authenticating to Docker to pull the analysis results of your
production
image.
If you use Docker Hub as your container registry, you only need to authenticate once.
メモ
Due to a limitation in the Docker Engine, loading multi-platform images or images with attestations to the image store isn't supported.
For the policy evaluation to work, you must load the image to the local image store of the runner. Ensure that you're building a single-platform image without attestations, and that you're loading the build results. Otherwise, the policy evaluation fails.
Also note the pull-requests: write
permission for the job. The Docker Scout
GitHub Action adds a pull request comment with the evaluation results by
default, which requires this permission. For details, see
Pull Request Comments.
name: Docker
on:
push:
tags: ["*"]
branches:
- "main"
pull_request:
branches: ["**"]
env:
REGISTRY: docker.io
IMAGE_NAME: <IMAGE_NAME>
DOCKER_ORG: <ORG>
jobs:
build:
permissions:
pull-requests: write
runs-on: ubuntu-latest
steps:
- name: Log into registry ${{ env.REGISTRY }}
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Setup Docker buildx
uses: docker/setup-buildx-action@v3
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.IMAGE_NAME }}
- name: Build image
id: build-and-push
uses: docker/build-push-action@v4
with:
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
sbom: ${{ github.event_name != 'pull_request' }}
provenance: ${{ github.event_name != 'pull_request' }}
push: ${{ github.event_name != 'pull_request' }}
load: ${{ github.event_name == 'pull_request' }}
- name: Authenticate with Docker
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PAT }}
- name: Compare
if: ${{ github.event_name == 'pull_request' }}
uses: docker/scout-action@v1
with:
command: compare
image: ${{ steps.meta.outputs.tags }}
to-env: production
platform: "linux/amd64"
ignore-unchanged: true
only-severities: critical,high
organization: ${{ env.DOCKER_ORG }}
exit-on: policy
The following screenshot shows what the GitHub PR comment looks like when a policy evaluation check fails because policy has become worse in the PR image compared to baseline.
This example has demonstrated how to run policy evaluation in CI with GitHub Actions. Docker Scout also supports other CI platforms. For more information, see Docker Scout CI integrations.