Container Building and Hardening: A Production-Ready Guide

Introduction

Building container images is easy. Building them securely, efficiently, and reliably at scale is not. Modern container workflows demand tools that work without privileged access, produce minimal attack surfaces, and integrate seamlessly with CI/CD pipelines.

This guide covers two critical concerns:

How to build — comparing modern container builders and when to choose each
How to harden — security techniques from base image to runtime

Part 1: Container Building Approaches

Tool Comparison Matrix

Tool	Daemonless	Rootless	K8s Native	Best For
Docker BuildKit	No	Limited	No	Local dev, fastest builds
Buildah	Yes	Yes	No	Rootless CI, podman ecosystems
Kaniko	Yes	Yes	Yes	K8s CI/CD pipelines
Podman Build	Yes	Yes	No	Local dev, RHEL ecosystems
Cloud Native Buildpacks	Yes	Yes	Yes	Heroku-style auto-detection

Docker BuildKit (Recommended Default)

BuildKit is now the standard builder in Docker 23+. It delivers 3x faster builds through parallel execution and advanced caching.

Key Features

Parallel build execution — DAG-based dependency resolution
Advanced caching — Inline cache, remote registry cache
Multi-platform builds — Cross-compile for arm64/amd64
Secrets mounting — --secret without leaving traces in images
SSH forwarding — --ssh for private repositories

Enable BuildKit

# Docker 23+ uses BuildKit by default
# For older versions:
export DOCKER_BUILDKIT=1

# Or use buildx (recommended)
docker buildx install

Multi-Platform Build with Remote Cache

# Create multi-platform builder
docker buildx create --name multiarch --driver docker-container --use

# Build and push with caching
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --cache-from type=registry,ref=registry.io/myapp:buildcache \
  --cache-to type=registry,ref=registry.io/myapp:buildcache,mode=max \
  --tag registry.io/myapp:v1.0.0 \
  --push \
  .

Secret Mounting Pattern

# Dockerfile
FROM python:3.11-slim

# Mount secret during build (not saved in image)
RUN --mount=type=secret,id=pip_token \
    pip config set global.extra-index-url $(cat /run/secrets/pip_token)

COPY requirements.txt .
RUN pip install -r requirements.txt

# Build with secret
docker buildx build \
  --secret id=pip_token,src=./pip_token.txt \
  -t myapp:latest .

Buildah: Rootless, Daemonless Building

Buildah builds OCI-compatible images without a daemon, ideal for rootless CI environments.

Installation

# Fedora/RHEL
sudo dnf install buildah

# Ubuntu/Debian
sudo apt-get install buildah

# macOS
brew install buildah

Key Commands

# Build from Dockerfile
buildah bud -t myapp:latest .

# Build with layers (faster rebuilds)
buildah bud --layers -t myapp:latest .

# Build as non-root user
buildah bud --userns=host -t myapp:latest .

# Inspect without running
buildah inspect myapp:latest

Buildah vs Docker

# Docker requires daemon
docker build -t myapp:latest .

# Buildah is daemonless
buildah bud -t myapp:latest .

# Docker creates intermediate containers
# Buildah creates intermediate images only (lighter)

Kaniko: Kubernetes-Native Building

Kaniko executes Dockerfile builds inside Kubernetes without requiring the Docker daemon or privileged mode.

⚠️ Note: Kaniko was archived by Google in June 2025. Consider BuildKit for new projects. Chainguard maintains a fork.

When to Use Kaniko

CI/CD pipelines inside Kubernetes clusters
Environments where Docker daemon access is prohibited
Security-restricted build environments

Kubernetes Job Example

apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-build
spec:
  template:
    spec:
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:latest
        args:
          - "--dockerfile=Dockerfile"
          - "--context=git://github.com/org/repo.git#refs/heads/main"
          - "--destination=registry.io/myapp:v1.0.0"
          - "--cache=true"
          - "--cache-ttl=168h"
        volumeMounts:
          - name: docker-config
            mountPath: /kaniko/.docker
      volumes:
        - name: docker-config
          secret:
            secretName: docker-credentials
      restartPolicy: Never

GitLab CI with Kaniko

build_image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:latest
    entrypoint: [""]
  script:
    - /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      --cache=true
      --cache-repo $CI_REGISTRY_IMAGE/cache

Cloud Native Buildpacks

Buildpacks automatically detect language runtime and create optimized images without Dockerfiles.

When to Use

Rapid prototyping without Dockerfile maintenance
Standardized builds across polyglot teams
Heroku-style developer experience

Example: Building with Pack CLI

# Install pack CLI
brew install buildpacks/tap/pack

# Build without Dockerfile
pack build myapp:latest --path ./src --builder paketobuildpacks/builder:base

# The buildpack auto-detects:
# - package.json → Node.js
# - requirements.txt → Python
# - pom.xml → Java
# - go.mod → Go

Part 2: Multi-Stage Build Patterns

Multi-stage builds separate build dependencies from runtime, producing minimal production images.

Go Application (Minimal: ~10MB)

# Build stage
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o main .

# Runtime stage (scratch = empty image)
FROM scratch
COPY --from=builder /app/main /main
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/main"]

Result: ~10MB image with no shell, no package manager, no OS.

Node.js Application (Distroless)

# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

# Runtime stage (distroless = minimal runtime only)
FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["dist/server.js"]

Result: ~100MB image vs ~1GB with full node image.

Python Application

# Build stage
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt
COPY . .

# Runtime stage
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY --from=builder /app .
ENV PATH=/root/.local/bin:$PATH
USER nobody
CMD ["python", "app.py"]

Part 3: Container Hardening Techniques

The Security Hardening Stack

flowchart TD
    subgraph ContainerSecurity["Container Security"]
        subgraph ImageSecurity["Image Security"]
            I1["1. Minimal Base (Distroless/Alpine/Scratch)"]
            I2["2. Non-root User"]
            I3["3. Read-only Filesystem"]
            I4["4. Dropped Capabilities"]
            I5["5. No Privileged Escalation"]
        end
        subgraph BuildSecurity["Build Security"]
            B1["6. Image Scanning (Trivy/Grype)"]
            B2["7. SBOM Generation"]
            B3["8. Image Signing (Cosign)"]
        end
        subgraph RuntimeSecurity["Runtime Security"]
            R1["9. Network Policies"]
            R2["10. Resource Limits"]
            R3["11. Seccomp/AppArmor Profiles"]
            R4["12. Pod Security Standards"]
        end
    end

Technique 1: Minimal Base Images

Base Image	Size	Contains	Use Case
`scratch`	0 MB	Nothing	Static binaries only
`distroless/static`	~2 MB	CA certs, tzdata	Go static binaries
`distroless/base`	~20 MB	glibc, CA certs	C/C++ applications
`alpine`	~5 MB	musl, busybox, apk	When you need a shell
`debian-slim`	~80 MB	glibc, apt	Full compatibility

Distroless Example

# Distroless images contain NO shell, NO package manager
FROM gcr.io/distroless/static-debian12

# This is all you can do - copy binaries
COPY --from=builder /app/main /

# No RUN, no CMD with shell syntax
ENTRYPOINT ["/main"]

What Distroless Removes

❌ Shell (bash, sh, ash)
❌ Package manager (apt, apk, yum)
❌ Utilities (curl, wget, ps, top)
❌ Build tools (gcc, make)
❌ Debuggers (gdb, strace)

Attack Surface Reduction: A typical node:20 image has ~500+ vulnerabilities. gcr.io/distroless/nodejs20 has ~0-5.

Technique 2: Run as Non-Root User

# Create dedicated user and group
RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid 1000 --shell /sbin/nologin appuser

# Set ownership
COPY --chown=appuser:appgroup . /app

# Switch to non-root
USER appuser

# All subsequent commands run as appuser

Kubernetes Pod Security Context

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true

Technique 3: Read-Only Root Filesystem

Immutable containers prevent runtime modifications and persistence mechanisms.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        readOnlyRootFilesystem: true
      # Mount writable volumes for required paths
      volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /var/cache/app
        - name: run
          mountPath: /var/run
  volumes:
    - name: tmp
      emptyDir: {}
    - name: cache
      emptyDir: {}
    - name: run
      emptyDir: {}

What This Prevents

Writing backdoors to /bin or /usr/bin
Modifying configuration files
Installing additional packages
Persistence through filesystem changes

Technique 4: Drop Linux Capabilities

Containers inherit a default set of Linux capabilities. Drop all unnecessary ones.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        capabilities:
          drop:
            - ALL          # Drop everything first
          add:
            - NET_BIND_SERVICE  # Add back only what's needed

Common Capabilities to Consider

Capability	What It Allows	When Needed
`NET_BIND_SERVICE`	Bind to ports < 1024	Web servers on port 80/443
`CHOWN`	Change file ownership	Never in production
`SETUID/SETGID`	Elevate privileges	Never in production
`SYS_ADMIN`	System administration	Almost never

Technique 5: Image Scanning (CI/CD Gate)

Trivy: Universal Scanner

# GitHub Actions
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'myapp:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'  # Fail on vulnerabilities
    ignore-unfixed: true  # Only fail on fixable issues

Grype: Risk-Prioritized Scanning

# Grype combines CVSS + EPSS + Known Exploited
grype myapp:latest --fail-on high

# Output includes exploitability data
# CVE-2023-12345 (HIGH) - EPSS: 0.82 (82% chance of exploitation)

Blocking Policy Example

# .trivyignore - Suppress specific CVEs with justification
CVE-2023-12345 # Not exploitable in our use case
CVE-2023-67890 # Pending upstream fix, tracked in JIRA-123

Technique 6: SBOM Generation

Software Bill of Materials provides supply chain transparency.

# Generate SBOM with Trivy
trivy image --format cyclonedx --output sbom.json myapp:latest

# Generate with Syft (more detailed)
syft myapp:latest -o cyclonedx > sbom.json

# Scan SBOM for vulnerabilities (shift-left)
grype sbom:sbom.json

Attach SBOM to Image (Cosign)

# Sign image
cosign sign --key cosign.key registry.io/myapp:v1.0.0

# Attach SBOM
cosign attach sbom --sbom sbom.json registry.io/myapp:v1.0.0

# Verify
cosign verify --key cosign.pub registry.io/myapp:v1.0.0

Technique 7: CIS Docker Benchmark Compliance

The CIS Docker Benchmark provides 100+ security controls. Automate compliance checking:

# Run Docker Bench Security
docker run --rm --net host --pid host --userns host \
  --cap-add audit_control \
  -v /etc:/etc:ro \
  -v /var/lib:/var/lib:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  docker/docker-bench-security

Key CIS Controls for Images

Control	Description	Implementation
4.1	Create user for container	`USER appuser` in Dockerfile
4.3	Use trusted base images	Pin to `@sha256` digests
4.6	Add HEALTHCHECK	`HEALTHCHECK CMD curl -f http://localhost/`
5.1	Set container resource limits	`docker run --memory=512m --cpus=0.5`
5.7	Don't use privileged containers	Never use `--privileged`
5.11	Don't use host network	Don't use `--net host`

Technique 8: Resource Limits (DoS Prevention)

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests:
          memory: "128Mi"
          cpu: "100m"
        limits:
          memory: "512Mi"    # Hard limit - OOM kill if exceeded
          cpu: "500m"        # Throttled if exceeded
          ephemeral-storage: "1Gi"

Why This Matters

Memory: Without limits, a compromised container can consume all host memory
CPU: Prevents crypto-mining from starving other workloads
Ephemeral Storage: Prevents disk exhaustion attacks

Part 4: Complete Hardened Deployment Example

Dockerfile

# syntax=docker/dockerfile:1.4

# Build stage
FROM golang:1.22-alpine AS builder

WORKDIR /app

# Dependency caching
COPY go.* ./
RUN go mod download

# Build
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app/server .

# Runtime stage
FROM gcr.io/distroless/static-debian12:nonroot

# Copy binary (nonroot user UID 65532 is built-in)
COPY --from=builder --chown=65532:65532 /app/server /server

# Health check (requires curl-enabled distroless variant or custom)
# For distroless/static, health checks are done via Kubernetes probes

# Use nonroot user (built into distroless)
USER 65532:65532

ENTRYPOINT ["/server"]

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hardened-app
spec:
  template:
    metadata:
      annotations:
        # Container runtime security
        container.apparmor.security.beta.kubernetes.io/app: runtime/default
    spec:
      securityContext:
        # Pod-level security
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
        seccompProfile:
          type: RuntimeDefault
      
      containers:
        - name: app
          image: registry.io/myapp:v1.0.0@sha256:abc123...
          
          securityContext:
            # Container-level security
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "256Mi"
              cpu: "200m"
          
          # Mount writable volumes for required paths
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: cache
              mountPath: /var/cache
          
          # Health checks
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
      
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: "100Mi"
        - name: cache
          emptyDir:
            sizeLimit: "200Mi"

CI/CD Pipeline (GitHub Actions)

name: Secure Build Pipeline

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # For OIDC token
      contents: read
      security-events: write  # For SARIF upload
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Login to Registry
        uses: docker/login-action@v3
        with:
          registry: registry.io
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      
      - name: Build image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          tags: registry.io/myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          sbom: true
          provenance: true
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: registry.io/myapp:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'
          ignore-unfixed: true
      
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'
      
      - name: Push image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            registry.io/myapp:${{ github.sha }}
            registry.io/myapp:latest
          cache-from: type=gha
          sbom: true
          provenance: true
      
      - name: Sign image with Cosign
        run: |
          cosign sign --yes registry.io/myapp:${{ github.sha }}

Summary Checklist

Build Security

Item	Check
Use BuildKit or buildx	☐
Multi-stage builds	☐
Pin base image to digest	☐
No secrets in images	☐
Generate SBOM	☐
Sign images	☐

Image Security

Item	Check
Minimal base (distroless/alpine)	☐
Non-root user	☐
Read-only root filesystem	☐
Dropped capabilities	☐
No privileged containers	☐

Runtime Security

Item	Check
Resource limits set	☐
Liveness/readiness probes	☐
Network policies	☐
Seccomp profile	☐
Pod Security Standards	☐

CI/CD Security

Item	Check
Vulnerability scanning gate	☐
SARIF upload to security tab	☐
Ignore unfixed CVEs only with justification	☐
Fail on HIGH/CRITICAL	☐

Key Takeaways

BuildKit is the modern standard — 3x faster, better caching, multi-platform support
Distroless images minimize attack surface — No shell, no package manager, ~0-5 CVEs vs 500+
Defense in depth — Layer non-root, read-only filesystem, dropped capabilities
Automate security gates — Trivy/Grype in CI/CD with fail thresholds
Immutability is security — Read-only filesystems prevent persistence mechanisms
SBOM + Signing = Supply Chain Security — Know what's in your images and prove it

The methodology produces containers that are smaller, faster, and significantly more secure than traditional approaches.