Implementing Immutable Infrastructure with Packer and Cloud-Init on AlmaLinux

Immutable infrastructure has become a cornerstone of modern DevOps practices, eliminating configuration drift and ensuring consistent deployments. This comprehensive guide will walk you through implementing immutable infrastructure using Packer for automated image building and Cloud-Init for instance initialization on AlmaLinux.

Understanding Immutable Infrastructure

Immutable infrastructure treats servers as disposable and replaceable rather than maintaining and updating them. Key benefits include:

• Consistency: Every deployment uses identical images
• Reliability: No configuration drift or snowflake servers
• Security: Reduced attack surface with minimal runtime changes
• Scalability: Quick provisioning of pre-configured instances
• Rollback: Easy reversion to previous versions

Prerequisites

Before implementing immutable infrastructure, ensure you have:

• AlmaLinux 9 build server
• Packer 1.9+ installed
• Cloud provider account (AWS/Azure/GCP) or local virtualization
• Git for version control
• Basic understanding of JSON/HCL and shell scripting
• Docker (optional, for container images)

Setting Up the Build Environment

Installing Packer on AlmaLinux

# Install required packages
sudo dnf install -y wget unzip git

# Download Packer
PACKER_VERSION="1.9.4"
wget https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip

# Install Packer
sudo unzip packer_${PACKER_VERSION}_linux_amd64.zip -d /usr/local/bin/
sudo chmod +x /usr/local/bin/packer

# Verify installation
packer version

# Install additional tools
sudo dnf install -y qemu-img libvirt virt-install
sudo dnf install -y cloud-utils-growpart cloud-init

Setting Up Project Structure

# Create project directory
mkdir -p ~/immutable-infrastructure/{packer,scripts,cloud-init,tests}
cd ~/immutable-infrastructure

# Initialize git repository
git init
echo "*.log" > .gitignore
echo "output-*/" >> .gitignore
echo "*.box" >> .gitignore
echo "packer_cache/" >> .gitignore

# Create directory structure
tree -d
# .
# ├── cloud-init
# ├── packer
# ├── scripts
# └── tests

Creating Base Images with Packer

Basic Packer Template

Create a Packer template for AlmaLinux:

// packer/almalinux-9-base.json
{
  "variables": {
    "iso_url": "https://repo.almalinux.org/almalinux/9/isos/x86_64/AlmaLinux-9.2-x86_64-minimal.iso",
    "iso_checksum": "sha256:f501de55f92e59a3fcf4ad252fdfc4e02ee2ad013fd4c54d24af0a997cf6f530",
    "ssh_username": "packer",
    "ssh_password": "packer",
    "vm_name": "almalinux-9-base",
    "version": "{{timestamp}}"
  },
  "builders": [
    {
      "type": "qemu",
      "iso_url": "{{user `iso_url`}}",
      "iso_checksum": "{{user `iso_checksum`}}",
      "output_directory": "output-{{user `vm_name`}}",
      "shutdown_command": "sudo -S shutdown -P now",
      "disk_size": "20480",
      "format": "qcow2",
      "accelerator": "kvm",
      "http_directory": "http",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_timeout": "20m",
      "vm_name": "{{user `vm_name`}}-{{user `version`}}",
      "net_device": "virtio-net",
      "disk_interface": "virtio",
      "boot_wait": "10s",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/kickstart.cfg<enter><wait>"
      ],
      "memory": 2048,
      "cpus": 2
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/base.sh",
        "scripts/security.sh",
        "scripts/cloud-init.sh",
        "scripts/cleanup.sh"
      ],
      "execute_command": "echo 'packer' | {{.Vars}} sudo -S -E bash '{{.Path}}'"
    }
  ],
  "post-processors": [
    {
      "type": "compress",
      "output": "builds/{{user `vm_name`}}-{{user `version`}}.tar.gz"
    }
  ]
}

HCL2 Format (Recommended)

Convert to modern HCL2 format:

// packer/almalinux-9-base.pkr.hcl
packer {
  required_plugins {
    qemu = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/qemu"
    }
    amazon = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

variable "iso_url" {
  type    = string
  default = "https://repo.almalinux.org/almalinux/9/isos/x86_64/AlmaLinux-9.2-x86_64-minimal.iso"
}

variable "iso_checksum" {
  type    = string
  default = "sha256:f501de55f92e59a3fcf4ad252fdfc4e02ee2ad013fd4c54d24af0a997cf6f530"
}

variable "vm_name" {
  type    = string
  default = "almalinux-9-base"
}

variable "version" {
  type    = string
  default = ""
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
  version   = var.version != "" ? var.version : local.timestamp
}

source "qemu" "almalinux" {
  iso_url          = var.iso_url
  iso_checksum     = var.iso_checksum
  output_directory = "output-${var.vm_name}"
  shutdown_command = "sudo -S shutdown -P now"
  disk_size        = "20G"
  format           = "qcow2"
  accelerator      = "kvm"
  http_directory   = "http"
  ssh_username     = "packer"
  ssh_password     = "packer"
  ssh_timeout      = "20m"
  vm_name          = "${var.vm_name}-${local.version}"
  memory           = 2048
  cpus             = 2
  
  boot_wait = "10s"
  boot_command = [
    "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/kickstart.cfg<enter><wait>"
  ]
}

build {
  name = "almalinux-base"
  sources = ["source.qemu.almalinux"]
  
  provisioner "shell" {
    scripts = [
      "scripts/base.sh",
      "scripts/security.sh",
      "scripts/cloud-init.sh",
      "scripts/cleanup.sh"
    ]
    execute_command = "echo 'packer' | {{.Vars}} sudo -S -E bash '{{.Path}}'"
  }
  
  provisioner "file" {
    source      = "cloud-init/99-datasource.cfg"
    destination = "/tmp/99-datasource.cfg"
  }
  
  provisioner "shell" {
    inline = [
      "sudo mv /tmp/99-datasource.cfg /etc/cloud/cloud.cfg.d/",
      "sudo cloud-init clean --logs --machine-id"
    ]
  }
  
  post-processor "compress" {
    output = "builds/${var.vm_name}-${local.version}.tar.gz"
  }
  
  post-processor "manifest" {
    output = "builds/manifest.json"
  }
}

Kickstart Configuration

Create automated installation file:

# http/kickstart.cfg
#version=RHEL9
text
skipx
install
cdrom

lang en_US.UTF-8
keyboard us
timezone UTC --isUtc

network --bootproto=dhcp --device=link --activate
network --hostname=almalinux-template

rootpw --plaintext packer
user --name=packer --plaintext --password packer --groups=wheel

firewall --enabled --service=ssh
selinux --enforcing
services --enabled=sshd,chronyd,cloud-init,cloud-init-local,cloud-config,cloud-final

bootloader --location=mbr --append="crashkernel=auto"
zerombr
clearpart --all --initlabel
autopart --type=lvm

firstboot --disable
reboot

%packages --ignoremissing
@^minimal-environment
@standard
cloud-init
cloud-utils-growpart
openssh-server
sudo
curl
wget
vim
git
python3
python3-pip
NetworkManager
chrony
kernel-devel
kernel-headers
gcc
make
perl
selinux-policy-devel
elfutils-libelf-devel
-plymouth
-plymouth-core-libs
-postfix
%end

%post
# Enable sudo without password for packer user
echo "packer ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/packer
chmod 440 /etc/sudoers.d/packer

# Configure SSH
sed -i 's/^#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config

# Update system
dnf update -y

# Clean up
dnf clean all
%end

Provisioning Scripts

Base Configuration Script

#!/bin/bash
# scripts/base.sh

set -ex

# Update system
sudo dnf update -y

# Install essential packages
sudo dnf install -y \
    epel-release \
    vim \
    wget \
    curl \
    git \
    htop \
    net-tools \
    bind-utils \
    bash-completion \
    yum-utils \
    device-mapper-persistent-data \
    lvm2

# Configure system settings
cat <<EOF | sudo tee /etc/sysctl.d/99-custom.conf
# Network optimizations
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_keepalive_time = 300

# Security settings
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0

# Performance tuning
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
EOF

sudo sysctl -p /etc/sysctl.d/99-custom.conf

# Configure time synchronization
sudo systemctl enable chronyd
sudo systemctl start chronyd

# Create standard directories
sudo mkdir -p /opt/app /var/log/app /etc/app
sudo chmod 755 /opt/app /var/log/app /etc/app

Security Hardening Script

#!/bin/bash
# scripts/security.sh

set -ex

# Install security tools
sudo dnf install -y \
    audit \
    aide \
    firewalld \
    fail2ban \
    rkhunter

# Configure firewall
sudo systemctl enable firewalld
sudo systemctl start firewalld
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload

# Configure SELinux
sudo setenforce 1
sudo sed -i 's/SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

# SSH hardening
sudo sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/#PubkeyAuthentication.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sudo sed -i 's/#ClientAliveInterval.*/ClientAliveInterval 300/' /etc/ssh/sshd_config
sudo sed -i 's/#ClientAliveCountMax.*/ClientAliveCountMax 2/' /etc/ssh/sshd_config

# Configure fail2ban
sudo systemctl enable fail2ban
cat <<EOF | sudo tee /etc/fail2ban/jail.local
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5

[sshd]
enabled = true
EOF

# Configure audit rules
cat <<EOF | sudo tee /etc/audit/rules.d/immutable.rules
# Monitor authentication
-w /etc/passwd -p wa -k passwd_changes
-w /etc/shadow -p wa -k shadow_changes
-w /etc/group -p wa -k group_changes
-w /etc/sudoers -p wa -k sudoers_changes

# Monitor system calls
-a always,exit -F arch=b64 -S execve -k exec_commands
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -k owner_mod

# Monitor configuration
-w /etc/ -p wa -k config_changes
EOF

sudo systemctl enable auditd

# Remove unnecessary packages
sudo dnf remove -y \
    telnet \
    rsh \
    rsh-server \
    xinetd \
    tftp-server \
    vsftpd \
    || true

# Set secure permissions
sudo chmod 600 /etc/crontab
sudo chmod 600 /etc/ssh/sshd_config
sudo chmod 644 /etc/passwd
sudo chmod 000 /etc/shadow
sudo chmod 644 /etc/group
sudo chmod 600 /etc/gshadow

Cloud-Init Installation Script

#!/bin/bash
# scripts/cloud-init.sh

set -ex

# Install cloud-init
sudo dnf install -y cloud-init cloud-utils-growpart gdisk

# Configure cloud-init
cat <<EOF | sudo tee /etc/cloud/cloud.cfg
# Cloud-Init Configuration for AlmaLinux

users:
  - default

disable_root: true
ssh_pwauth: false

mount_default_fields: [~, ~, 'auto', 'defaults,nofail,x-systemd.requires=cloud-init.service', '0', '2']
resize_rootfs_tmp: /dev
ssh_deletekeys: true
ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']
syslog_fix_perms: ~

cloud_init_modules:
  - migrator
  - seed_random
  - bootcmd
  - write-files
  - growpart
  - resizefs
  - disk_setup
  - mounts
  - set_hostname
  - update_hostname
  - update_etc_hosts
  - ca-certs
  - rsyslog
  - users-groups
  - ssh

cloud_config_modules:
  - emit_upstart
  - snap
  - ssh-import-id
  - locale
  - set-passwords
  - grub-dpkg
  - apt-pipelining
  - apt-configure
  - ubuntu-advantage
  - ntp
  - timezone
  - disable-ec2-metadata
  - runcmd
  - byobu

cloud_final_modules:
  - package-update-upgrade-install
  - fan
  - landscape
  - lxd
  - ubuntu-drivers
  - write-files-deferred
  - puppet
  - chef
  - salt-minion
  - mcollective
  - rightscale_userdata
  - scripts-vendor
  - scripts-per-once
  - scripts-per-boot
  - scripts-per-instance
  - scripts-user
  - ssh-authkey-fingerprints
  - keys-to-console
  - phone-home
  - final-message
  - power-state-change

system_info:
  distro: almalinux
  default_user:
    name: cloud-user
    lock_passwd: true
    gecos: Cloud User
    groups: [wheel, adm, systemd-journal]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
  paths:
    cloud_dir: /var/lib/cloud
    templates_dir: /etc/cloud/templates
    upstart_dir: /etc/init/
  ssh_svcname: sshd

datasource_list: ['NoCloud', 'ConfigDrive', 'OpenStack', 'Ec2', 'None']
EOF

# Create cloud-init datasource configuration
cat <<EOF | sudo tee /etc/cloud/cloud.cfg.d/99-datasource.cfg
datasource_list: [ NoCloud, ConfigDrive, OpenStack, Ec2, None ]
datasource:
  NoCloud:
    seedfrom: /var/lib/cloud/seed/nocloud-net
EOF

# Enable services
sudo systemctl enable cloud-init-local
sudo systemctl enable cloud-init
sudo systemctl enable cloud-config
sudo systemctl enable cloud-final

# Clean cloud-init for fresh run
sudo cloud-init clean --logs --machine-id

Cleanup Script

#!/bin/bash
# scripts/cleanup.sh

set -ex

# Clean package cache
sudo dnf clean all
sudo rm -rf /var/cache/dnf/*

# Clean logs
sudo find /var/log -type f -name "*.log" -exec truncate -s 0 {} \;
sudo rm -rf /var/log/journal/*
sudo rm -f /var/log/audit/audit.log

# Clean temporary files
sudo rm -rf /tmp/* /var/tmp/*
sudo rm -rf /root/.bash_history
sudo rm -rf /home/packer/.bash_history

# Clean SSH keys (will be regenerated)
sudo rm -f /etc/ssh/ssh_host_*

# Clean machine ID (for unique instances)
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id

# Clean network configuration
sudo rm -f /etc/udev/rules.d/70-persistent-net.rules
sudo rm -f /etc/NetworkManager/system-connections/*

# Remove packer user (if needed)
# sudo userdel -r packer || true

# Zero out free space
sudo dd if=/dev/zero of=/EMPTY bs=1M || true
sudo rm -f /EMPTY

# Sync to ensure data is written
sync

Cloud-Init Templates

User Data Template

# cloud-init/user-data.yaml
#cloud-config
users:
  - name: admin
    groups: [wheel, adm]
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC... [email protected]

# Update packages on first boot
package_update: true
package_upgrade: true

# Install additional packages
packages:
  - htop
  - vim
  - git
  - docker
  - python3-pip

# Configure hostname
hostname: ${hostname}
fqdn: ${hostname}.${domain}
manage_etc_hosts: true

# Configure timezone
timezone: UTC

# Run commands on first boot
runcmd:
  - echo "Instance initialized at $(date)" >> /var/log/cloud-init-custom.log
  - systemctl enable docker
  - systemctl start docker
  - usermod -aG docker admin
  - curl -L "https://github.com/docker/compose/releases/download/v2.20.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  - chmod +x /usr/local/bin/docker-compose

# Configure Docker daemon
write_files:
  - path: /etc/docker/daemon.json
    content: |
      {
        "log-driver": "json-file",
        "log-opts": {
          "max-size": "10m",
          "max-file": "3"
        },
        "storage-driver": "overlay2",
        "live-restore": true
      }
  
  - path: /etc/systemd/system/app.service
    content: |
      [Unit]
      Description=Application Service
      After=docker.service
      Requires=docker.service
      
      [Service]
      Type=simple
      Restart=always
      RestartSec=5
      ExecStart=/usr/local/bin/docker-compose -f /opt/app/docker-compose.yml up
      ExecStop=/usr/local/bin/docker-compose -f /opt/app/docker-compose.yml down
      
      [Install]
      WantedBy=multi-user.target

# Configure swap
swap:
  filename: /swapfile
  size: 2G
  maxsize: 2G

# Phone home on boot
phone_home:
  url: https://monitoring.example.com/api/v1/instance-boot
  post:
    - instance_id
    - hostname
    - fqdn
  tries: 5

Network Configuration Template

# cloud-init/network-config.yaml
version: 2
ethernets:
  eth0:
    dhcp4: true
    dhcp6: false
    match:
      name: e*
    set-name: eth0
  eth1:
    addresses:
      - ${private_ip}/24
    match:
      name: e*
    set-name: eth1

Building Multi-Cloud Images

AWS AMI Configuration

// packer/aws-ami.pkr.hcl
source "amazon-ebs" "almalinux" {
  ami_name      = "almalinux-9-immutable-${local.timestamp}"
  instance_type = "t3.medium"
  region        = var.aws_region
  
  source_ami_filter {
    filters = {
      name                = "AlmaLinux OS 9*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
      architecture        = "x86_64"
    }
    most_recent = true
    owners      = ["679593333241"] # AlmaLinux official
  }
  
  ssh_username = "ec2-user"
  
  launch_block_device_mappings {
    device_name           = "/dev/sda1"
    volume_size           = 20
    volume_type           = "gp3"
    delete_on_termination = true
  }
  
  tags = {
    Name        = "AlmaLinux 9 Immutable"
    Version     = local.version
    BuildDate   = local.timestamp
    Builder     = "Packer"
    Base_AMI_ID = "{{ .SourceAMI }}"
  }
}

Azure Image Configuration

// packer/azure-image.pkr.hcl
source "azure-arm" "almalinux" {
  use_azure_cli_auth = true
  
  managed_image_name                = "almalinux-9-immutable-${local.timestamp}"
  managed_image_resource_group_name = var.azure_resource_group
  
  os_type         = "Linux"
  image_publisher = "almalinux"
  image_offer     = "almalinux"
  image_sku       = "9-gen2"
  
  azure_tags = {
    Name      = "AlmaLinux 9 Immutable"
    Version   = local.version
    BuildDate = local.timestamp
    Builder   = "Packer"
  }
  
  location = var.azure_location
  vm_size  = "Standard_DS2_v2"
}

Google Cloud Image Configuration

// packer/gcp-image.pkr.hcl
source "googlecompute" "almalinux" {
  project_id   = var.gcp_project_id
  source_image = "almalinux-cloud/almalinux-9"
  zone         = var.gcp_zone
  
  image_name        = "almalinux-9-immutable-${local.timestamp}"
  image_description = "AlmaLinux 9 Immutable Infrastructure Image"
  image_family      = "almalinux-9-immutable"
  
  ssh_username = "packer"
  
  machine_type = "n1-standard-2"
  disk_size    = 20
  disk_type    = "pd-standard"
  
  image_labels = {
    name      = "almalinux-9-immutable"
    version   = local.version
    builddate = local.timestamp
    builder   = "packer"
  }
}

Container Images

Building Container Images

# docker/Dockerfile.almalinux
FROM almalinux:9-minimal

# Install systemd
RUN microdnf install -y systemd systemd-libs && \
    microdnf clean all

# Install cloud-init and dependencies
RUN microdnf install -y \
    cloud-init \
    cloud-utils-growpart \
    openssh-server \
    sudo \
    python3 \
    NetworkManager \
    && microdnf clean all

# Configure systemd
RUN systemctl enable sshd && \
    systemctl enable cloud-init-local && \
    systemctl enable cloud-init && \
    systemctl enable cloud-config && \
    systemctl enable cloud-final

# Clean machine-id
RUN truncate -s 0 /etc/machine-id

VOLUME ["/sys/fs/cgroup"]
CMD ["/usr/sbin/init"]

Packer Container Configuration

// packer/container.pkr.hcl
source "docker" "almalinux" {
  image  = "almalinux:9"
  commit = true
  
  changes = [
    "ENV LANG=en_US.UTF-8",
    "ENV CONTAINER=docker",
    "VOLUME /sys/fs/cgroup",
    "CMD [\"/usr/sbin/init\"]"
  ]
}

build {
  sources = ["source.docker.almalinux"]
  
  provisioner "shell" {
    inline = [
      "dnf install -y systemd cloud-init",
      "dnf clean all",
      "systemctl enable cloud-init-local cloud-init cloud-config cloud-final"
    ]
  }
  
  post-processor "docker-tag" {
    repository = "myregistry/almalinux"
    tags       = ["9-immutable", "9-immutable-${local.version}"]
  }
  
  post-processor "docker-push" {
    login          = true
    login_username = var.docker_username
    login_password = var.docker_password
  }
}

Automation and CI/CD

GitLab CI Pipeline

# .gitlab-ci.yml
stages:
  - validate
  - build
  - test
  - publish

variables:
  PACKER_VERSION: "1.9.4"

before_script:
  - wget -q https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip
  - unzip -o packer_${PACKER_VERSION}_linux_amd64.zip -d /usr/local/bin/
  - chmod +x /usr/local/bin/packer

validate:
  stage: validate
  script:
    - packer validate packer/almalinux-9-base.pkr.hcl
    - packer validate packer/aws-ami.pkr.hcl
    - packer validate packer/azure-image.pkr.hcl

build:qemu:
  stage: build
  script:
    - packer build -var "version=$CI_COMMIT_SHORT_SHA" packer/almalinux-9-base.pkr.hcl
  artifacts:
    paths:
      - builds/
    expire_in: 7 days
  only:
    - main

build:aws:
  stage: build
  script:
    - packer build -var "version=$CI_COMMIT_SHORT_SHA" packer/aws-ami.pkr.hcl
  only:
    - tags

test:
  stage: test
  script:
    - ./tests/test-image.sh builds/almalinux-9-base-*.tar.gz
  dependencies:
    - build:qemu

publish:
  stage: publish
  script:
    - ./scripts/publish-image.sh
  only:
    - tags

GitHub Actions Workflow

# .github/workflows/build.yml
name: Build Immutable Images

on:
  push:
    branches: [main]
    tags: ['v*']
  pull_request:
    branches: [main]

env:
  PACKER_VERSION: "1.9.4"

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Packer
        uses: hashicorp/setup-packer@main
        with:
          version: ${{ env.PACKER_VERSION }}
      
      - name: Validate Templates
        run: |
          packer validate packer/almalinux-9-base.pkr.hcl
          packer validate packer/aws-ami.pkr.hcl

  build-qemu:
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Packer
        uses: hashicorp/setup-packer@main
        with:
          version: ${{ env.PACKER_VERSION }}
      
      - name: Build QEMU Image
        run: |
          packer build \
            -var "version=${{ github.sha }}" \
            packer/almalinux-9-base.pkr.hcl
      
      - name: Upload Artifacts
        uses: actions/upload-artifact@v3
        with:
          name: qemu-image
          path: builds/*.tar.gz

  build-aws:
    needs: validate
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Build AWS AMI
        run: |
          packer build \
            -var "version=${{ github.ref_name }}" \
            packer/aws-ami.pkr.hcl

Testing Immutable Images

Automated Testing Script

#!/bin/bash
# tests/test-image.sh

set -e

IMAGE_PATH=$1
TEMP_DIR=$(mktemp -d)

echo "Testing image: $IMAGE_PATH"

# Extract image
tar -xzf "$IMAGE_PATH" -C "$TEMP_DIR"

# Convert to raw format for mounting
qemu-img convert -f qcow2 -O raw "$TEMP_DIR"/*.qcow2 "$TEMP_DIR/disk.raw"

# Create loop device
LOOP_DEVICE=$(sudo losetup -f --show -P "$TEMP_DIR/disk.raw")

# Mount root partition
sudo mkdir -p /mnt/test-image
sudo mount "${LOOP_DEVICE}p2" /mnt/test-image

# Run tests
echo "Running filesystem tests..."

# Check for required files
test -f /mnt/test-image/etc/cloud/cloud.cfg || exit 1
test -d /mnt/test-image/var/lib/cloud || exit 1

# Check installed packages
sudo chroot /mnt/test-image rpm -q cloud-init || exit 1
sudo chroot /mnt/test-image rpm -q openssh-server || exit 1

# Check services
sudo chroot /mnt/test-image systemctl is-enabled cloud-init || exit 1
sudo chroot /mnt/test-image systemctl is-enabled sshd || exit 1

# Security checks
test ! -f /mnt/test-image/etc/ssh/ssh_host_rsa_key || exit 1
test -z "$(sudo cat /mnt/test-image/etc/machine-id)" || exit 1

echo "All tests passed!"

# Cleanup
sudo umount /mnt/test-image
sudo losetup -d "$LOOP_DEVICE"
rm -rf "$TEMP_DIR"

Integration Testing

#!/usr/bin/env python3
# tests/integration_test.py

import boto3
import time
import paramiko
import pytest

class TestImmutableInstance:
    @pytest.fixture
    def ec2_instance(self):
        """Launch test instance"""
        ec2 = boto3.resource('ec2')
        
        # Launch instance
        instances = ec2.create_instances(
            ImageId='ami-xxxxxxxxx',  # Your AMI ID
            MinCount=1,
            MaxCount=1,
            InstanceType='t3.micro',
            KeyName='test-key',
            UserData='''#cloud-config
users:
  - name: test
    ssh_authorized_keys:
      - ssh-rsa AAAAB3... [email protected]
'''
        )
        
        instance = instances[0]
        instance.wait_until_running()
        instance.reload()
        
        yield instance
        
        # Cleanup
        instance.terminate()
    
    def test_cloud_init_completed(self, ec2_instance):
        """Test that cloud-init completed successfully"""
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        
        # Wait for SSH
        time.sleep(60)
        
        ssh.connect(
            hostname=ec2_instance.public_ip_address,
            username='test',
            key_filename='test-key.pem'
        )
        
        # Check cloud-init status
        stdin, stdout, stderr = ssh.exec_command('cloud-init status')
        assert 'done' in stdout.read().decode()
        
        # Check services
        stdin, stdout, stderr = ssh.exec_command('systemctl is-active sshd')
        assert 'active' in stdout.read().decode()
        
        ssh.close()

Deployment Strategies

Blue-Green Deployment

#!/bin/bash
# deploy/blue-green.sh

set -e

NEW_AMI_ID=$1
ASG_NAME="app-asg"
ELB_NAME="app-elb"

# Create new launch configuration
aws autoscaling create-launch-configuration \
    --launch-configuration-name "lc-${NEW_AMI_ID}" \
    --image-id "$NEW_AMI_ID" \
    --instance-type t3.medium \
    --security-groups sg-xxxxxxxx \
    --key-name production-key \
    --user-data file://user-data.yaml

# Update Auto Scaling Group
aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name "$ASG_NAME" \
    --launch-configuration-name "lc-${NEW_AMI_ID}"

# Start rolling update
aws autoscaling start-instance-refresh \
    --auto-scaling-group-name "$ASG_NAME" \
    --preferences '{"MinHealthyPercentage": 90, "InstanceWarmup": 300}'

# Monitor deployment
while true; do
    STATUS=$(aws autoscaling describe-instance-refreshes \
        --auto-scaling-group-name "$ASG_NAME" \
        --query 'InstanceRefreshes[0].Status' \
        --output text)
    
    if [ "$STATUS" == "Successful" ]; then
        echo "Deployment completed successfully"
        break
    elif [ "$STATUS" == "Failed" ] || [ "$STATUS" == "Cancelled" ]; then
        echo "Deployment failed"
        exit 1
    fi
    
    echo "Status: $STATUS"
    sleep 30
done

Terraform Integration

# terraform/main.tf
data "aws_ami" "almalinux" {
  most_recent = true
  owners      = ["self"]
  
  filter {
    name   = "name"
    values = ["almalinux-9-immutable-*"]
  }
  
  filter {
    name   = "tag:Version"
    values = [var.image_version]
  }
}

resource "aws_launch_template" "app" {
  name_prefix   = "app-"
  image_id      = data.aws_ami.almalinux.id
  instance_type = var.instance_type
  
  vpc_security_group_ids = [aws_security_group.app.id]
  key_name              = var.key_name
  
  user_data = base64encode(templatefile("${path.module}/user-data.yaml", {
    app_version = var.app_version
    environment = var.environment
  }))
  
  block_device_mappings {
    device_name = "/dev/sda1"
    
    ebs {
      volume_size           = 20
      volume_type           = "gp3"
      delete_on_termination = true
      encrypted            = true
    }
  }
  
  tag_specifications {
    resource_type = "instance"
    
    tags = {
      Name        = "app-instance"
      Environment = var.environment
      AMI_ID      = data.aws_ami.almalinux.id
    }
  }
}

resource "aws_autoscaling_group" "app" {
  name               = "app-asg"
  vpc_zone_identifier = var.subnet_ids
  target_group_arns  = [aws_lb_target_group.app.arn]
  health_check_type  = "ELB"
  health_check_grace_period = 300
  
  min_size         = var.min_size
  max_size         = var.max_size
  desired_capacity = var.desired_capacity
  
  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }
  
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 90
    }
  }
}

Monitoring and Compliance

Image Compliance Scanning

#!/bin/bash
# scripts/compliance-scan.sh

set -e

IMAGE_ID=$1

# Mount image for scanning
MOUNT_POINT=$(mktemp -d)
# ... mount logic ...

# Run OpenSCAP compliance scan
oscap xccdf eval \
    --profile xccdf_org.ssgproject.content_profile_cis \
    --results scan-results.xml \
    --report scan-report.html \
    /usr/share/xml/scap/ssg/content/ssg-almalinux9-ds.xml

# Check for vulnerabilities
grype dir:$MOUNT_POINT -o json > vulnerability-report.json

# Validate configuration
ansible-playbook -i localhost, -c local \
    --extra-vars "mount_point=$MOUNT_POINT" \
    compliance-checks.yml

# Generate compliance report
python3 generate-compliance-report.py \
    --scan-results scan-results.xml \
    --vuln-report vulnerability-report.json \
    --output compliance-report.pdf

Drift Detection

#!/usr/bin/env python3
# monitor/drift-detection.py

import boto3
import hashlib
import json
from datetime import datetime

def calculate_instance_hash(instance):
    """Calculate hash of instance configuration"""
    config = {
        'ami_id': instance.image_id,
        'instance_type': instance.instance_type,
        'security_groups': [sg['GroupId'] for sg in instance.security_groups],
        'subnet_id': instance.subnet_id,
        'tags': dict(instance.tags or [])
    }
    
    config_str = json.dumps(config, sort_keys=True)
    return hashlib.sha256(config_str.encode()).hexdigest()

def detect_drift():
    """Detect configuration drift in running instances"""
    ec2 = boto3.resource('ec2')
    
    # Get expected configuration
    with open('expected-config.json', 'r') as f:
        expected_config = json.load(f)
    
    drift_report = {
        'timestamp': datetime.utcnow().isoformat(),
        'instances': []
    }
    
    # Check all running instances
    for instance in ec2.instances.filter(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    ):
        instance_hash = calculate_instance_hash(instance)
        
        if instance_hash != expected_config.get('expected_hash'):
            drift_report['instances'].append({
                'instance_id': instance.id,
                'current_hash': instance_hash,
                'expected_hash': expected_config.get('expected_hash'),
                'ami_id': instance.image_id,
                'drift_detected': True
            })
    
    # Send alerts if drift detected
    if drift_report['instances']:
        send_alert(drift_report)
    
    return drift_report

if __name__ == '__main__':
    report = detect_drift()
    print(json.dumps(report, indent=2))

Best Practices

1. Version Control Everything

   • Track all configuration in Git
   • Tag releases properly
   • Use semantic versioning

2. Security First

   • Scan images for vulnerabilities
   • Implement least privilege
   • Regular security updates

3. Testing

   • Automated testing pipeline
   • Integration tests
   • Compliance validation

4. Documentation

   • Document build process
   • Maintain change logs
   • Create runbooks

5. Monitoring

   • Track image usage
   • Monitor for drift
   • Alert on anomalies

Conclusion

Implementing immutable infrastructure with Packer and Cloud-Init on AlmaLinux provides a robust foundation for reliable, scalable deployments. By treating infrastructure as code and automating image creation, you eliminate configuration drift, improve security, and enable rapid, consistent deployments across any environment.

The combination of Packer’s powerful image building capabilities and Cloud-Init’s flexible initialization system creates a perfect solution for modern infrastructure needs. Continue to iterate on your image pipeline, adding new security measures, optimizations, and compliance checks as your infrastructure evolves.