Introduction

Welcome to the part 4 of the homelab series! In the previous parts, we built a server, deployed a suite of services, and configured our network. Now, it’s time to make it resilient and self-maintaining. A homelab isn’t just about setting things up; it’s about keeping them running reliably.

This guide will show you how to set up the three pillars of modern IT operations: Automated Backups, Automated Updates, and Proactive Alerting. By the end, you’ll have a homelab that runs itself, ensures your data is safe, stays up-to-date, and notifies you when something goes wrong.

Chapter 1: The Automated Backup Strategy (at 3 AM)

A solid backup strategy is non-negotiable. I implemented a robust system inspired by the “3-2-1” rule, focusing on redundancy and an off-site copy. My strategy involves maintaining two copies of my data in two separate locations: one local backup on the server itself for fast recovery, and one automated, off-site backup to Google Drive to protect against a local disaster like a fire or hardware failure.

This script runs at 3 AM, creates a local backup, uploads it, and then notifies Discord.

Step 1: Configure rclone for Google Drive

First, you need a tool to communicate with Google Drive. We’ll use rclone.

  1. Install rclone on your Debian server:
    sudo -v ; curl https://rclone.org/install.sh | sudo bash
  2. Run the interactive setup:
    rclone config
  3. Follow the Prompts:
    • n (New remote) *
    • name>: gdrive (You can name it anything)
    • storage>: Find and select drive (Google Drive).
    • client_id> & client_secret>: Press Enter for both to leave blank.
    • scope>: Choose 1 (Full access).
    • Use auto config? y/n>: This is a critical step. Since we are on a headless server, type n and press Enter.
  4. Authorize Headless:
    • rclone will give you a command to run on a machine with a web browser (like your main computer).
    • On your main computer (where you have rclone installed), run the rclone authorize "drive" "..." command.
    • This will open your browser, ask you to log in to Google, and grant permission.
    • Your main computer’s terminal will then output a block of text (your config_token).
  5. Paste Token: Copy the token from your main computer and paste it back into your server’s rclone prompt.
  6. Finish the prompts, and your connection is complete.

Step 2: Create the Backup Script

Next, create a shell script to perform the backup.

  1. Create the file and make it executable:

    nano ~/backup.sh
chmod +x ~/backup.sh

  2. Paste in the following script. You must edit the first 7 variables to match your setup.

    #!/bin/bash

# --- Configuration ---
SOURCE_DIR="/path/to/your/docker"  # <-- Change to your Docker projects directory
BACKUP_DIR="/path/to/your/backups"  # <-- Change to your backups folder
FILENAME="homelab-backup-$(date +%Y-%m-%d).tar.gz"
LOCAL_RETENTION_DAYS=3
CLOUD_RETENTION_DAYS=3
RCLONE_REMOTE="gdrive"  # <-- Must match your rclone remote name
RCLONE_DEST="Homelab Backups"  # <-- Folder name in Google Drive

# --- "https://discordapp.com/api/webhooks/141949178941/6Tx6f1yjf26LztQ" ---
DISCORD_WEBHOOK_URL="YOUR_DISCORD_WEBHOOK_URL"

# --- Notification Function ---
send_notification() {
    MESSAGE=$1
    curl -H "Content-Type: application/json" -X POST -d "{\"content\": \"$MESSAGE\"}" "$DISCORD_WEBHOOK_URL"
}

# --- Script Logic ---
echo "--- Starting Homelab Backup: $(date) ---"
send_notification "✅ Starting Homelab Backup..."

# 1. Create local backup
echo "Creating local backup..."
tar -czf "${BACKUP_DIR}/${FILENAME}" -C "${SOURCE_DIR}" .
echo "Local backup created at ${BACKUP_DIR}/${FILENAME}"

# 2. Upload to Google Drive
echo "Uploading backup to ${RCLONE_REMOTE}..."
rclone copy "${BACKUP_DIR}/${FILENAME}" "${RCLONE_REMOTE}:${RCLONE_DEST}"
echo "Upload complete."

# 3. Clean up local backups
echo "Cleaning up local backups older than ${LOCAL_RETENTION_DAYS} days..."
find "${BACKUP_DIR}" -type f -name "*.tar.gz" -mtime +${LOCAL_RETENTION_DAYS} -delete
echo "Local cleanup complete."

# 4. Clean up cloud backups
echo "Cleaning up cloud backups older than ${CLOUD_RETENTION_DAYS} days..."
rclone delete "${RCLONE_REMOTE}:${RCLONE_DEST}" --min-age ${CLOUD_RETENTION_DAYS}d
echo "Cloud cleanup complete."

echo "Backup process finished."
send_notification "🎉 Homelab backup and cloud upload completed successfully!"

Step 3: Automate with Cron

To run this script automatically, you must add it to the root user’s crontab. This is critical for giving the script permission to read all Docker files.

  1. Open the root crontab editor:
    sudo crontab -e
  2. Add the following line to schedule the backup for 3:00 AM every morning: 0 3 * * * /path/to/your/backup.sh You will now get a fresh, onsite and off-site backup every night and a Discord message when it’s done.

Chapter 2: Automated Updates with Watchtower (at 6 AM)

Manually updating every Docker container is tedious. We can automate this by deploying Watchtower.

Step 1: The Docker Compose File

Create a docker-compose.yml for Watchtower. This configuration schedules it to run once a day at 6:00 AM, clean up old images, and send a Discord notification only if it finds an update.

  1. mkdir -p ~/docker/watchtower

  2. cd ~/docker/watchtower

  3. nano docker-compose.yml

  4. Paste in this configuration:

    services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
	    # Timezone setting
	    TZ: America/Chicago

	    # Discord notification settings
	    WATCHTOWER_NOTIFICATIONS: shoutrrr
	    WATCHTOWER_NOTIFICATION_URL: "discord://YOUR_DISCORD_WEBHOOK_ID_URL>

	    # Notification settings
	    WATCHTOWER_NOTIFICATIONS_LEVEL: info
	    WATCHTOWER_NOTIFICATION_REPORT: "true"
	    WATCHTOWER_NOTIFICATIONS_HOSTNAME: Homelab-Laptop

	    # Update settings
	    WATCHTOWER_CLEANUP: "true"
	    WATCHTOWER_INCLUDE_STOPPED: "false"
	    WATCHTOWER_INCLUDE_RESTARTING: "true"
	    WATCHTOWER_SCHEDULE: "0 0 6 * * *"

    Note: The WATCHTOWER_NOTIFICATION_URL uses a special shoutrrr format for Discord, which looks like discord://token@webhook-id.

Now, every morning at 6:00 AM, Watchtower will scan all running containers and update any that have a new image available.

Chapter 3: Proactive Alerting (24/7)

The final piece of automation is proactive alerting. This setup ensures you are immediately notified via Discord if something goes wrong.

Step 1: The Alerting Pipeline

The pipeline we’ll build is: Prometheus (detects problems) -> Alertmanager (groups and routes alerts) -> Discord (notifies you).

Step 2: Deploy Alertmanager

First, deploy Alertmanager. It must be on the same npm_default network as Prometheus.

  1. mkdir -p ~/docker/alertmanager

  2. cd ~/docker/alertmanager

  3. Create the alertmanager.yml configuration file:

    nano alertmanager.yml

  4. Paste in this configuration. It uses advanced routing to send critical alerts every 2 hours and warning alerts every 12 hours.

    global:
  resolve_timeout: 5m

route:
  group_by: ["alertname", "severity"]
  group_wait: 30s
  group_interval: 10m
  repeat_interbal: 12h
  receiver: "discord-notifications"
  routes:
    - receiver: "discord-notifications"
      matchers:
        - severity="critical"
      repeat_interval: 2h
    - receiver: "discord-notifications"
      matchers:
        - severity="warning"
      repeat_interval: 12h

receivers:
  - name: "discord-notifications"
    discord_configs:
      - webhook_url: "YOUR_DISCORD_WEBHOOK_URL"
        send_resolved: true

  5. Now create the docker-compose.yml for Alertmanager:

    nano docker-compose.yml

  6. Paste in the following:

    services:
  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    networks:
      - npm_default

networks:
  npm_default:
    external: true

  7. Launch it: docker compose up -d

Step 3: Configure Prometheus

Finally, tell Prometheus to send alerts to Alertmanager and load your rules.

  1. Create your rules file, ~/docker/monitoring/alert_rules.yml, with rules for “Instance Down,” “High CPU,” “Low Disk Space,” etc.

    cd ~/docker/monitoring
nano alert_rules.yml

  2. Add the alert_rules.yml as a volume in your ~/docker/monitoring/docker-compose.yml.

    volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./alert_rules.yml:/etc/prometheus/alert_rules.yml
- prometheus_data:/prometheus

  3. Add the alerting and rule_files blocks to your ~/docker/monitoring/prometheus.yml:

    groups:
  -name: Critical System Alerts
  interval: 30s
  rules:
    - alert: InstanceDown
    expr: up == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "🔴 Instance {{ $labels.instance }} is DOWN"
      description: "Service {{ $labels.job }} has been unreachable for 2 minutes."

    - alert: LaptopOnBattery
      expr: node_power_supply_online == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "🔋 Server running on BATTERY"
        description: "Homelab has been unplugged for 5 minutes. Check power connection!"

    - alert: LowBatteryLevel
      expr: node_power_supply_capacity < 20 and node_power_supply_online == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "⚠️ CRITICAL: Battery at {{ $value }}%"
        description: "Battery below 20%. Server may shut down soon!"

    - alert: DiskAlmostFull
      expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 10
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "💾 Disk space critically low: {{ $value | humanize }}% remaining"
        description: "Root filesystem has less than 10% free space."

    - alert: OutOfMemory
      expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 5
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "🧠 Memory critically low: {{ $value | humanize }}% available"
        description: "Less than 5% memory available. System may become unresponsive."

    - alert: CriticalCpuTemperature
      expr: node_hwmon_temp_celsius{chip="coretemp"} > 95
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "🔥 CRITICAL CPU Temperature: {{ $value }}°C"
        description: "CPU temperature exceeds 95°C. Thermal throttling or shutdown imminent!"

  - name: Warning System Alerts
  interval: 1m
  rules:
    - alert: HighCpuUsage
      expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "⚡ High CPU usage: {{ $value | humanize }}%"
        description: "CPU usage above 80% for 5 minutes on {{ $labels.instance }}"

    - alert: HighSystemLoad
      expr: node_load5 / on(instance) count(node_cpu_seconds_total{mode="idle"}) by (instance) > 1.5
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "📊 High system load: {{ $value | humanize }}"
        description: "5-minute load average is 1.5x CPU cores for 10 minutes."

    - alert: HighMemoryUsage
      expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 20
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "🧠 High memory usage: {{ $value | humanize }}% available"
        description: "Less than 20% memory available."

    - alert: HighCpuTemperature
      expr: node_hwmon_temp_celsius{chip="coretemp"} > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "🌡️ High CPU temperature: {{ $value }}°C"
        description: "CPU temperature above 85°C. Consider improving cooling."


    - alert: HighNvmeTemperature
      expr: node_hwmon_temp_celsius{chip="nvme"} > 65
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "💿 High NVMe temperature: {{ $value }}°C"
        description: "NVMe drive temperature above 65°C for 10 minutes."

    - alert: DiskSpaceLow
      expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100 < 20
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "💾 Disk space low: {{ $value | humanize }}% remaining"
        description: "Root filesystem has less than 20% free space."

    - alert: HighSwapUsage
      expr: ((node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes * 100) > 50
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "💱 High swap usage: {{ $value | humanize }}%"
        description: "Swap usage above 50%. System may be memory-constrained."

    # Monitor your USB-C hub ethernet adapter (enx00)
    - alert: EthernetInterfaceDown
      expr: node_network_up{device="enx00"} == 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "🌐 USB-C Ethernet adapter is DISCONNECTED"
        description: "Your USB-C hub ethernet connection (enx00) is down. Check cable or hub."

    - alert: HighNetworkErrors
      expr: rate(node_network_receive_errs_total{device="enx00"}[5m]) > 10 or rate(node_network_transmit_errs_total{device="enx00"}[5m]) > 10
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "🌐 High network errors on USB-C ethernet"
        description: "Your ethernet adapter is experiencing high error rate. Check cable quality."

  - name: Docker Container Alerts
  interval: 1m
  rules:
    # Simplified alert - just checks if container exporter is working
    - alert: ContainerMonitoringDown
      expr: absent(container_last_seen)
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "🐳 Container monitoring is down"
        description: "cAdvisor or container metrics are not available. Check if containers are being monitored."

    - alert: ContainerRestarting
      expr: rate(container_start_time_seconds[5m]) > 0.01
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "🐳 Container {{ $labels.name }} is restarting"
        description: "Container {{ $labels.name }} has restarted recently."

    - alert: ContainerHighCpu
      expr: rate(container_cpu_usage_seconds_total{name!~".*POD.*",name!=""}[5m]) * 100 > 80
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "🐳 Container {{ $labels.name }} high CPU: {{ $value | humanize }}%"
        description: "Container CPU usage above 80% for 10 minutes."

  4. Restart Prometheus to apply the changes:

    cd ~/docker/monitoring
docker compose up -d --force-recreate prometheus

    Now, if any service fails or your server’s resources run low, you will get an instant notification in Discord.

Step 3: The Critical Firewall Fix

You may find your alerts are not sending. This is often due to a conflict between Docker and ufw.

  1. Open the main ufw configuration file:

    sudo nano /etc/default/ufw

  2. Change DEFAULT_FORWARD_POLICY="DROP" to DEFAULT_FORWARD_POLICY="ACCEPT".

  3. Reload the firewall:

    sudo ufw reload

  4. Restart your containers that need internet access:

    docker compose restart

Now, if any service fails or your server’s resources run low, you will get an instant notification in Discord.

Conclusion

Our homelab has now truly come to life. It’s no longer just a collection of services but a resilient, self-maintaining platform. With automated backups to Google Drive, daily updates via Watchtower, and proactive alerts with Prometheus and Alertmanager, our server can now run 24/7 with minimal manual intervention. We’ve built a solid, reliable, and intelligent system.

But there’s one critical piece still missing: end-to-end security for our local services.

Right now, we’re accessing our dashboards at addresses like http://grafana.local, which browsers flag as “Not Secure.” What if we could use a real, public domain name for our internal services and get a valid HTTPS certificate, all without opening a single port on our router?

In the next part of this series, I’ll show you exactly how to do that. We’ll dive into an advanced but powerful setup using Cloudflare and Nginx Proxy Manager to bring trusted, zero-exposure SSL to everything we’ve built.

Stay tuned!