Behind the Scenes: How We Automated Global Server Deployment

Deploying servers across multiple continents presents a unique challenge: how do you ensure consistent configuration while minimizing the burden on local IT teams? Here's a look at how we solved this challenge at Hope Channel International (HCI) – and why it matters for our mission to share eternal hope with the world.

The Challenge

Imagine coordinating server deployments to locations as diverse as a university in Australia, a hospital in Kenya, and an administrative office in Germany. Each location has different IT staff, varying technical expertise, and unique local requirements – not to mention that they are doing us a big favor in volunteering to host a server for us! Traditional approaches would require significant coordination, documentation, and technical support across multiple time zones.

With a network of 80+ channels sharing the message of hope in over 100 languages, HCI needed something different: a system that would let us handle all the complexities before we ship the servers, so that global IT teams could simply plug in a server and have it work.

Our Solution: Pre-Flight Provisioning

Instead of shipping blank servers and providing installation instructions, we completely configure each server before it ships. Here's how we do it:

When a new server arrives at HCI, we connect it to our provisioning network.
Our PXE server handles the initial operating system installation.
Ansible automation configures everything else.
We test the complete configuration.
Finally, we ship the server to its destination.

Let's look at how each piece works.

The DHCP/PXE Boot Process

When we connect a new server to our provisioning network, a carefully choreographed sequence begins:

DHCP Discovery: The server broadcasts a DHCP discover message, including its MAC address and a flag indicating it supports PXE booting.
DHCP Response: Our DHCP server responds with:
TFTP Boot File Transfer: The server downloads its initial boot files from our TFTP server, including the UEFI boot loader, GRUB configuration, and the Linux kernel.

PXE Boot Infrastructure

Once the initial boot files are transferred, our PXE infrastructure takes over. We've configured GRUB to present a customized menu that lets us select the exact role this server will play in our network. Here's how we define these options:

This configuration tells the installer exactly what type of server we're building and where to find its specific configuration files. The intel_iommu parameters ensure proper hardware support for features we'll need later, like GPU passthrough for video transcoding.

The two file URLs are hosted on the same PXE server, but with a simple HTTP server instead of TFTP.

And here’s what it looks like:

Cloud-Init: The Foundation

Cloud-init handles the initial system configuration, setting up:

Mirrored OS drives for reliability
Network bonding for redundancy
Basic security hardening
Initial access credentials

This automation ensures consistent storage configuration across all our servers, optimized for their specific roles in our content delivery network.

Ansible: The Final Touch

After the base system is ready, our Ansible playbooks handle the rest:

Installing required packages
Configuring ZFS storage arrays
Setting up monitoring
Deploying security configurations
Installing and configuring NetBird for our private overlay network

Every configuration detail, from optimal ZFS parameters for video streaming to security hardening, is automated and verified before the server leaves our facility.

Simplified On-Site Installation

When a server arrives at its destination, local IT staff receive a server that's fully configured and ready to serve Hope Channel’s life-changing content.

The on-site installation process is remarkably simple:

Mount the server in the rack.
Connect power.
Connect network cable.
Power on.

That's it. No complex configuration, no waiting for remote support, no troubleshooting required. This simplicity ensures that our mission to deliver the gospel remains unhindered by technical challenges.

QR Code Server Portal

Each server ships with a QR code, which provides instant access to our custom server portal with simple installation steps, a real-time server health dashboard, and emergency contact information.

This means local IT staff have everything they need at their fingertips, without waiting for us to be available.

Looking Forward

As we continue expanding our global network, this pre-flight provisioning approach becomes increasingly valuable.

By handling all technical complexity before we ship servers, we've created a deployment process that lets us focus on what matters most: ensuring Hope Channel's message reaches more people, more reliably, in more places around the world.