Self-Hosted Infrastructure from Homelab to Company Part 1

Choosing hardware for a server

Introduction

This series will begin by identifying the appropriate hardware components for the server and assembling them. Then, set up the operating system and install and configure the desired services. Additionally, it will cover aspects related to security, data integrity, and data backup.

To choose the proper components, the first thing to consider is whether you’ll be using the server in a homelab setting or an enterprise one. The second is to determine the server’s primary purpose. Typical Uses for servers include virtualization, data storage (NAS), multimedia/streaming, email servers, web servers, game servers, etc. The third important thing to consider is how many users the server will serve; the more users, the more resources the server will require.

Homelab

In a homelab setting, you’ll generally use consumer-grade hardware, such as desktop components or older server-grade parts from a few generations back. In addition, most homelabs won’t use ECC memory, a feature not found on most consumer-grade CPUs and Motherboards.

Enterprise / Business

Enterprise and business servers are usually purchased preconfigured rather than building up the hardware from components. In cases where custom server configurations are needed, they’ll generally use workstation or server-grade motherboards and CPUs with support for ECC memory.

Processor

When choosing a processor, a few things to consider are the number of cores and threads, cache size, the clock speed of each CPU core, and virtualization support.

Cores and threads

The cores are the individual physical processing units within a CPU. The more compute units, the more simultaneous tasks the CPU can perform. Threads are sometimes called virtual CPUs in chips with hyperthreading support, such as the Intel Core i9-9900K. Threading, like the CPU cores, allows for better handling of more parallel tasks.

Clock speed

The clock speed of a CPU indicates how fast a single core can perform a task. Typically, there is base clock speed, which is how fast the CPU will run by default, and boost clock speed, which is the maximum speed the CPU can run.

Cache size

The cache is fast-access memory stored in the processor; typically, there are L1, L2, and L3 caches. As the cache number increases, so does the cache size; however, the memory speed decreases.

Virtualization support

Some processors support virtualization extensions, allowing multiple operating systems to run simultaneously. Most modern CPUs, both consumer-grade and server-grade, offer virtualization extensions. Intel has VT-x, and AMD has AMD-vi. They both have extensions allowing PCIe passthrough VT-d for Intel and IOMMU for AMD. You’ll need both CPU extensions to run virtual machines with fully accelerated graphics.

Memory

It is generally better to get error-correcting code (ECC) memory if possible, even if the data hosted on the server is not critical; however, many Homelab setups don’t use it due to the higher cost of components to support it. ZFS and other fault-tolerant technologies can help avoid and correct potential data corruption without ECC memory; however, it’s still better to have it for critical systems.

The memory needed will heavily depend on what services will be running, how many users will use the system, the storage configuration, and whether or not the server will employ virtualization. I recommend at least 64GiB for general use and 128GiB to 256GiB for ZFS and virtualization.

Storage

When choosing storage, two things to consider are how much is needed and how fast it needs to be. Standard SATA or SAS hard drives generally offer high storage capacity at a lower cost but are also much slower than SATA or NVMe SSDs. You can combine multiple drives of different types using ZFS or software RAID to offer better reliability, redundancy, and speed. As seen in the figure below, one such configuration uses two mirrors, each with two drives, a read cache (l2arc), and a write cache (slog). This configuration offers some speed up on cached reads and reliability in case of power failure because of the write cache.

flowchart TD
    zpool
    l2arc
    slog
    mirror0
    mirror1
    sda[(18TiB HDD)]
    sdb[(18TiB HDD)]
    sdc[(18TiB HDD)]
    sdd[(18TiB HDD)]
    nvme0[128GiB NVMe]
    nvme1[32GiB NVMe]
    zpool --> mirror0
    zpool --> mirror1
    zpool --> slog
    zpool --> l2arc
    mirror0 --> sda
    mirror0 --> sdb
    mirror1 --> sdc
    mirror1 --> sdd
    slog --> nvme1
    l2arc --> nvme0

A few helpful things about storage configurations

• For better performance under ZFS, 1GiB of memory per 1TB of usable disk space is good.
• For Hard drives larger than 2TiB, RAID1 (mirrors) or RAID10 is ideal due to rebuild time.
• For SSDs smaller than 2TiB, RAID5 or RAID6 are better due to the space available.
• SSDs in a mirror configuration will wear at about the same rate and likely fail around the same time.

Graphics

Most servers are administrated via the command line and don’t require a graphics card to host a desktop environment. Typical uses for a graphics card in a server include

• Transcoding for media servers
• Supplying accelerated graphics to docker containers for 3D applications
• Running virtual machines with accelerated graphics For general use cases, a single consumer-grade or workstation-grade GPU will work fine; however, in cases where the number of virtual machines needing GPUs exceeds the number of PCIe slots on the server. Then, you’ll need vGPU support, which allows a single GPU to be partitioned into many virtual GPUs and individually assigned to virtual machines or docker containers.

Case

Choosing a case will depend on two main things: first, how big it needs to be to fit all the components correctly, and second, where you intend to store it in a typical home without a rack, you’ll likely use a tower or desktop-style case; however, if you can rack mount your server, it’s generally the preferred choice.

My builds

I’ve worked on several servers which have been used both in companies as well as home settings. Most of the servers I’ve built are more general use as they can cover most of the purposes mentioned in the introduction. Below, I’ve listed the components used for a few.

This post contains affiliate links, which means I may earn a commission if you purchase through these links; this does not increase the amount you pay for the items.

AMD Epyc server

AMD Ryzen server

Intel Core server

Intel Xeon server