Kubernetes The Hard Way using Terraform, Ansible and Proxmox

Introduction

The scripts are available in https://github.com/rinoymjoseph/k8s-hardway repo.

Kubernetes The Hard Way is a popular git hub repo which take the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.

1. Prerequisites

Proxmox server.
An Ubuntu 22.04 template provisioned on Proxmox.
Terraform. Please refer Terraform installation on Ubuntu 22.04 for installation of Terraform.
A way to provision static ip addresses mapped to mac addresses (I used a home router).

2. System Diagram

Single Controller and Three Worker Nodes

3. Configuration

Infrastructure Configuration

A terraform script creates the machines required for the setup. The script is available in terraform/five-nodes path. Script is using Telmate/proxmox provider to create virtual machines in Proxmox server from an Ubuntu 22.04 machine template. So make sure that an Ubuntu 22.04 server template is available in Proxmox server.

System configuration for individual nodes

Bootstrap, CPU – 1 core, RAM – 2 GB
k8s-controller-1, CPU – 2 core, RAM – 4 GB
k8s-worker-1, CPU – 2 core, RAM – 4 GB
k8s-worker-2, CPU – 2 core, RAM – 4 GB
k8s-worker-3, CPU – 2 core, RAM – 4 GB

Proxmox settings

Change proxmox settings in below code block of k8s-infra.tf

provider "proxmox" {
  # Configuration options
  pm_api_url    =   "https://192.168.0.111:8006/api2/json"
  pm_user       =   "root@pam"
  pm_password   =   ""
}

Change the parameter values for each node in the tf file such as name, vmid, macaddr etc. In this setup a static IP is provisioned when VMs are created. The mac addresses specified for each VM is mapped to an IP address in the router.

# Provision bootstrap
resource "proxmox_vm_qemu" "k8s-bootstrap" {
  name          = "k8s-bootstrap"
  vmid          = 149
  target_node   = "proxmox"
  clone         = "Ubuntu-Server-22.04"
  full_clone    = false
  memory        = 2048
  cores         = 1
  network   {
    model       = "virtio"
    macaddr     = "86:AC:41:71:70:7D"
    bridge      = "vmbr0"
    firewall    = true
  }
}

Ansible hosts.ini configuration

localhost ansible_host=127.0.0.1 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"

[bootstrap]
k8s-bootstrap ansible_host=192.168.0.149 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"

[load_balancer]
load_balancer ansible_host=192.168.0.151 ansible_user=rinoy ansible_ssh_pass="welcome123#"

[controllers]
k8s-controller-1 ansible_host=192.168.0.151 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"

[workers]
k8s-worker-1 ansible_host=192.168.0.161 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"
k8s-worker-2 ansible_host=192.168.0.162 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"
k8s-worker-3 ansible_host=192.168.0.163 ansible_user=rinoy ansible_ssh_pass="welcome123#" ansible_sudo_pass="welcome123#"

[cluster:children]
controllers
workers

[nodes:children]
bootstrap
controllers
workers

The package configuration is available in ansible_scripts/inventory/group_vars/all.yml file

k8s_version: v1.23.0
cfssl_version: 1.6.3
etcd_version: v3.5.1
cni_plugins_version: v1.0.1
crio_version: 1.23
crio_os: xUbuntu_22.04

Certificate settings is also available in all.yml file

# Cerfificate settings
TLS_C: "HY"
TLS_L: "Hyrule"
TLS_OU: "Link"
TLS_ST: "Akkala"

4. Provisioning

Terraform script execution

link@hyrule:~$ git clone https://github.com/rinoymjoseph/k8s-hardway.git
Cloning into 'k8s-hardway'...
remote: Enumerating objects: 550, done.
remote: Counting objects: 100% (550/550), done.
remote: Compressing objects: 100% (296/296), done.
remote: Total 550 (delta 184), reused 492 (delta 129), pack-reused 0
Receiving objects: 100% (550/550), 159.36 KiB | 2.12 MiB/s, done.
Resolving deltas: 100% (184/184), done.
link@hyrule:~$ cd k8s-hardway/
link@hyrule:~/k8s-hardway$ ls
ansible_scripts  ansible_scripts.zip  docs  k8s-infra.md  LICENSE  README.md  terraform  tests  tests.md  two-nodes

Zip ansible_scripts folder using below command

zip -r ansible_scripts.zip ansible_scripts/

cd to terraform/five-nodes folder

terraform init

Terminal output for reference

link@hyrule:~/k8s-hardway/terraform/five-nodes$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching "2.9.14"...
- Finding latest version of hashicorp/null...
- Installing telmate/proxmox v2.9.14...
- Installed telmate/proxmox v2.9.14 (self-signed, key ID A9EBBE091B35AFCE)
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Validate the tf file

terraform validate