GlusterFS

GlusterFS with heketi provides persistent storage for your k8s cluster.

GlusterFS is a distributed cluster-ready storage backend. We’re going to use it together with heketi – REST wrapper on top of Gluster.

Please refer for official documentation for more details.

Install required packages:

If you’re not planning to run Gluster on every node, skip glusterfs-server for them.

sudo apt-get install -y xfsprogs glusterfs-server glusterfs-client lvm2 thin-provisioning-tools

In this scenario 3 worker nodes are used as Gluster backend. You can use master as well, it doesn’t matter. First worker then used as heketi server.

From first RPi do:

sudo gluster peer probe 192.168.0.101
sudo gluster peer probe 192.168.0.102
sudo gluster peer probe 192.168.0.103

At this point you can either choose to use separate storage for GlusterFS or create a loop device.

WARN Instructions below are applicable to separate storage, for the loop device you'll have different paths.

Get your device name:

pi@k8s-agent-1:~ $ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    1 29.9G  0 disk                         <- raspbian here
|-sda1   8:1    1 43.2M  0 part /boot
`-sda2   8:2    1 29.9G  0 part /
sdb      8:16   1 28.9G  0 disk                         <- second attached USB
`-sdb1   8:17   1 28.9G  0 part

Format you cards on every machine:

sudo wipefs -a /dev/sdb1

We need to remove existing glusterfs-server to glusterd since heketi is using a new name and debian pckg is not updated yet:

sudo /etc/init.d/glusterfs-server stop
sudo mv /etc/init.d/glusterfs-server /etc/init.d/glusterd
sudo /etc/init.d/glusterd start

Create new /etc/init.d/glusterfs-server:

#! /bin/sh

/etc/init.d/glusterd $@

Make it executable:

sudo chmod +x /etc/init.d/glusterfs-server

Install heketi:

wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-v7.0.0.linux.arm.tar.gz
sudo mkdir -p /etc/heketi
sudo tar xzvf heketi-v7.0.0.linux.arm.tar.gz -C /etc/heketi
rm heketi-v7.0.0.linux.arm.tar.gz
sudo ln /etc/heketi/heketi/heketi-cli /usr/bin/heketi-cli
sudo ln /etc/heketi/heketi/heketi /usr/bin/heketi

Generate keys:

sudo ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
sudo ssh-copy-id -i /etc/heketi/heketi_key.pub pi@192.168.0.101
sudo ssh-copy-id -i /etc/heketi/heketi_key.pub pi@192.168.0.102
sudo ssh-copy-id -i /etc/heketi/heketi_key.pub pi@192.168.0.103

Create service definition:

[Unit]
Description=Heketi Server

[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.env
User=heketi
ExecStart=sudo /usr/bin/heketi --config=/etc/heketi/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

Start service:

sudo chown -R pi:pi /etc/heketi/heketi_key* /var/lib/heketi
sudo systemctl daemon-reload
sudo systemctl start heketi

Patch access in /etc/heketi/heketi/heketi.json:

...
    executor": "ssh",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "pi",
      "sudo": true,
      "port": "22",
      "fstab": "/etc/fstab",
      "backup_lvm_metadata": false
    },

Create your topology.json:

{
    "clusters": [{
        "nodes": [{
                "node": {
                    "hostnames": {
                        "manage": [
                            "192.168.0.101"
                        ],
                        "storage": [
                            "192.168.0.101"
                        ]
                    },
                    "zone": 1
                },
                "devices": [
                    "/dev/sdb1"
                ]
            },
            {
                "node": {
                    "hostnames": {
                        "manage": [
                            "192.168.0.102"
                        ],
                        "storage": [
                            "192.168.0.102"
                        ]
                    },
                    "zone": 1
                },
                "devices": [
                    "/dev/sdb1"
                ]
            },
            {
                "node": {
                    "hostnames": {
                        "manage": [
                            "192.168.0.103"
                        ],
                        "storage": [
                            "192.168.0.103"
                        ]
                    },
                    "zone": 1
                },
                "devices": [
                    "/dev/sdb1"
                ]
            }
        ]
    }]
}

And start add to the heketi:

heketi-cli topology load --json=topology.json

Last thing is to create heketi StorageClass:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://192.168.0.101:8080"
kubectl create -f class.yaml && kubectl patch storageclass gluster -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Now cluster is ready to provision PVCs.

If you want to have access to the Heketi status without SSH-ing into the nodes, you can install heketi-ui app.