Multi tenancy in Kubernetes cluster

Photo by Abraham Barrera on Unsplash

1. Introduction

Multi tenancy on k8s cluster can be achieved using Namespace and RBAC approach. A multi-tenant cluster is shared by multiple users and/or workloads which are referred to as “tenants”. The operators of multi-tenant clusters must isolate tenants from each other to minimize the damage that a compromised or malicious tenant can do to the cluster and other tenants. Also, cluster resources must be fairly allocated among tenants. Multi-tenancy design considers layers of resource isolation in Kubernetes: cluster, namespace, node, pod, and container. One can separate each tenant and their Kubernetes resources into their own namespaces. We can then use policies to enforce tenant isolation. Policies are usually scoped by namespace and can be used to restrict API access, to constrain resource usage, and to restrict what containers are allowed to do.

The following diagram depicts multi-tenancy in k8s cluster.

For information on setting up multiple multi-tenant clusters for an enterprise organization, see Best practices for enterprise multi- tenancy.

2. How to create Custom RBAC and Kubeconfig for Tenant

In order to setup RBAC per tenant, we need to create namespace per tenant and create service account, role and role binding for the tenant. Eventually , we need to create kubeconfig file to access the tenant k8s resources.

2.1 Create Service account

apiVersion: v1
kind: ServiceAccount
metadata:
name: <tenant-namespace>-user
namespace: <tenant-namespace>

2.2 Create Role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: <tenant-namespace>-user-full-access
namespace: <tenant-namespace>
rules:
- apiGroups: [“”, “extensions”, “apps”]
resources: [“*”]
verbs: [“*”]
- apiGroups: [“batch”]
resources:
— jobs
— cronjobs
verbs: [“*”]

2.3 Create Role binding

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: <tenant-namespace>-user-view
namespace: <tenant-namespace>
subjects:
- kind: ServiceAccount
name: <tenant-namespace>-user
namespace: <tenant-namespace>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: <tenant-namespace>-user-full-access

2.4 Create kubeconfig

The script below creates kubeconfig for the tenant. This script takes service account name and namespace created for the tenant.

The script is also available here:

https://github.com/ykumar-rb/POC/blob/master/multi-tenanat-k8s-access/create_k3s_config_with_rbac.sh

#!/bin/bash
set -e
set -o pipefail

# Add user to k8s using service account, no RBAC (must create RBAC after this script)
if [[ -z “$1” ]] || [[ -z “$2” ]]; then
echo “usage: $0 <service_account_name> <namespace>”
exit 1
fi

SERVICE_ACCOUNT_NAME=$1
NAMESPACE=”$2"
KUBECFG_FILE_NAME=”/tmp/kube/k8s-${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-conf”
TARGET_FOLDER=”/tmp/kube”

create_target_folder() {
echo -n “Creating target directory to hold files in ${TARGET_FOLDER}…”
mkdir -p “${TARGET_FOLDER}”
printf “done”
}

create_service_account() {
echo -e “\\nCreating a service account in ${NAMESPACE} namespace: ${SERVICE_ACCOUNT_NAME}”
kubectl create sa “${SERVICE_ACCOUNT_NAME}” — namespace “${NAMESPACE}”
}

get_secret_name_from_service_account() {
echo -e “\\nGetting secret of service account ${SERVICE_ACCOUNT_NAME} on ${NAMESPACE}”
SECRET_NAME=$(kubectl get sa “${SERVICE_ACCOUNT_NAME}” — namespace=”${NAMESPACE}” -o json | jq -r .secrets[].name)
echo “Secret name: ${SECRET_NAME}”
}

extract_ca_crt_from_secret() {
echo -e -n “\\nExtracting ca.crt from secret…”
kubectl get secret — namespace “${NAMESPACE}” “${SECRET_NAME}” -o json | jq \
-r ‘.data[“ca.crt”]’ | base64 -d > “${TARGET_FOLDER}/ca.crt”
printf “done”
}

get_user_token_from_secret() {
echo -e -n “\\nGetting user token from secret…”
USER_TOKEN=$(kubectl get secret — namespace “${NAMESPACE}” “${SECRET_NAME}” -o json | jq -r ‘.data[“token”]’ | base64 -d)
printf “done”
}

set_kube_config_values() {
context=$(kubectl config current-context)
echo -e “\\nSetting current context to: $context”

CLUSTER_NAME=$(kubectl config get-contexts “$context” | awk ‘{print $3}’ | tail -n 1)
echo “Cluster name: ${CLUSTER_NAME}”

ENDPOINT=$(kubectl config view \
-o jsonpath=”{.clusters[?(@.name == \”${CLUSTER_NAME}\”)].cluster.server}”)
echo “Endpoint: ${ENDPOINT}”

# Set up the config
echo -e “\\nPreparing k8s-${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-conf”
echo -n “Setting a cluster entry in kubeconfig…”
kubectl config set-cluster “${CLUSTER_NAME}” \
— kubeconfig=”${KUBECFG_FILE_NAME}” \
— server=”${ENDPOINT}” \
— certificate-authority=”${TARGET_FOLDER}/ca.crt” \
— embed-certs=true

echo -n “Setting token credentials entry in kubeconfig…”
kubectl config set-credentials \
“${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}” \
— kubeconfig=”${KUBECFG_FILE_NAME}” \
— token=”${USER_TOKEN}”

echo -n “Setting a context entry in kubeconfig…”
kubectl config set-context \
“${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}” \
— kubeconfig=”${KUBECFG_FILE_NAME}” \
— cluster=”${CLUSTER_NAME}” \
— user=”${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}” \
— namespace=”${NAMESPACE}”

echo -n “Setting the current-context in the kubeconfig file…”
kubectl config use-context “${SERVICE_ACCOUNT_NAME}-${NAMESPACE}-${CLUSTER_NAME}” \
— kubeconfig=”${KUBECFG_FILE_NAME}”
}

create_target_folder
get_secret_name_from_service_account
extract_ca_crt_from_secret
get_user_token_from_secret
set_kube_config_values

echo -e “\\nAll done! kubeconfig is created for tenant”
echo “KUBECONFIG=${KUBECFG_FILE_NAME} kubectl get pods”
echo “Now we have permissions by default — you have just created the authentication part”
echo “Create RBAC permissions if not created.”
KUBECONFIG=${KUBECFG_FILE_NAME} kubectl get pods

3. Demonstrating Multi-tenancy on k8s cluster

The pre-requisites for this demo is k8s environment. Setup kubernetes cluster following the below link.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Demonstrated using two tenants “zero-tenant” and “spike-tenant”. Created “hello-node” pod in respective tenant namespace.

Each tenant uses it’s own “kubeconfig” to do kubectl operations. It can only operate on resources running in own namespace ( restricted / managed via service-account).

Executing multi-tenancy demo