Pivert's Blog

This post is outdated, and will be replaced

From K8S 1.24, it’s not possible to use the deprecated Cephfs rootfs Provisioner.
The solution is to use the new ceph-csi solution for rbd, cephfs or nfs.

In this post, we will focus on CephFS mode.

The K8S is 3 nodes RKE2 cluster on Ubuntu 22.04 LTS Server; I won’t detail the installation since it’s quite straightforward.

Prerequisites

• You have admin access to a running ceph cluster (In the example below, nodes are pve1, pve2, pve3)
• You have a running Kubernetes cluster, and your .kube/config.yaml

I’ll use the provided helm chart and Seashell container. Seashell provides all the tools (kubectl, git, helm, neovim with yaml & json linters, jq, aliases, …)

Configure a new cephfs for testing

From one of the ceph cluster node. The chosen pg of 16 and 32 are very small, but it’s for testing.

root@pve1:~# ceph osd pool create cephfs_metadata_tst 16 16
pool 'cephfs_metadata_tst' created
root@pve1:~# ceph osd pool create cephfs_data_tst 32 32
pool 'cephfs_data_tst' created
root@pve1:~# ceph fs new cephfstst cephfs_data_tst cephfs_metadata_tst 
new fs with metadata pool 9 and data pool 8
root@pve1:~# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
name: cephfstst, metadata pool: cephfs_data_tst, data pools: [cephfs_metadata_tst ]
root@pve1:~# 

Generate minimal ceph.conf

From a ceph node

root@pve1:~# ceph config generate-minimal-conf
# minimal ceph.conf for e7628d51-32b5-4f5c-8eec-1cafb41ead74
[global]
        fsid = e7628d51-32b5-4f5c-8eec-1cafb41ead74
        mon_host = [v2:192.168.178.2:3300/0,v1:192.168.178.2:6789/0] [v2:192.168.178.3:3300/0,v1:192.168.178.3:6789/0] [v2:192.168.178.4:3300/0,v1:192.168.178.4:6789/0]

Get ceph.admin key

ceph auth get client.admin

Get Seashell (optional) and check your K8S access

Seashell will search for ~/.kube/config and mount the current working directory to /workdir for persisence. It contains all the tools needed for this tutorial. It’s configured for solarized dark colors. Feel free to use it or install the helm/editors/git/kubectl on your workstation.

Get ceph-csi

kubectl create namespace ceph-csi-cephfs

Create the secret (option 1)

You need paste the key from the «Get ceph.admin key» above.

read -s -p 'Please enter key from ceph auth get-key client.admin : ' KEY
kubectl --namespace ceph-csi-cephfs create secret generic csi-cephfs-secret --from-literal=adminID=admin --from-literal=adminKey=${KEY}

Create the secret (option 2)

Direct, if you have ssh access to your cephfs cluster

KEY=$(ssh root@pve1 'ceph auth get client.admin -f json' | jq -r '.[0].key')
kubectl --namespace ceph-csi-cephfs create secret generic csi-cephfs-secret --from-literal=adminID=admin --from-literal=adminKey=${KEY}

Checks & ceph-config configmap

kubectl --namespace ceph-csi-cephfs describe secret csi-cephfs-secret
kubectl --namespace ceph-csi-cephfs create configmap ceph-config --from-file=ceph.conf=./ceph-minimal.conf

Helm

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm show values ceph-csi/ceph-csi-cephfs > defaultValues.yaml
cp defaultValues.yaml values.yaml
vim values.yaml

The helm values.yaml

You need to update values.yaml with your values, example:

---
rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccounts:
  nodeplugin:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the fullname
    name:
  provisioner:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the fullname
    name:

csiConfig:
  - clusterID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    monitors:
      - 192.168.178.2
      - 192.168.178.3
      - 192.168.178.4
    cephFS:
      subvolumeGroup: "csi"
      netNamespaceFilePath: "{{ .kubeletDir }}/plugins/{{ .driverName }}/net"

# Labels to apply to all resources
commonLabels: {}

# Set logging level for csi containers.
# Supported values from 0 to 5. 0 for general useful logs,
# 5 for trace level verbosity.
# logLevel is the variable for CSI driver containers's log level
logLevel: 5
# sidecarLogLevel is the variable for Kubernetes sidecar container's log level
sidecarLogLevel: 1

nodeplugin:
  name: nodeplugin
  # if you are using ceph-fuse client set this value to OnDelete
  updateStrategy: RollingUpdate

  # set user created priorityclassName for csi plugin pods. default is
  # system-node-critical which is highest priority
  priorityClassName: system-node-critical

  httpMetrics:
    # Metrics only available for cephcsi/cephcsi => 1.2.0
    # Specifies whether http metrics should be exposed
    enabled: true
    # The port of the container to expose the metrics
    containerPort: 8081

    service:
      # Specifies whether a service should be created for the metrics
      enabled: true
      # The port to use for the service
      servicePort: 8080
      type: ClusterIP

      # Annotations for the service
      # Example:
      # annotations:
      #   prometheus.io/scrape: "true"
      #   prometheus.io/port: "9080"
      annotations: {}

      clusterIP: ""

      ## List of IP addresses at which the stats-exporter service is available
      ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
      ##
      externalIPs: []

      loadBalancerIP: ""
      loadBalancerSourceRanges: []

  profiling:
    enabled: false

  registrar:
    image:
      repository: registry.k8s.io/sig-storage/csi-node-driver-registrar
      tag: v2.6.2
      pullPolicy: IfNotPresent
    resources: {}

  plugin:
    image:
      repository: quay.io/cephcsi/cephcsi
      tag: v3.8.0
      pullPolicy: IfNotPresent
    resources: {}

  nodeSelector: {}

  tolerations: []

  affinity: {}

  # Set to true to enable Ceph Kernel clients
  # on kernel < 4.17 which support quotas
  # forcecephkernelclient: true

  # common mount options to apply all mounting
  # example: kernelmountoptions: "recover_session=clean"
  kernelmountoptions: ""
  fusemountoptions: ""

provisioner:
  name: provisioner
  replicaCount: 3
  strategy:
    # RollingUpdate strategy replaces old pods with new ones gradually,
    # without incurring downtime.
    type: RollingUpdate
    rollingUpdate:
      # maxUnavailable is the maximum number of pods that can be
      # unavailable during the update process.
      maxUnavailable: 50%
  # Timeout for waiting for creation or deletion of a volume
  timeout: 60s
  # cluster name to set on the subvolume
  # clustername: "k8s-cluster-1"

  # set user created priorityclassName for csi provisioner pods. default is
  # system-cluster-critical which is less priority than system-node-critical
  priorityClassName: system-cluster-critical

  # enable hostnetwork for provisioner pod. default is false
  # useful for deployments where the podNetwork has no access to ceph
  enableHostNetwork: false

  httpMetrics:
    # Metrics only available for cephcsi/cephcsi => 1.2.0
    # Specifies whether http metrics should be exposed
    enabled: true
    # The port of the container to expose the metrics
    containerPort: 8081

    service:
      # Specifies whether a service should be created for the metrics
      enabled: true
      # The port to use for the service
      servicePort: 8080
      type: ClusterIP

      # Annotations for the service
      # Example:
      # annotations:
      #   prometheus.io/scrape: "true"
      #   prometheus.io/port: "9080"
      annotations: {}

      clusterIP: ""

      ## List of IP addresses at which the stats-exporter service is available
      ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
      ##
      externalIPs: []

      loadBalancerIP: ""
      loadBalancerSourceRanges: []

  profiling:
    enabled: false

  provisioner:
    image:
      repository: registry.k8s.io/sig-storage/csi-provisioner
      tag: v3.3.0
      pullPolicy: IfNotPresent
    resources: {}
    ## For further options, check
    ## https://github.com/kubernetes-csi/external-provisioner#command-line-options
    extraArgs: []

  # set metadata on volume
  setmetadata: true

  resizer:
    name: resizer
    enabled: true
    image:
      repository: registry.k8s.io/sig-storage/csi-resizer
      tag: v1.6.0
      pullPolicy: IfNotPresent
    resources: {}
    ## For further options, check
    ## https://github.com/kubernetes-csi/external-resizer#recommended-optional-arguments
    extraArgs: []

  snapshotter:
    image:
      repository: registry.k8s.io/sig-storage/csi-snapshotter
      tag: v6.1.0
      pullPolicy: IfNotPresent
    resources: {}
    ## For further options, check
    ## https://github.com/kubernetes-csi/external-snapshotter#csi-external-snapshotter-sidecar-command-line-options
    extraArgs: []

  nodeSelector: {}

  tolerations: []

  affinity: {}

# Mount the host /etc/selinux inside pods to support
# selinux-enabled filesystems
selinuxMount: true

storageClass:
  # Specifies whether the Storage class should be created
  create: true
  name: csi-cephfs-sc
  # Annotations for the storage class
  # Example:
  # annotations:
  #   storageclass.kubernetes.io/is-default-class: "true"
  annotations: {}

  # String representing a Ceph cluster to provision storage from.
  # Should be unique across all Ceph clusters in use for provisioning,
  # cannot be greater than 36 bytes in length, and should remain immutable for
  # the lifetime of the StorageClass in use.
  clusterID: "e7628d51-32b5-4f5c-8eec-1cafb41ead74"
  # (required) CephFS filesystem name into which the volume shall be created
  # eg: fsName: myfs
  fsName: cephfstst
  # (optional) Ceph pool into which volume data shall be stored
  # pool: <cephfs-data-pool>
  # For eg:
  # pool: "replicapool"
  pool: ""
  # (optional) Comma separated string of Ceph-fuse mount options.
  # For eg:
  # fuseMountOptions: debug
  fuseMountOptions: ""
  # (optional) Comma separated string of Cephfs kernel mount options.
  # Check man mount.ceph for mount options. For eg:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes
  kernelMountOptions: ""
  # (optional) The driver can use either ceph-fuse (fuse) or
  # ceph kernelclient (kernel).
  # If omitted, default volume mounter will be used - this is
  # determined by probing for ceph-fuse and mount.ceph
  # mounter: kernel
  mounter: ""
  # (optional) Prefix to use for naming subvolumes.
  # If omitted, defaults to "csi-vol-".
  # volumeNamePrefix: "foo-bar-"
  volumeNamePrefix: ""
  # The secrets have to contain user and/or Ceph admin credentials.
  provisionerSecret: csi-cephfs-secret
  # If the Namespaces are not specified, the secrets are assumed to
  # be in the Release namespace.
  provisionerSecretNamespace: ""
  controllerExpandSecret: csi-cephfs-secret
  controllerExpandSecretNamespace: ""
  nodeStageSecret: csi-cephfs-secret
  nodeStageSecretNamespace: ""
  reclaimPolicy: Delete
  allowVolumeExpansion: true
  mountOptions: []
  # Mount Options
  # Example:
  # mountOptions:
  #   - discard

secret:
  # Specifies whether the secret should be created
  create: false
  name: csi-cephfs-secret
  # Key values correspond to a user name and its key, as defined in the
  # ceph cluster. User ID should have required access to the 'pool'
  # specified in the storage class
  adminID: <plaintext ID>
  adminKey: <Ceph auth key corresponding to ID above>

cephconf: |
  [global]
           auth_client_required = cephx
           auth_cluster_required = cephx
           auth_service_required = cephx
           cluster_network = 192.168.178.2/24
           fsid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
           mon_allow_pool_delete = true
           mon_host = 192.168.178.2 192.168.178.4 192.168.178.3
           ms_bind_ipv4 = true
           ms_bind_ipv6 = false
           osd_pool_default_min_size = 2
           osd_pool_default_size = 3
           public_network = 192.168.178.2/24
           fuse_set_user_groups = false
           fuse_big_writes = true

  [mds.pve1]
           host = pve1
           mds_standby_for_name = pve

  [mds.pve2]
           host = pve2
           mds_standby_for_name = pve

  [mds.pve3]
           host = pve3
           mds_standby_for_name = pve

  [mon.pve1]
           public_addr = 192.168.178.2

  [mon.pve2]
           public_addr = 192.168.178.3

  [mon.pve3]
           public_addr = 192.168.178.4

#########################################################
# Variables for 'internal' use please use with caution! #
#########################################################

# The filename of the provisioner socket
provisionerSocketFile: csi-provisioner.sock
# The filename of the plugin socket
pluginSocketFile: csi.sock
# kubelet working directory,can be set using `--root-dir` when starting kubelet.
kubeletDir: /var/lib/kubelet
# Name of the csi-driver
driverName: cephfs.csi.ceph.com
# Name of the configmap used for state
configMapName: ceph-csi-config
# Key to use in the Configmap if not config.json
# configMapKey:
# Use an externally provided configmap
externallyManagedConfigmap: false
# Name of the configmap used for ceph.conf
cephConfConfigMapName: ceph-config

Install

helm install --namespace "ceph-csi-cephfs" "ceph-csi-cephfs" ceph-csi/ceph-csi-cephfs --values ./values.yaml