CD2H gitForager

ceph/ceph-csi

Name: ceph-csi

Owner: Ceph

Description: CSI driver for ceph

Created: 2018-01-08 15:21:11.0

Updated: 2018-05-23 17:19:57.0

Pushed: 2018-05-23 17:19:55.0

Homepage: null

Size: 52447

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Ceph CSI

Overview

Ceph CSI plugins implement an interface between CSI enabled Container Orchestrator and CEPH cluster. It allows dynamically provision CEPH volumes and attach it to workloads. Current implementation of Ceph CSI plugins was tested in Kubernetes environment (requires Kubernetes 1.10+), but the code does not rely on any Kubernetes specific calls (WIP to make it k8s agnostic) and should be able to run with any CSI enabled CO (Containers Orchestration).

Container Storage Interface (CSI) driver, provisioner, and attacher for Ceph RBD and CephFS

RBD Plugin

An RBD CSI plugin is available to help simplify storage management. Once user creates PVC with the reference to a RBD storage class, rbd image and corresponding PV object gets dynamically created and becomes ready to be used by workloads.

Configuration Requirements

Secret object with the authentication key for ceph cluster
StorageClass with rbdplugin (default CSI RBD plugin name) as a provisioner name and information about ceph cluster (monitors, pool, etc)
Service Accounts with required RBAC permissions

Feature Status

1.9: Alpha

Important: CSIPersistentVolume and MountPropagation feature gates must be enabled starting in 1.9. Also API server must run with running config set to: storage.k8s.io/v1alpha1

Compiling

CSI RBD plugin can be compiled in a form of a binary file or in a form of a container. When compiled as a binary file, it gets stored in _output folder with the name rbdplugin. When compiled as a container, the resulting image is stored in a local docker's image store.

To compile just a binary file:

ke rbdplugin

To build a container:

ke rbdplugin-container

By running:

cker images | grep rbdplugin

You should see the following line in the output:

.io/cephcsi/rbdplugin              v0.2.0                            76369a8f8528        15 minutes ago      372.5 MB

Testing

Prerequisite Enable Mount Propagation in Docker

Comment out MountFlags=slave in docker systemd service then restart docker service.

stemctl daemon-reload
stemctl restart docker

Enable Kubernetes Feature Gates

Enable features MountPropagation=true,CSIPersistentVolume=true and runtime config storage.k8s.io/v1alpha1=true

Step 1: Create Secret

bectl create -f ./deploy/rbd/kubernetes/rbd-secrets.yaml

Important: rbd-secrets.yaml, must be customized to match your ceph environment.

Step 2: Create StorageClass

bectl create -f ./deploy/rbd/kubernetes/rbd-storage-class.yaml

Important: rbd-storage-class.yaml, must be customized to match your ceph environment.

Step 3: Start CSI CEPH RBD plugin

bectl create -f ./deploy/rbd/kubernetes/rbdplugin.yaml

Step 4: Start CSI External Attacher

bectl create -f ./deploy/rbd/kubernetes/csi-attacher.yaml

Step 5: Start CSI External Provisioner

bectl create -f ./deploy/rbd/kubernetes/csi-provisioner.yaml

Important: Deployment yaml files includes required Service Account definitions and required RBAC rules.

Step 6: Check status of CSI RBD plugin

bectl get pods | grep csi

The following output should be displayed:

SPACE     NAME                                                READY     STATUS    RESTARTS   AGE          
ult       csi-attacher-0                                      1/1       Running   0          1d           
ult       csi-rbdplugin-qxqtl                                 2/2       Running   0          1d           
ult       csi-provisioner-0                                   1/1       Running   0          1d

Step 7: Create PVC

bectl create -f ./deploy/rbd/kubernetes/pvc.yaml

Step 8: Check status of provisioner PV

bectl get pv

The following output should be displayed:

                                                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
rnetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea   5Gi        RWO            Delete           Bound     default/csi-pvc   rbdv2                    10s


bectl describe pv kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
:            kubernetes-dynamic-pvc-1b19ddf1-0047-11e8-85ab-760f2eed12ea
tations:     csi.volume.kubernetes.io/volume-attributes={"monitors":"192.168.80.233:6789","pool":"kubernetes"}
             csiProvisionerIdentity=1516716490787-8081-rbdplugin  <------ !!!
             pv.kubernetes.io/provisioned-by=rbdplugin
ageClass:    rbdv2              <------ !!!
us:          Bound              <------ !!!
m:           default/csi-pvc    <------ !!!
aim Policy:  Delete
ss Modes:    RWO
meMode:      Filesystem
city:        5Gi
age:         
ce:
Type:    CSI <------ !!!

Step 9: Create a test pod

bectl create -f ./deploy/rbd/pod.yaml

CephFS plugin

A CephFS CSI plugin is available to help simplify storage management. Once user creates PVC with the reference to a CephFS CSI storage class, corresponding PV object gets dynamically created and becomes ready to be used by workloads.

Configuration Requirements

Secret object with the authentication user ID userID and key userKey for ceph cluster
StorageClass with csi-cephfsplugin (default CSI CephFS plugin name) as a provisioner name and information about ceph cluster (monitors, pool, rootPath, …)
Service Accounts with required RBAC permissions

Mounter options: specifies whether to use FUSE or ceph kernel client for mounting. By default, the plugin will probe for ceph-fuse. If this fails, the kernel client will be used instead. Command line argument --volumemounter=[fuse|kernel] overrides this behaviour.

StorageClass options:

provisionVolume: "bool": if set to true, the plugin will provision and mount a new volume. Admin credentials adminID and adminKey are required in the secret object, since this also creates a dedicated RADOS user used for mounting the volume.
rootPath: /path-in-cephfs: required field if provisionVolume=true. CephFS is mounted from the specified path. User credentials userID and userKey are required in the secret object.
mounter: "kernel" or "fuse": (optional) per-StorageClass mounter configuration. Overrides the default mounter.

Feature Status

1.10: Alpha

Important: CSIPersistentVolume and MountPropagation feature gates must be enabled starting in 1.9. Also API server must run with running config set to: storage.k8s.io/v1alpha1

kube-apiserver must be launched with --feature-gates=CSIPersistentVolume=true,MountPropagation=true and --runtime-config=storage.k8s.io/v1alpha1=true
kube-controller-manager must be launched with --feature-gates=CSIPersistentVolume=true
kubelet must be launched with --feature-gates=CSIPersistentVolume=true,MountPropagation=true

Compiling

CSI CephFS plugin can be compiled in a form of a binary file or in a form of a container. When compiled as a binary file, it gets stored in _output folder with the name cephfsplugin. When compiled as a container, the resulting image is stored in a local docker's image store.

To compile just a binary file:

ke cephfsplugin

To build a container:

ke cephfsplugin-container

By running:

cker images | grep cephfsplugin

You should see the following line in the output:

.io/cephcsi/cephfsplugin              v0.2.0                            79482e644593        4 minutes ago       305MB

Testing

Prerequisite Enable Mount Propagation in Docker

Comment out MountFlags=slave in docker systemd service then restart docker service.

stemctl daemon-reload
stemctl restart docker

Enable Kubernetes Feature Gates

Enable features MountPropagation=true,CSIPersistentVolume=true and runtime config storage.k8s.io/v1alpha1=true

Step 1: Create Secret

bectl create -f ./deploy/cephfs/kubernetes/secret.yaml

Important: secret.yaml, must be customized to match your ceph environment.

Step 2: Create StorageClass

bectl create -f ./deploy/cephfs/kubernetes/cephfs-storage-class.yaml

Important: cephfs-storage-class.yaml, must be customized to match your ceph environment.

Step 3: Start CSI CEPH CephFS plugin

bectl create -f ./deploy/cephfs/kubernetes/cephfsplugin.yaml

Step 4: Start CSI External Attacher

bectl create -f ./deploy/cephfs/kubernetes/csi-attacher.yaml

Step 5: Start CSI External Provisioner

bectl create -f ./deploy/cephfs/kubernetes/csi-provisioner.yaml

Important: Deployment yaml files includes required Service Account definitions and required RBAC rules.

Step 6: Check status of CSI CephFS plugin

bectl get pods | grep csi 
attacher-0           1/1       Running   0          6m
cephfsplugin-hmqpk   2/2       Running   0          6m
provisioner-0        1/1       Running   0          6m

Step 7: Create PVC

bectl create -f ./deploy/cephfs/kubernetes/pvc.yaml

Step 8: Check status of provisioner PV

bectl get pv
                                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
rnetes-dynamic-pv-715cef0b30d811e8   5Gi        RWX            Delete           Bound     default/csi-cephfs-pvc   csi-cephfs               5s


bectl describe pv kubernetes-dynamic-pv-715cef0b30d811e8
:            kubernetes-dynamic-pv-715cef0b30d811e8
ls:          <none>
tations:     pv.kubernetes.io/provisioned-by=csi-cephfsplugin
ageClass:    csi-cephfs
us:          Bound
m:           default/csi-cephfs-pvc
aim Policy:  Delete
ss Modes:    RWX
city:        5Gi
age:         
ce:
Type:    CSI (a Container Storage Interface (CSI) volume source)
Driver:      ReadOnly:  %v

VolumeHandle:                                                                    csi-cephfsplugin
XTRA string=csi-cephfs-7182b779-30d8-11e8-bf01-5254007d7491, bool=false)Events:  <none>

Step 9: Create a test pod

bectl create -f ./deploy/cephfs/kubernetes/pod.yaml

Troubleshooting

Please submit an issue at:Issues

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.