others-how to solve the problem of creating redis cluster in kubernetes?

Nov 11, 2020

Problem

When we create redis cluster in kubernetes, sometimes, we would get this error:

root@redis-0:/data# redis-cli --cluster create --cluster-replicas 1 10.42.3.27:6379 10.42.4.56:6379 10.42.7.113:6379 10.42.5.200:6379 10.42.6.179:6379 10.42.5.201:6379

[ERR] Node 10.42.3.27:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

why?

Environment

Kubernetes 1.19

Reason

You are using the old PVC in the newly created redis cluster statefulsets. If you just delete the statefulset like this:

kubectl delete statefulset xxx

Then the PVC bound to the statefulset is not deleted automatically by default.

According to the official kubernetes document :

Persistent Volumes

Deleting the Pods in a StatefulSet will not delete the associated volumes. This is to ensure that you have the chance to copy data off the volume before deleting it. Deleting the PVC after the pods have terminated might trigger deletion of the backing Persistent Volumes depending on the storage class and reclaim policy. You should never assume ability to access a volume after claim deletion.

Solution

You can delete the PVC as following:

kubectl delete statefulset -l app=myapp
kubectl delete pvc -l app=myapp

Please note the label “app=myapp” should be changed according to your statefulset app labels.