26M18b

I’m wondering whether it’s possible to rebuild an existing AIStore-managed environment without any data loss in case the entire Kubernetes cluster is destroyed and needs to be recreated.
If it’s possible, I’d like to know the exact steps or requirements for that recovery process.

I heard that after rebuilding the Kubernetes cluster, it’s sufficient to simply reinstall the Volume Manager, ObjectStore Operator, and ObjectStore using Helm, and that the system will automatically recognize and reuse the existing data without any additional action.
Is this really true? If so, could you please explain in detail how it works?

J
Jiuker11:57 AM
Yes, and most importantly, the volume can be reused to recover data. @Bala cc

B
Bala12:02 PM
@Youngkyoo Kim As I mentioned earlier, you are not allowed to destroy kubernetes cluster without taking necessary backup. If you have a backup of CRDs, just applying the backup in a new kubernetes works without any issues. If you have specific questions, feel free to ask.

Y
Youngkyoo Kim12:14 PM
Could you share the commands to backup CRDs?

Just directpvvolume and directpvdrives will be ok for CRDs? How about pvc, pv or something like that needed to backup?

J
Jiuker02:07 PM
@Bala cc

Backup all the K8S resource should use Velero

B
Bala02:13 PM

Y
Youngkyoo Kim
Just directpvvolume and directpvdrives will be ok for CRDs? How about pvc, pv or something like that needed to backup?

@Youngkyoo Kim You would need to backup PV, PVC, directpvdrives and directpvvolumes from DirectPV point of view like

$ kubectl get directpvdrives -o yaml > directpvdrives.backup.yaml
$ kubectl get directpvvolumes -o yaml > directpvvolumes.backup.yaml

Usual backup method of kubernetes standard resources

@Jeeva Kandasamy @Ramon de Klein Please share AIStor, operator, eos and other component backup procedures here

J
Jiuker02:19 PM
objectstore use kubectl get objectstores -o yaml > objectstores.backup.yaml

Ramon de Klein04:47 PM
You should also backup the secrets, because it holds some information. I generally advice to backup all the resources in the namespace of the object-store.

So also include kubectl get secrets -o yaml > objectstore-secrets.backup.yaml.

PS: Are you using DirectPV as the storage class?

Y
Youngkyoo Kim05:42 PM
Yes. We’re using DirectPV. So we need to backup and use CRDs( directpvdrives, directpvvolumes, objectstores), pvcs, pvs ,and secrets for recovery. These 6 objects are sufficient for full recovery after K8s cluster collapse?

Ramon de Klein06:36 PM
Yes, when you reinstall the operator and bring back the ObjectStore resource and the configuration secrets that are used in the resource, then it should be able to recreate the PVCs.

When rebuilding the Kubernetes cluster, it is important to assign the same names to the nodes as the previous Kubernetes cluster. This is required so DirectPV can attach the drives to the proper nodes.