26M23a

When a pool is added, AIStor operator creates a new statefulset for that pool and will restart AIStor to use this new configuration. Kubernetes will create persistent volume claims, persistent volumes and pods. If that takes time, then AIStor will not be able to reach the pods in the second pool and may be unresponsive.

The DNS errors indicate that pods were created slowly. I know you cannot share kubectl output, but could you check the output of the following commands:

kubectl get sts -n -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
kubectl get pods -n -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
This shows the creation timestamps of the statefulset of the pools and all pods. Please check if the creation timestamps of the pods in the new pool are close to the creation timestamp of the statefulset. If it's not, then Kubernetes is slow to create the pods. You may want to describe the pods to determine why it's slow.

If the pods were created quickly, then run (this requires jq to be installed):

kubectl get pv -o json | jq -r '.items[] | select(.spec.claimRef.namespace=="minio") | [.metadata.name, .spec.claimRef.name, .metadata.creationTimestamp] | @tsv'
That shows when the actual persistent volumes have been created. If the persistent volumes are created slowly, then use kubectl describe pv to find out what may have caused this delay.

If all Kubernetes resources have been created quickly, then it may be a problem with the DNS that is too slow to actually catch up.

We will also investigate this issue. We may not want to restart AIStor using the new pool configuration, until all resources have been created and the DNS responds to the pods. That would reduce the down-time.