Airbyte Monitoring

Monitoring

1. Check the updates
   To determine how many workers are currently running in your Airbyte deployment, you can use Kubernetes commands since Airbyte is deployed as a set of Kubernetes pods when using abctl.

   First, ensure that you have configured your kubectl to communicate with your Airbyte cluster. If you're using abctl, you can do this by running:

$ kind export kubeconfig -n airbyte-abctl

Here's how you can check the number of running workers:

$ kubectl get pods -n airbyte-abctl | grep worker
or
$ kubectl get pods -n airbyte-abctl | grep worker | wc -l

2. Monitoring Usage of resource:
   Use infrastructure monitoring tools to measure CPU and Memory usage, both requested and actual.

# List of airbyte-abctl pods
$ kubectl get pods -n airbyte-abctl

# cpu and memory usage of pods
$ kubectl top pods -n airbyte-abctl

# All Resource of airbyte pods
$ kubectl get all -n <namespace> | grep airbyte

• To use Metrics API using kubectl, you need to install it.

3. Monitoring log generated in airbyte:

# Log of pod
$ kubectl logs <pod-name> -n <namespace> --all-containers

k8s metrics server

1. check metrics server is installed

$ kubectl get deployment metrics-server -n kube-system

• NotFound : not installed

2. install metrics server

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

3. check metrics server is running

$ kubectl get pods -n kube-system

# checking log
$ kubectl logs -n kube-system -l k8s-app=metrics-server

4. set the flag

$ kubectl edit deployment metrics-server -n kube-system

# add the flog "--kubelet-insecure-tls"
spec:
  containers:
  - name: metrics-server
    args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls  # 추가

5. save and apply

$ kubectl rollout restart deployment metrics-server -n kube-system

6. execute

$ kubectl top nodes

---

기본 문법

1. delete pod

# Restart specific pod
$ kubectl delete pod --field-selector=status.phase=Failed -n 

# restart all pods
$ kubectl delete pod --all -n airbyte-abctl