Apache superset

Report dashboard 역할을 하는 open source를 조사하던 중 apache superset을 알게 되었고, Air-gap kubernetes 환경에서 helm chart로 설치 및 구성하는 방법을 정리한다.

---

1. download & install

• k3d로 cluster 구성
  https://velog.io/@utcloud/k3d%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%98%EC%97%AC-docker%EC%83%81%EC%97%90-k8s-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B8%B0

브라우저에서 https://github.com/apache/superset/tree/master에서 소스를 zip으로 다운로드 한 다음 helm chart 디렉토리로 이동

cd superset-master/helm/superset
cp values.yaml override-values.yaml

vi override-values.yaml

# ModuleNotFoundError: No module named 'psycopg2' during k8 installation 에러 발생하여
# https://github.com/apache/superset/discussions/31431 참고
bootstrapScript: |
  #!/bin/bash

  # Install system-level dependencies
  apt-get update && apt-get install -y \
    python3-dev \
    default-libmysqlclient-dev \
    build-essential \
    pkg-config

  # Install required Python packages
  pip install \
    authlib \
    psycopg2-binary \
    mysqlclient \

  # Create bootstrap file if it doesn't exist
  if [ ! -f ~/bootstrap ]; then
    echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap;
  fi

# SECRET_KEY는 "openssl rand -base64 42" 명령으로 생성
extraSecretEnv:
  SUPERSET_SECRET_KEY: 'ekwfeHSOLYXLq5VBDhg8PUMgnfG0XAkvQ6AYnvkealQabIZWrkjAC1p1'  
  
# SECRET_KEY는 "openssl rand -base64 42" 명령으로 생성
configOverrides:
  secret: |
    SECRET_KEY = 'T3+PLslso0Z1VDlas2x4PUjYdXaq1DoZ90O2UATkM5ZCg6OViz3HnHvf'

# Superset은 Scarf Gateway를 사용하여 개인 식별 정보(PII)를 제거하고 
# 원격 측정 데이터를 수집함으로 image
# apachesuperset.docker.scarf.sh/apache/supersetapache/superset 에서 
# apache/superset로 변경함.
image:
  #repository: apachesuperset.docker.scarf.sh/apache/superset
  repository: apache/superset
  tag: ~
  pullPolicy: IfNotPresent
service:
  type: NodePort
  port: 8088
  annotations: {}
  loadBalancerIP: ~
  nodePort:
    http: 30000  

$ helm upgrade -i -n superset --create-namespace superset -f override-values.yaml .

Release "superset" does not exist. Installing it now.
Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: postgresql, redis

$ helm dependency build
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "awx-operator" chart repository (https://ansible.github.io/awx-operator/):
	failed to fetch https://ansible.github.io/awx-operator/index.yaml : 404 Not Found
...Successfully got an update from the "superset" chart repository
...Successfully got an update from the "grafana" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading postgresql from repo oci://registry-1.docker.io/bitnamicharts
Pulled: registry-1.docker.io/bitnamicharts/postgresql:12.1.6
Digest: sha256:640c97bc5971fccdd1be6b2c7b399da8858fa83a0cae9ff5ab3dca53a17d1c56
Downloading redis from repo oci://registry-1.docker.io/bitnamicharts
Pulled: registry-1.docker.io/bitnamicharts/redis:17.9.4
Digest: sha256:e20bf7d76f2df0182ff7dd89b19708a80e721450efad891eab314ea892639f78
Deleting outdated charts

---

2. 결과 확인

kubectl -n superset get po
NAME                               READY   STATUS      RESTARTS   AGE
superset-redis-master-0            1/1     Running     0          116m
superset-postgresql-0              1/1     Running     0          116m
superset-worker-768d59d57c-h2w6f   1/1     Running     0          27m
superset-58bd566c55-hkfjz          1/1     Running     0          27m
superset-init-db-hpd9n             0/1     Completed   0          27m

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: superset
    meta.helm.sh/release-namespace: superset
  creationTimestamp: "2025-02-28T23:07:16Z"
  generation: 1
  labels:
    app.kubernetes.io/component: master
    app.kubernetes.io/instance: superset
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: redis
    helm.sh/chart: redis-17.9.4
  name: superset-redis-master
  namespace: superset
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: master
      app.kubernetes.io/instance: superset
      app.kubernetes.io/name: redis
  serviceName: superset-redis-headless
  template:
    metadata:
      annotations:
        checksum/configmap: fe1cca2dea4020cb4b1623fc950515f7338ac9fd6e0328821274263d9e0e4d64
        checksum/health: 73f335f1a98f6f244d0ef2e3f93af8046efda1328e0421e6099d13241de96558
        checksum/scripts: 9790c77d02ac85c0f4b6067139404f8bc34ddcdf1da13c05021f0904c0451a8d
        checksum/secret: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: master
        app.kubernetes.io/instance: superset
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: redis
        helm.sh/chart: redis-17.9.4
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/component: master
                  app.kubernetes.io/instance: superset
                  app.kubernetes.io/name: redis
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - -c
        - /opt/bitnami/scripts/start-scripts/start-master.sh
        command:
        - /bin/bash
        env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: REDIS_REPLICATION_MODE
          value: master
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
        image: docker.io/bitnami/redis:7.0.10-debian-11-r4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - /health/ping_liveness_local.sh 5
          failureThreshold: 5
          initialDelaySeconds: 20
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 6
        name: redis
        ports:
        - containerPort: 6379
          name: redis
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - /health/ping_readiness_local.sh 1
          failureThreshold: 5
          initialDelaySeconds: 20
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 2
        resources: {}
        securityContext:
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/bitnami/scripts/start-scripts
          name: start-scripts
        - mountPath: /health
          name: health
        - mountPath: /data
          name: redis-data
        - mountPath: /opt/bitnami/redis/mounted-etc
          name: config
        - mountPath: /opt/bitnami/redis/etc/
          name: redis-tmp-conf
        - mountPath: /tmp
          name: tmp
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: superset-redis
      serviceAccountName: superset-redis
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 493
          name: superset-redis-scripts
        name: start-scripts
      - configMap:
          defaultMode: 493
          name: superset-redis-health
        name: health
      - configMap:
          defaultMode: 420
          name: superset-redis-configuration
        name: config
      - emptyDir: {}
        name: redis-tmp-conf
      - emptyDir: {}
        name: tmp
      - emptyDir: {}
        name: redis-data
  updateStrategy:
    type: RollingUpdate

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: superset
    meta.helm.sh/release-namespace: superset
  creationTimestamp: "2025-02-28T23:07:16Z"
  generation: 1
  labels:
    app.kubernetes.io/component: primary
    app.kubernetes.io/instance: superset
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    helm.sh/chart: postgresql-12.1.6
  name: superset-postgresql
  namespace: superset
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: primary
      app.kubernetes.io/instance: superset
      app.kubernetes.io/name: postgresql
  serviceName: superset-postgresql-hl
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: primary
        app.kubernetes.io/instance: superset
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: postgresql
        helm.sh/chart: postgresql-12.1.6
      name: superset-postgresql
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/component: primary
                  app.kubernetes.io/instance: superset
                  app.kubernetes.io/name: postgresql
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: POSTGRESQL_PORT_NUMBER
          value: "5432"
        - name: POSTGRESQL_VOLUME_DIR
          value: /bitnami/postgresql
        - name: PGDATA
          value: /bitnami/postgresql/data
        - name: POSTGRES_USER
          value: superset
        - name: POSTGRES_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgres-password
              name: superset-postgresql
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: superset-postgresql
        - name: POSTGRES_DB
          value: superset
        - name: POSTGRESQL_ENABLE_LDAP
          value: "no"
        - name: POSTGRESQL_ENABLE_TLS
          value: "no"
        - name: POSTGRESQL_LOG_HOSTNAME
          value: "false"
        - name: POSTGRESQL_LOG_CONNECTIONS
          value: "false"
        - name: POSTGRESQL_LOG_DISCONNECTIONS
          value: "false"
        - name: POSTGRESQL_PGAUDIT_LOG_CATALOG
          value: "off"
        - name: POSTGRESQL_CLIENT_MIN_MESSAGES
          value: error
        - name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES
          value: pgaudit
        image: docker.io/bitnami/postgresql:14.6.0-debian-11-r13
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - exec pg_isready -U "superset" -d "dbname=superset" -h 127.0.0.1 -p 5432
          failureThreshold: 6
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: postgresql
        ports:
        - containerPort: 5432
          name: tcp-postgresql
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - -e
            - |
              exec pg_isready -U "superset" -d "dbname=superset" -h 127.0.0.1 -p 5432
              [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
          failureThreshold: 6
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
        securityContext:
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /bitnami/postgresql
          name: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
        name: dshm
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
      storageClassName: local-path
      volumeMode: Filesystem
    status:
      phase: Pending

• superset deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: superset
    meta.helm.sh/release-namespace: superset
  creationTimestamp: "2025-02-28T23:07:16Z"
  generation: 2
  labels:
    app: superset
    app.kubernetes.io/managed-by: Helm
    chart: superset-0.14.0
    heritage: Helm
    release: superset
  name: superset
  namespace: superset
  resourceVersion: "9599"
  uid: 617fa1b3-93da-4430-9e82-dfba8f18b2be
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: superset
      release: superset
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/configOverrides: f9e043eed33c175bc43df71e5d6890fab3b92f54897e0f2cde14f9b53ee85eb0
        checksum/configOverridesFiles: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/connections: 2d41a2c51ec2fc809be84fc0ea0e603c946d529b4a0e43f7257e617909561a75
        checksum/extraConfigs: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/extraSecretEnv: c2ca24e065817de8284d86c3585877e7bb4a86d455dca7e4711ec79347a4db84
        checksum/extraSecrets: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/superset_bootstrap.sh: a9970fd258b84ccabecf5e1654fa337b26cfb094da4525137aa05bd3f3450746
        checksum/superset_config.py: ab03fa8b8640fff8266ddb856f120674270f718088fc0babd7b957c0b0f8ed47
        checksum/superset_init.sh: e6b1e8eac1f7a79a07a6c72a0e2ee6e09654eeb439c6bbe61bfd676917c41e02
      creationTimestamp: null
      labels:
        app: superset
        release: superset
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - . /app/pythonpath/superset_bootstrap.sh; /usr/bin/run-server.sh
        env:
        - name: SUPERSET_PORT
          value: "8088"
        envFrom:
        - secretRef:
            name: superset-env
        image: apachesuperset.docker.scarf.sh/apache/superset:4.1.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        name: superset
        ports:
        - containerPort: 8088
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          failureThreshold: 60
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app/pythonpath
          name: superset-config
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - dockerize -wait "tcp://$DB_HOST:$DB_PORT" -timeout 120s
        envFrom:
        - secretRef:
            name: superset-env
        image: apache/superset:dockerize
        imagePullPolicy: IfNotPresent
        name: wait-for-postgres
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: superset-config
        secret:
          defaultMode: 420
          secretName: superset-config

• superset-worker deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: superset
    meta.helm.sh/release-namespace: superset
  creationTimestamp: "2025-02-28T23:07:16Z"
  generation: 2
  labels:
    app: superset-worker
    app.kubernetes.io/managed-by: Helm
    chart: superset-0.14.0
    heritage: Helm
    release: superset
  name: superset-worker
  namespace: superset
  resourceVersion: "9550"
  uid: 9fa1543e-161f-4986-8c7c-a599ca0b5219
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: superset-worker
      release: superset
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/configOverrides: f9e043eed33c175bc43df71e5d6890fab3b92f54897e0f2cde14f9b53ee85eb0
        checksum/configOverridesFiles: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/connections: 2d41a2c51ec2fc809be84fc0ea0e603c946d529b4a0e43f7257e617909561a75
        checksum/extraConfigs: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/extraSecretEnv: c2ca24e065817de8284d86c3585877e7bb4a86d455dca7e4711ec79347a4db84
        checksum/extraSecrets: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
        checksum/superset_bootstrap.sh: a9970fd258b84ccabecf5e1654fa337b26cfb094da4525137aa05bd3f3450746
        checksum/superset_config.py: ab03fa8b8640fff8266ddb856f120674270f718088fc0babd7b957c0b0f8ed47
      creationTimestamp: null
      labels:
        app: superset-worker
        release: superset
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - . /app/pythonpath/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app
          worker
        env:
        - name: SUPERSET_PORT
          value: "8088"
        envFrom:
        - secretRef:
            name: superset-env
        image: apachesuperset.docker.scarf.sh/apache/superset:4.1.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME
          failureThreshold: 3
          initialDelaySeconds: 120
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 60
        name: superset
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app/pythonpath
          name: superset-config
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - /bin/sh
        - -c
        - dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT"
          -timeout 120s
        envFrom:
        - secretRef:
            name: superset-env
        image: apache/superset:dockerize
        imagePullPolicy: IfNotPresent
        name: wait-for-postgres-redis
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: superset-config
        secret:
          defaultMode: 420
          secretName: superset-config

initial account: admin/admin