26Z10c8

draw.io에서 바로 불러올 수 있는 단색(Monochrome) 스타일의 XML 다이어그램 코드와 각 구성요소 설명입니다.

---

1. draw.io XML 코드

아래 XML 전체를 복사한 후, draw.io(또나 diagrams.net)에서 [File] $\rightarrow$ [Import from] $\rightarrow$ [Text...] 메뉴에 붙여넣거나, [Edit] $\rightarrow$ [Insert] $\rightarrow$ [Advanced] $\rightarrow$ [XML...]을 통해 불러오면 다이어그램이 생성됩니다.

<mxGraphModel dx="1200" dy="800" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" background="#ffffff" math="0" shadow="0">
  <root>
    <mxCell id="0" />
    <mxCell id="1" parent="0" />
    
    <!-- Title -->
    <mxCell id="title" value="MinIO S3 Metadata &amp; Prefix Aggregation Pipeline (StarRocks)" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=18;fontStyle=1;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="320" y="20" width="540" height="30" as="geometry" />
    </mxCell>

    <!-- Source 1: Kafka -->
    <mxCell id="kafka" value="&lt;b&gt;[Kafka Topic]&lt;/b&gt;&lt;br&gt;&lt;i&gt;s3.events&lt;/i&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;&amp;bull; &lt;b&gt;Payload:&lt;/b&gt; S3 Event Log&lt;br&gt;&amp;bull; ObjectCreated / ObjectRemoved&lt;br&gt;&amp;bull; Real-time Stream&lt;/div&gt;" style="shape=mxgraph.flowchart.direct_data;whiteSpace=wrap;html=1;fillColor=#f8f9fa;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="40" y="90" width="220" height="100" as="geometry" />
    </mxCell>

    <!-- Process: Routine Load -->
    <mxCell id="routine_load" value="Routine Load&lt;br&gt;&lt;font size=&quot;1&quot;&gt;(Real-time Ingestion)&lt;/font&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=12;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="310" y="120" width="130" height="40" as="geometry" />
    </mxCell>

    <!-- Table 1: object_current_state -->
    <mxCell id="tbl_current_state" value="&lt;b&gt;[StarRocks Table]&lt;/b&gt;&lt;br&gt;&lt;b&gt;object_current_state&lt;/b&gt;&lt;br&gt;&lt;font color=&quot;#555555&quot;&gt;Primary Key Model&lt;/font&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;&amp;bull; &lt;b&gt;PK:&lt;/b&gt; object_id (md5)&lt;br&gt;&amp;bull; bucket, object_key, prefix&lt;br&gt;&amp;bull; size_bytes, etag, event_time&lt;br&gt;&amp;bull; &lt;b&gt;__deleted:&lt;/b&gt; TINYINT (0/1)&lt;br&gt;&lt;hr&gt;&lt;i&gt;Role: Real-time S3 Object State Store&lt;/i&gt;&lt;/div&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;align=center;verticalAlign=top;spacingTop=6;" vertex="1" parent="1">
      <mxGeometry x="500" y="80" width="260" height="160" as="geometry" />
    </mxCell>

    <!-- Source 2: MinIO Inventory Parquet -->
    <mxCell id="aistor_s3" value="&lt;b&gt;[AIStor S3 Bucket]&lt;/b&gt;&lt;br&gt;&lt;i&gt;Inventory Parquet Files&lt;/i&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;&amp;bull; Daily Snapshot Export&lt;br&gt;&amp;bull; Bucket, Key, Size, LastModified&lt;/div&gt;" style="shape=mxgraph.flowchart.stored_data;whiteSpace=wrap;html=1;fillColor=#f8f9fa;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="40" y="290" width="220" height="90" as="geometry" />
    </mxCell>

    <!-- Process: Python Reconcile Pipeline -->
    <mxCell id="reconcile_script" value="Reconciliation Script&lt;br&gt;&lt;font size=&quot;1&quot;&gt;(Python + FILES() TVF)&lt;/font&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;1. Stage to Temp Table&lt;br&gt;2. Upsert Active Objects&lt;br&gt;3. Soft Delete Missing (`__deleted=1`)&lt;/div&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="290" y="280" width="170" height="110" as="geometry" />
    </mxCell>

    <!-- Process: Batch INSERT OVERWRITE -->
    <mxCell id="batch_overwrite" value="Daily Batch Update&lt;br&gt;&lt;font size=&quot;1&quot;&gt;(INSERT OVERWRITE)&lt;/font&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="550" y="290" width="160" height="40" as="geometry" />
    </mxCell>

    <!-- Table 2: prefix_summary -->
    <mxCell id="tbl_prefix_summary" value="&lt;b&gt;[StarRocks Table]&lt;/b&gt;&lt;br&gt;&lt;b&gt;prefix_summary&lt;/b&gt;&lt;br&gt;&lt;font color=&quot;#555555&quot;&gt;Aggregate Key Model&lt;/font&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;&amp;bull; &lt;b&gt;AGG KEY:&lt;/b&gt; bucket, prefix&lt;br&gt;&amp;bull; &lt;b&gt;total_object_count:&lt;/b&gt; SUM&lt;br&gt;&amp;bull; &lt;b&gt;total_size_bytes:&lt;/b&gt; SUM&lt;br&gt;&amp;bull; &lt;b&gt;last_updated:&lt;/b&gt; MAX&lt;br&gt;&lt;hr&gt;&lt;i&gt;Role: Pre-computed Daily Aggregates&lt;/i&gt;&lt;/div&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;align=center;verticalAlign=top;spacingTop=6;" vertex="1" parent="1">
      <mxGeometry x="500" y="380" width="260" height="150" as="geometry" />
    </mxCell>

    <!-- View: v_realtime_prefix_summary -->
    <mxCell id="view_hybrid" value="&lt;b&gt;[StarRocks Logical View]&lt;/b&gt;&lt;br&gt;&lt;b&gt;v_realtime_prefix_summary&lt;/b&gt;&lt;hr&gt;&lt;div align=&quot;left&quot;&gt;&amp;bull; &lt;b&gt;Base Data:&lt;/b&gt; prefix_summary (99%)&lt;br&gt;&amp;bull; &lt;b&gt;Delta Data:&lt;/b&gt; object_current_state (1%)&lt;br&gt;&amp;nbsp;&amp;nbsp;(Where event_time &amp;gt; last_updated)&lt;br&gt;&amp;bull; FULL OUTER JOIN &amp;amp; COALESCE Aggregation&lt;br&gt;&lt;hr&gt;&lt;i&gt;Role: Sub-second Real-time Prefix Metrics&lt;/i&gt;&lt;/div&gt;" style="html=1;whiteSpace=wrap;rounded=1;fillColor=#f8f9fa;strokeColor=#333333;strokeWidth=2;fontSize=11;fontColor=#111111;align=center;verticalAlign=top;spacingTop=6;" vertex="1" parent="1">
      <mxGeometry x="840" y="210" width="280" height="160" as="geometry" />
    </mxCell>

    <!-- Consumer: API / Dashboard -->
    <mxCell id="consumer" value="&lt;b&gt;[External API / Dashboards]&lt;/b&gt;&lt;br&gt;Grafana, Prometheus, Web Service&lt;br&gt;&lt;i&gt;(Fast Response &amp;lt; 10ms)&lt;/i&gt;" style="shape=mxgraph.flowchart.display;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#333333;strokeWidth=1.5;fontSize=11;fontColor=#111111;" vertex="1" parent="1">
      <mxGeometry x="880" y="430" width="200" height="70" as="geometry" />
    </mxCell>

    <!-- Connectors -->
    <mxCell id="edge1" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="kafka" target="routine_load">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

    <mxCell id="edge2" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="routine_load" target="tbl_current_state">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

    <mxCell id="edge3" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="aistor_s3" target="reconcile_script">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

    <mxCell id="edge4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="reconcile_script" target="tbl_current_state">
      <mxGeometry relative="1" as="geometry">
        <Array as="points">
          <mxPoint x="480" y="335" />
          <mxPoint x="480" y="200" />
        </Array>
      </mxGeometry>
    </mxCell>

    <mxCell id="edge5" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="tbl_current_state" target="batch_overwrite">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

    <mxCell id="edge6" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="batch_overwrite" target="tbl_prefix_summary">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

    <mxCell id="edge7" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="tbl_current_state" target="view_hybrid">
      <mxGeometry relative="1" as="geometry">
        <Array as="points">
          <mxPoint x="800" y="160" />
          <mxPoint x="800" y="260" />
        </Array>
      </mxGeometry>
    </mxCell>

    <mxCell id="edge8" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="tbl_prefix_summary" target="view_hybrid">
      <mxGeometry relative="1" as="geometry">
        <Array as="points">
          <mxPoint x="800" y="455" />
          <mxPoint x="800" y="310" />
        </Array>
      </mxGeometry>
    </mxCell>

    <mxCell id="edge9" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#333333;strokeWidth=1.5;" edge="1" parent="1" source="view_hybrid" target="consumer">
      <mxGeometry relative="1" as="geometry" />
    </mxCell>

  </root>
</mxGraphModel>

---

2. 아키텍처 흐름 및 각 데이터베이스 테이블/뷰 요약

[Kafka (Real-time S3 Events)] ──(Routine Load)──┐
                                                 ▼
[AIStor Parquet (Daily)] ──(Reconcile Script)──► [object_current_state] (Primary Key)
                                                 │   │
                                                 │   └──(INSERT OVERWRITE)──► [prefix_summary] (Aggregate Key)
                                                 │                                 │
                                                 └───────────────┬─────────────────┘
                                                                 ▼
                                                    [v_realtime_prefix_summary] (Hybrid View)
                                                                 │
                                                                 ▼
                                                    [External API / Dashboard]

1. object_current_state (Primary Key 모델)

• 용도: S3 객체 단위의 최신 실시간 메타데이터 및 삭제 여부(__deleted) 관리
• 데이터 수입 경로:

1. Routine Load를 통한 실시간 Kafka S3 이벤트 수신
2. 일일 Python Reconciliation 스크립트를 통한 정기 동기화/소프트 삭제 마킹

• 샘플 스키마:
• object_id (PK, VARCHAR)
• bucket / object_key / prefix (VARCHAR)
• size_bytes (BIGINT)
• event_time (DATETIME - sequence_col 지정용)
• __deleted (TINYINT - 0: 존재, 1: 삭제)

2. prefix_summary (Aggregate Key 모델)

• 용도: 일일 배치 시점에 전체 S3 객체를 Prefix별 깊이로 재귀적(substring_index) 해체하여 사전 계산해둔 요약 테이블
• 데이터 수입 경로: 일일 Reconciliation 직후 Python 스크립트가 INSERT OVERWRITE 배치 실행
• 샘플 스키마:
• bucket / prefix (AGGREGATE KEY)
• total_object_count (BIGINT SUM)
• total_size_bytes (BIGINT SUM)
• last_updated (DATETIME MAX - 배치 수행 스냅샷 시각 저장)

3. v_realtime_prefix_summary (하이브리드 논리 뷰)

• 용도: 외부 API/대시보드가 수ms 이내의 응답 속도로 현재 시점의 실시간 Prefix 용량을 조회할 수 있도록 제공하는 뷰
• 동작 원리:
• 99%의 과거 데이터는 사전 집계된 prefix_summary에서 고속 스캔
• last_updated 이후에 들어온 1% 미만의 실시간 Kafka 변경분만 object_current_state에서 즉석 계산하여 FULL OUTER JOIN 합산