Iceberg (Time Travel)

1. CREAT TABLE

create table time_travel (
    ID BIGINT,
    NAME STRING
) USING ICEBERG
    LOCATION 's3a://martinispark/timetravel'

2. INSERT INTO Data

INSERT INTO time_travel VALUES
(1,'martini'),
(2,'kebin'),
(3,'jack')

3. INSERT INTO Data

INSERT INTO time_travel VALUES
(4,'dayoung'),
(5,'elly')

4. Data 조회

5. DELETE Data

DELETE FROM time_travel WHERE id=3

6. Data 조회

7. SNAPSHOT 조회

SELECT * FROM default.time_travel.snapshots;

• SNAPSHOT 조회 시 Catalog 이름을 붙여줘야 에러가 발생하지 않는다.

  • Table Column 정보

    • commited_at: 언제 커밋했는지
    • snapshot_id, parent_id : snapshot 및 부모 snapshot이 누군지
    • operation: 어떤 Operation을 했는지
    • manifest_list: Data 파일이 뭔지

• 이외에 메타데이터 정보는 AWS 공식문서 참조

  • $files – Shows a table's current data files.
  • $manifests – Shows a table's current file manifests.
  • $history – Shows a table's history.
  • $partitions – Shows a table's current partitions.
  • $snapshots – Shows a table's snapshots.
  • $refs – Shows a table's references.

8. Time Travel 기능 이용

SELECT * FROM T 
FOR SYSTEM_TIME AS OF <TIMESTAMP>;

SELECT * FROM t 
FOR SYSTEM_VERSION AS OF <SNAPSHOT_ID>;

• Spark3.3.0 이상부터 지원, 2가지 SQL로 사용
  
  
  
  

Snapshot_Id 이용

SELECT * FROM time_travel FOR SYSTEM_VERSION AS OF 8940556835085920119;

TIMESTAMP 이용

SELECT * FROM time_travel FOR SYSTEM_TIME AS OF '2024-01-18 20:59:44.562000000';

참고

• https://hoony-612.tistory.com/89