Python pyiceberg 패키지로 aws glue iceberg table 조회 하는 방법

패키지 설치

$ python3 -m pip install "pyiceberg[s3fs,glue]"

1. 기본 사용법

from pyiceberg.catalog.glue import GlueCatalog

glue_catalog = GlueCatalog(name='default')
glue_catalog.load_table('prod_de_neat.sports_league_v1').scan().to_pandas()

2. WHERE필터링

from pyiceberg.catalog.glue import GlueCatalog

glue_catalog = GlueCatalog(name='default')
glue_catalog.load_table('prod_de_neat.sports_league_v1').scan(row_filter="""sports_code='S1'""").to_pandas()

3. 컬럼 전체 선택

from pyiceberg.catalog.glue import GlueCatalog

glue_catalog = GlueCatalog(name='default')
glue_catalog.load_table('prod_de_neat.sports_league_v1').scan(row_filter="""sports_code='S1'""", selected_fields=("*", )).to_pandas()

4. 일부 컬럼 선택

from pyiceberg.catalog.glue import GlueCatalog

glue_catalog = GlueCatalog(name='default')
glue_catalog.load_table('prod_de_neat.sports_league_v1').scan(row_filter="""sports_code='S1'""", selected_fields=("sports_code", "sports_name")).to_pandas()

참고 문서

• https://py.iceberg.apache.org/reference/pyiceberg/catalog/glue/
• https://py.iceberg.apache.org/