구글 드라이브 데이터 다운로드

leeway·2022년 12월 30일

gdown linux python

python

목록 보기

4/4

서버 간의 대용량 파일 이동이나 복제가 잦아지면서 원격 저장소가 필요해짐

구글 드라이브 저장용량을 구매해서 사용하고 있고, 다른 프로그램과 연동이 가능한 점에서 구글 드라이브를 선택함

흔히 사용되는 wget을 활용하는 방법을 알아보다가 용량이나 권한의 한계로 gdown을 사용함

wget 보안 문제 해결 방법

wget 옵션에 쿠키를 설정해서 다운로드 할 수 있는 방법

gdown 사용하기

1. Installation

pip install gdown

to upgrade

pip install --upgrade gdown

2. Usage

From Command Line

$ gdown --help
usage: gdown [-h] [-V] [-O OUTPUT] [-q] [--fuzzy] [--id] [--proxy PROXY]
             [--speed SPEED] [--no-cookies] [--no-check-certificate]
             [--continue] [--folder] [--remaining-ok]
             url_or_id
...

$ # a large file (~500MB)
$ gdown https://drive.google.com/uc?id=1l_5RK28JRL19wpT22B-DY9We3TVXnnQQ
$ md5sum fcn8s_from_caffe.npz
256c2a8235c1c65e62e48d3284fbd384

$ # same as the above but with the file ID
$ gdown 1l_5RK28JRL19wpT22B-DY9We3TVXnnQQ

$ # a small file
$ gdown https://drive.google.com/uc?id=0B9P1L--7Wd2vU3VUVlFnbTgtS2c
$ cat spam.txt
spam

$ # download with fuzzy extraction of a file ID
$ gdown --fuzzy 'https://drive.google.com/file/d/0B9P1L--7Wd2vU3VUVlFnbTgtS2c/view?usp=sharing&resourcekey=0-WWs_XOSctfaY_0-sJBKRSQ'
$ cat spam.txt
spam

$ # --fuzzy option also works with Microsoft Powerpoint files
$ gdown --fuzzy "https://docs.google.com/presentation/d/15umvZKlsJ3094HNg5S4vJsIhxcFlyTeK/edit?usp=sharing&ouid=117512221203072002113&rtpof=true&sd=true"

$ # a folder
$ gdown https://drive.google.com/drive/folders/15uNXeRBIhVvZJIhL4yTw4IsStMhUaaxl -O /tmp/folder --folder

$ # as an alternative to curl/wget
$ gdown https://httpbin.org/ip -O ip.json
$ cat ip.json
{
  "origin": "126.169.213.247"
}

$ # write stdout and pipe to extract
$ gdown https://github.com/wkentaro/gdown/archive/refs/tags/v4.0.0.tar.gz -O - --quiet | tar zxvf -
$ ls gdown-4.0.0/
gdown  github2pypi  LICENSE  MANIFEST.in  pyproject.toml  README.md  setup.cfg  setup.py  tests

From Python

import gdown

# a file
url = "https://drive.google.com/uc?id=1l_5RK28JRL19wpT22B-DY9We3TVXnnQQ"
output = "fcn8s_from_caffe.npz"
gdown.download(url, output, quiet=False)

# same as the above, but with the file ID
id = "0B9P1L--7Wd2vNm9zMTJWOGxobkU"
gdown.download(id=id, output=output, quiet=False)

# same as the above, and you can copy-and-paste a URL from Google Drive with fuzzy=True
url = "https://drive.google.com/file/d/0B9P1L--7Wd2vNm9zMTJWOGxobkU/view?usp=sharing"
gdown.download(url=url, output=output, quiet=False, fuzzy=True)

# cached download with identity check via MD5
md5 = "fa837a88f0c40c513d975104edf3da17"
gdown.cached_download(url, output, md5=md5, postprocess=gdown.extractall)

# a folder
url = "https://drive.google.com/drive/folders/15uNXeRBIhVvZJIhL4yTw4IsStMhUaaxl"
gdown.download_folder(url, quiet=True, use_cookies=False)

# same as the above, but with the folder ID
id = "15uNXeRBIhVvZJIhL4yTw4IsStMhUaaxl"
gdown.download_folder(id=id, quiet=True, use_cookies=False)

오류

- Access denied

Error

gdown.download(url, output_name, quiet=False, fuzzy=True)

Access denied with the following error:
Cannot retrieve the public link of the file. You may need to change the permission to 'Anyone with the link', or have had many accesses. You may still be able to access the file from the browser: ***

Solution

pip install --upgrade --no-cache-dir gdown

다른 원격저장소 알아보기

1. Git LFS

git은 100MB 이상의 파일 업로드에 제한이 있으며, 대안으로 Git Large File Storage(LFS)가 있음

Git LFS 사용하기

다만 lfs 용량 제한이 있는데, Github와 Bitbucket은 1GB, GitLab은 10GB의 용량을 제공하고, 용량 추가시 돈을 지불해야한다.

Reference

leeway

AI Engineer

이전 포스트