๐Ÿณ[์—๋Ÿฌ]Runtime Error with DataLoader: exited unexpectedly

๐Ÿ™Œ๐Ÿปยท2021๋…„ 6์›” 7์ผ
0

๋„์ปค

๋ชฉ๋ก ๋ณด๊ธฐ
1/2
post-thumbnail

ํŒŒ์ดํ† ์น˜ dataloader๋กœ ํ•™์Šตํ•  ๋ฐ์ดํ„ฐ๋ฅผ ๋ถˆ๋Ÿฌ์˜จ ํ›„ ํ•™์Šต์„ ์ง„ํ–‰ํ•  ๋•Œ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•˜์˜€๋‹ค..

RuntimeError: DataLoader worker (pid 21427) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

์ด๋Š” docker ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๋„์šธ ๋•Œ ๋””์Šคํฌ ์šฉ๋Ÿ‰์„ ๊ณต์œ ํ•˜๋Š”๋ฐ ๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ๋ถ€์กฑํ•ด์„œ ๋ฐœ์ƒํ•˜๋Š” ์—๋Ÿฌ์ด๋‹ค.

shm ์‚ฌ์ด์ฆˆ๊ฐ€ ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์ƒ์„ฑํ•  ๋•Œ ๋””ํดํŠธ๊ฐ’์ด 64m ์ด๋‹ค

๋”ฐ๋ผ์„œ

docker run --name 1_ne_ -ti --shm-size=400G -v

์ด์™€ ๊ฐ™์ด ์ง„ํ–‰ ํ•ด์ฃผ๋ฉด

๋ฉ”๋ชจ๋ฆฌ๊ฐ€ ์ฆ๊ฐ€ํ•จ์„ ์•Œ ์ˆ˜ ์žˆ๋‹ค.

1ne : ์ปจํ…Œ์ด๋„ˆ ์ด๋ฆ„
--shm-size=400G : ์šฉ๋Ÿ‰ 400G ์„ค์ •

overlay 879G 739G 95G 89% /
tmpfs 64M 0 64M 0% /dev
tmpfs 126G 0 126G 0% /sys/fs/cgroup
shm 400G 28K 400G 1% /dev/shm
/dev/sdb1 3.5T 2.7T 576G 83% /Data
/dev/sda2 879G 739G 95G 89% /Files
tmpfs 126G 12K 126G 1% /proc/driver/nvidia
tmpfs 26G 1.5G 24G 6% /run/nvidia-persistenced/socket
udev 126G 0 126G 0% /dev/nvidia0
tmpfs 126G 0 126G 0% /proc/asound
tmpfs 126G 0 126G 0% /proc/acpi
tmpfs 126G 0 126G 0% /proc/scsi
tmpfs 126G 0 126G 0% /sys/firmware

0๊ฐœ์˜ ๋Œ“๊ธ€