json으로 데이터가 제공되는 API 크롤링하기

다방 클론 프로젝트를 하면서, 완성도를 높이고자 다방에 뿌려지고 있는 수 많은 데이터를 크롤링 해야했는데요, 이때 작성한 코드를 공유하고자 합니다.

이번에 크롤링할 목록은 방찾기 페이지의 방 리스트 입니다. 이 리스트에서 각 방의 아이디 값과 방 타입, 광고 여부를 가져올 예정입니다.

다행히 이전 스타일쉐어 프로젝트 때와는 다르게 다방에서는 json 형태의 response 데이터를 API로 규칙성에 맞게 제공하고 있었습니다. 그래서 beautifulsoup이나 selenium을 사용하지 않고도 크롤링을 할 수 있었습니다.

먼저 크롤링에 사용할 모듈들을 임포트해옵니다. 그리고 크롤링한 데이터를 csv에 저장할 때 각 데이터를 명확히 구분하고자 첫 행에 데이터의 필드명을 입력받을 수 있게 코드를 작성했습니다. 크롤링한 데이터들이 추가되기 전에 먼저 필드명을 입력해놓는 방식으로요.

import csv
import json
import requests

limit = 30 

room_list_result = [{'room_id' : 'room_id', 'room_type' : 'room_type', 'is_quick' : 'is_quick'}]

그리고 이제 API를 따와야하는데요, 크롬개발자 도구에서 Network-XHR을 보면 다방에서 쏴주는 API를 확인할 수 있습니다. 새로고침을 하면 아래 보이는 세개의 API를 쏴주는데, info는 제 계정 정보를 보내주고, bbox는 방 리스트, multi-room 은 지도 상에 노출되는 데이터를 쏴주고 있었습니다.

오른쪽에 있는 Headers의 RequestURL을 따서 다른 탭에서 열어봅시다.

크롬에 json view가 확장프로그램으로 깔려있다면, API가 쏴준 json 형식의 데이터가 아래와 같이 깔끔하게 정돈되어 나옵니다. 이제 이 데이터를 활용해서 크롤링을 하면 됩니다.

쿼리가 붙은 URL이 너무 길어서 줄바꿈을 좀 했습니다. requests.get으로 API에 보낸 요청의 응답을 req에 담아줍니다. URL의 마지막 부분에 있는 {page}부분에는 각 페이지의 값을 계속 불러올 수 있도록 for 문으로 바뀐 페이지 번호를 전달해줍니다.

for page in range(1,limit):
    # one_room etc
    req = requests.get(f'https://www.dabangapp.com/api/3/room/list/multi-room/bbox?\
    api_version=3.0.1&call_type=web&filters=%7B%22multi_room_type%22%3A%5B0%2C1%2C2%5D%2C%22\
    selling_type%22%3A%5B0%2C1%2C2%5D%2C%22deposit_range%22%3A%5B0%2C999999%5D%2C%22price_range%22%3A%5B0%2C999999%5D%2C%22\
    trade_range%22%3A%5B0%2C999999%5D%2C%22maintenance_cost_range%22%3A%5B0%2C999999%5D%2C%22include_maintenance_option1%22%3Atrue%2C%22\
    room_size%22%3A%5B0%2C999999%5D%2C%22supply_space_range%22%3A%5B0%2C999999%5D%2C%22room_floor_multi%22%3A%5B1%2C2%2C3%2C4%2C5%2C6%2C7%2C-1%2C0%5D%2C%22\
    division%22%3Afalse%2C%22duplex%22%3Afalse%2C%22room_type%22%3A%5B1%2C2%5D%2C%22enter_date_range%22%3A%5B0%2C999999%5D%2C%22parking_average_range%22%3A%5B0%2C999999%5D%2C%22\
    household_num_range%22%3A%5B0%2C999999%5D%2C%22parking%22%3Afalse%2C%22animal%22%3Afalse%2C%22short_lease%22%3Afalse%2C%22full_option%22%3Afalse%2C%22built_in%22%3Afalse%2C%22\
    elevator%22%3Afalse%2C%22balcony%22%3Afalse%2C%22loan%22%3Afalse%2C%22safety%22%3Afalse%2C%22pano%22%3Afalse%2C%22\
    deal_type%22%3A%5B0%2C1%5D%7D&location=%5B%5B126.72381706093961%2C37.62761714418107%5D%2C%5B\
    126.82116752313505%2C37.69563064198876%5D%5D&page={page}')

그리고 나서, 원하는 데이터를 골라내어 결과 리스트에 저장해줍니다.

    data = req.json()

    for room in data['rooms']:
        room_list_result.append(
            {
                'room_id' : room['id'],
                'room_type' : room['room_type'],
                'is_quick' : room['is_quick']
            }
        )

마지막으로 결과리스트에서 각 행에 들어갈 값을 뽑아 새로 만들 CSV 파일에 저장해주면 크롤링이 마무리 됩니다. (물론 이제 DB에 넣을 수 있도록 데이터 가공은 해줘야합니다.

with open('./results/01_room_lists.csv', mode = 'w') as room_lists:
    room_writer = csv.writer(room_lists)

    for room in room_list_result:
        room_writer.writerow([room['room_id'], room['room_type'], room['is_quick']])

완성된 코드

import csv
import json
import requests

lastpage = 30 

room_list_result = [{'room_id' : 'room_id', 'room_type' : 'room_type', 'is_quick' : 'is_quick'}]
for page in range(1,lastpage+1):
    # one_room etc
    req = requests.get(f'https://www.dabangapp.com/api/3/room/list/multi-room/bbox?api_version=3.0.1&call_type=web&filters=%7B%22multi_room_type%22%3A%5B0%2C1%2C2%5D%2C%22selling_type%22%3A%5B0%2C1%2C2%5D%2C%22deposit_range%22%3A%5B0%2C999999%5D%2C%22price_range%22%3A%5B0%2C999999%5D%2C%22trade_range%22%3A%5B0%2C999999%5D%2C%22maintenance_cost_range%22%3A%5B0%2C999999%5D%2C%22include_maintenance_option1%22%3Atrue%2C%22room_size%22%3A%5B0%2C999999%5D%2C%22supply_space_range%22%3A%5B0%2C999999%5D%2C%22room_floor_multi%22%3A%5B1%2C2%2C3%2C4%2C5%2C6%2C7%2C-1%2C0%5D%2C%22division%22%3Afalse%2C%22duplex%22%3Afalse%2C%22room_type%22%3A%5B1%2C2%5D%2C%22enter_date_range%22%3A%5B0%2C999999%5D%2C%22parking_average_range%22%3A%5B0%2C999999%5D%2C%22household_num_range%22%3A%5B0%2C999999%5D%2C%22parking%22%3Afalse%2C%22animal%22%3Afalse%2C%22short_lease%22%3Afalse%2C%22full_option%22%3Afalse%2C%22built_in%22%3Afalse%2C%22elevator%22%3Afalse%2C%22balcony%22%3Afalse%2C%22loan%22%3Afalse%2C%22safety%22%3Afalse%2C%22pano%22%3Afalse%2C%22deal_type%22%3A%5B0%2C1%5D%7D&location=%5B%5B126.72381706093961%2C37.62761714418107%5D%2C%5B126.82116752313505%2C37.69563064198876%5D%5D&page={page}')

    data = req.json()

    for room in data['rooms']:
        room_list_result.append(
            {
                'room_id' : room['id'],
                'room_type' : room['room_type'],
                'is_quick' : room['is_quick']
            }
        )

for page in range(1,lastpage+1):
    # apartment etc
    req = requests.get(f'https://www.dabangapp.com/api/3/room/list/multi-room/bbox?api_version=3.0.1&call_type=web&filters=%7B%22multi_room_type%22%3A%5B3%5D%2C%22selling_type%22%3A%5B0%2C1%2C2%5D%2C%22deposit_range%22%3A%5B0%2C999999%5D%2C%22price_range%22%3A%5B0%2C999999%5D%2C%22trade_range%22%3A%5B0%2C999999%5D%2C%22maintenance_cost_range%22%3A%5B0%2C999999%5D%2C%22room_size%22%3A%5B0%2C999999%5D%2C%22supply_space_range%22%3A%5B0%2C999999%5D%2C%22room_floor_multi%22%3A%5B1%2C2%2C3%2C4%2C5%2C6%2C7%2C-1%2C0%5D%2C%22division%22%3Afalse%2C%22duplex%22%3Afalse%2C%22room_type%22%3A%5B%5D%2C%22enter_date_range%22%3A%5B0%2C999999%5D%2C%22parking_average_range%22%3A%5B0%2C999999%5D%2C%22household_num_range%22%3A%5B0%2C999999%5D%2C%22parking%22%3Afalse%2C%22animal%22%3Afalse%2C%22short_lease%22%3Afalse%2C%22full_option%22%3Afalse%2C%22built_in%22%3Afalse%2C%22elevator%22%3Afalse%2C%22balcony%22%3Afalse%2C%22loan%22%3Afalse%2C%22safety%22%3Afalse%2C%22pano%22%3Afalse%2C%22deal_type%22%3A%5B0%2C1%5D%7D&location=%5B%5B126.72381706093961%2C37.62761714418107%5D%2C%5B126.83477379498828%2C37.69565049135809%5D%5D&page={page}')

    data = req.json()

    for room in data['rooms']:
        room_list_result.append(
            {
                'room_id' : room['id'],
                'room_type' : room['room_type'],
                'is_quick' : room['is_quick']
            }
        )

with open('./results/01_room_lists.csv', mode = 'w') as room_lists:
    room_writer = csv.writer(room_lists)

    for room in room_list_result:
        room_writer.writerow([room['room_id'], room['room_type'], room['is_quick']])