Ch6 Naver API 01-07 (Naver API 1-2)

김민지·2023년 4월 5일

Part 04. EDA/웹크롤링/파이썬프로그래밍

목록 보기

9/12

Naver API 등록

네이버, 페이스북, 구글 등 IT 거대 기업들부터 많은 S/W 관련 회사들이 자사 제품을 사용하는 것에 대해 API를 제공함
특히 요즘처럼 웹크롤링에 대한 방어가 점점 심해지는 이때, 해당 회사의 API를 사용하는 것이 오히려 더 바람직할 수 있음
-> API 사용에 익숙해질 필요성
네이버 개발자 센터 : https://developers.naver.com/main/
Application
-어플리케이션 등록
-어플리케이션 이름 : ds_study
-사용 API :
-검색
-데이터랩(검색어트렌드)
-데이터랩(쇼핑인사이트)
-환경추가
-WEB 설정
-http://localhost
-Client ID : 1K8p6vfg4HaGE6FmLj
-Client Secret : ktW9S7cSqB
-https://developers.naver.com/apps/#/myapps/1K8p6vfg4HaGE6FmLj/overview

Naver API 사용

https://developers.naver.com/docs/serviceapi/search/blog/blog.md#%EB%B8%94%EB%A1%9C%EA%B7%B8
urllib : http 프로토콜에 따라서 서버의 요청/응답을 처리하기 위한 모듈
urllib.request : 클라이언트의 요청을 처리하는 모듈
urllib.parse : url 주소에 대한 분석

# 네이버 검색 API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("크라임씬 리뷰")
url = "https://openapi.naver.com/v1/search/blog?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

# 글자로 읽을 경우, decode utf-8 설정
print(response_body.decode("utf-8"))

검색: 책(book)

import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("코난")
url = "https://openapi.naver.com/v1/search/book?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

검색: 카페(cafearticle)

import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("플레이브")
url = "https://openapi.naver.com/v1/search/cafearticle?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

검색: 쇼핑(shop)

import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("코난")
url = "https://openapi.naver.com/v1/search/shop?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

검색: 백과사전(encyc)

import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("크라임씬 리뷰")
url = "https://openapi.naver.com/v1/search/encyc?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

Naver API에서 모은 몰스킨 데이터 정리

import os
import sys
import urllib.request
client_id = "1K8__p6vfg4HaGE6FmLj"
client_secret = "ktW9S7cSqB"
encText = urllib.parse.quote("몰스킨")
url = "https://openapi.naver.com/v1/search/shop?query=" + encText # JSON 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # XML 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
    response_body = response.read()
    print(response_body.decode('utf-8'))
else:
    print("Error Code:" + rescode)

함수 하나씩 만들어가기

(1) gen_search_url (generate url)
-encText = urllib.parse.quote("몰스킨")
-url = "https://openapi.naver.com/v1/search/shop?query=" + encText

def gen_search_url(api_node, search_text, start_num, disp_num):
    base = "https://openapi.naver.com/v1/search"
    node = "/" + api_node + ".json"
    param_query = "?query=" + urllib.parse.quote(search_text)
    param_start = "&start=" + str(start_num)
    param_disp = "&display=" + str(disp_num)
    
    return base + node + param_query + param_start + param_disp

gen_search_url("shop", "TEST", 10, 3)

-> 'https://openapi.naver.com/v1/search/shop.json?query=TEST&start=10&display=3'

(2) get_result_onpage (Get data on one page)

import json
import datetime

def get_result_onpage(url):
    request = urllib.request.Request(url)
    request.add_header("X-Naver-Client-Id",client_id)
    request.add_header("X-Naver-Client-Secret",client_secret)
    response = urllib.request.urlopen(request)
    print("[%s] Url Request Success" % datetime.datetime.now())
    return json.loads(response.read().decode("utf-8"))

datetime.datetime.now()

-> datetime.datetime(2023, 4, 6, 0, 27, 1, 552115)

url = gen_search_url("shop", "몰스킨", 1, 5)
one_result = get_result_onpage(url)

-> [2023-04-06 00:27:56.403297] Url Request Success

one_result["items"][0]["title"]

-> '몰스킨 노트 가죽 하드커버 감성 고급 업무용 이쁜 심플'

one_result["items"][0]["link"]

-> 'https://search.shopping.naver.com/gate.nhn?id=82526953942'

one_result["items"][0]["lprice"]

-> '28800'

one_result["items"][0]["mallName"]

-> '베스트펜'

(3) get_fields (Convert pandas data frame)

one_result["items"][0]

{'title': '몰스킨 노트 가죽 하드커버 감성 고급 업무용 이쁜 심플',
'link': 'https://search.shopping.naver.com/gate.nhn?id=82526953942',
'image': 'https://shopping-phinf.pstatic.net/main_8252695/82526953942.7.jpg',
'lprice': '28800',
'hprice': '',
'mallName': '베스트펜',
'productId': '82526953942',
'productType': '2',
'brand': '몰스킨',
'maker': '',
'category1': '생활/건강',
'category2': '문구/사무용품',
'category3': '노트/수첩',
'category4': '노트'}

import pandas as pd

def get_fields(json_data):
    title = [each["title"] for each in json_data["items"]]
    link = [each["link"] for each in json_data["items"]]
    lprice = [each["lprice"] for each in json_data["items"]]
    mall_name = [each["mallName"] for each in json_data["items"]]
    
    result_pd = pd.DataFrame({
        "title": title,
        "link": link,
        "lprice": lprice,
        "mall": mall_name
    }, columns=["title", "lprice", "link", "mall"])
    return result_pd

get_fields(one_result)

(4) delete_tag()

def delete_tag(input_str):
    input_str = input_str.replace("<b>", "")
    input_str = input_str.replace("</b>", "")
    return input_str

import pandas as pd

def get_fields(json_data):
    title = [delete_tag(each["title"]) for each in json_data["items"]]
    link = [each["link"] for each in json_data["items"]]
    lprice = [each["lprice"] for each in json_data["items"]]
    mall_name = [each["mallName"] for each in json_data["items"]]
    
    result_pd = pd.DataFrame({
        "title": title,
        "link": link,
        "lprice": lprice,
        "mall": mall_name
    }, columns=["title", "lprice", "link", "mall"])
    return result_pd

get_fields(one_result)

url = gen_search_url("shop", "몰스킨", 1, 5)
json_result = get_result_onpage(url)
pd_result = get_fields(json_result)

(5) actMain (All Data gathering)

반복문 돌려서 1000개의 데이터 가져오기

for n in range(1, 1000, 100):
    print(n)

-> 1
101
201
301
401
501
601
701
801
901

result_mol = []

for n in range(1, 1000, 100):
    url = gen_search_url("shop", "몰스킨", n, 100)
    json_result = get_result_onpage(url)
    pd_result = get_fields(json_result)
    
    result_mol.append(pd_result)

result_mol = pd.concat(result_mol)

# index 다시 달아주고 원래의 인덱스 삭제
result_mol.reset_index(drop=True, inplace=True)
result_mol.info()

# 가격을 float 형태로 바꾸기
result_mol["lprice"] = result_mol["lprice"].astype("float")
result_mol.info()

(6) toExcel (Export to Excel)

!pip install xlsxwriter

writer = pd.ExcelWriter("../data/06_molskin_diary_in_naver_shop.xlsx", engine="xlsxwriter")
result_mol.to_excel(writer, sheet_name="Sheet1")

workbook = writer.book
worksheet = writer.sheets["Sheet1"]
worksheet.set_column("A:A", 4)    # 숫자는 간격, A:A 칼럼 간격을 지정
worksheet.set_column("B:B", 60)
worksheet.set_column("C:C", 10)
worksheet.set_column("D:D", 10)
worksheet.set_column("E:E", 50)
worksheet.set_column("F:F", 10)

worksheet.conditional_format("C2:C1001", {"type":"3_color_scale"})
writer.save()

데이터 시각화

import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import rc
rc("font", family="Malgun Gothic")

import seaborn as sns

plt.figure(figsize=(15, 6))
sns.countplot(
    x=result_mol["mall"],   # x축에 쇼핑몰별로 보기
    data=result_mol,
    palette="RdYlGn",
    order=result_mol["mall"].value_counts().index  # 몰별로 개수 세기
)
plt.xticks(rotation=90)
plt.show()

<제로베이스 데이터 취업 스쿨>

김민지

이전 포스트

Ch5 유가 분석 01-13 (유가분석1-5)

다음 포스트

Ch6 Naver API 01-07 (Naver API 1-2)

Part 04. EDA/웹크롤링/파이썬프로그래밍

Ch5 유가 분석 01-13 (유가분석1-5)

Ch7 인구 분석 01-12 (인구분석 1-3)

0개의 댓글