협업필터링 (식당 평점 예측)

• 특정 사이트에 평점이 매겨지지 않은 식당의 평점을 협업필터링으로 예측해보았다.
• 가령 구글, 네이버 등엔 있는데, 카카오에는 평점이 없는 식당이 있을 때,
• 카카오에선 몇 점이 부여될까를 예측해보는 것이다.
• 표본 수(사이트 수)가 적어 제대로 된 예측결과가 나오진 않았지만 협업필터링 프로세스를 살펴볼 수 있었다.

import pandas as pd
import numpy as np

mango = pd.read_csv('mango_revise.csv')
kakao = pd.read_csv('kao_revise.csv')
google = pd.read_csv('google_stores.csv')

print(mango.shape)
print(kakao.shape)
print(google.shape)

(200, 7)
(94, 5)
(261, 6)

# Id 생성
mango['Site'] = 'mango'
mango.head()

kakao.head()

google.head()

사용자-아이템 평점 행렬로 변환

# 필요한 컬럼만 추출
mango2 = mango2[['Site', 'Title', 'Point']]
kakao = kakao[['Site_kao', 'Title', 'Point_kao']]
google = google[['Site', 'Title', 'Point']]

mango2.head()

kakao.head()

google.head()

kakao.rename(columns={'Site_kao':'Site', 'Point_kao':'Point'}, inplace=True)
kakao.head()

ratings = pd.concat([mango2, kakao, google]).reset_index(drop=True)
ratings

ratings.rename(columns={'Site':'userId', 'Point':'rating'}, inplace=True)
ratings

stores = mango[['Title', 'Type']]
stores # movies

# columns='title' 로 title 컬럼으로 pivot 수행
ratings_matrix = ratings.pivot_table('rating', index='userId', columns='Title')
ratings_matrix

ratings_stores = pd.merge(ratings, stores, on='Title')

# 중복 행 제거
ratings_stores = ratings_stores.drop_duplicates(['userId','Title']).reset_index(drop=True) 
ratings_stores
ratings_stores

# columns='title' 로 title 컬럼으로 pivot 수행
ratings_matrix = ratings_stores.pivot_table('rating', index='userId', columns='Title')
ratings_matrix

# NaN 값을 모두 0 으로 변환
ratings_matrix = ratings_matrix.fillna(0)
ratings_matrix

-> 사용자-아이템 행렬이 만들어졌다.

# 아이템-사용자 행렬로 transpose 한다.
ratings_matrix_T = ratings_matrix.transpose()  # 전치 행렬

print(ratings_matrix_T.shape)
ratings_matrix_T.head(3)

(193, 3)

# 영화와 영화들 간 코사인 유사도 산출
from sklearn.metrics.pairwise import cosine_similarity

item_sim = cosine_similarity(ratings_matrix_T, ratings_matrix_T)

# cosine_similarity() 로 반환된 넘파이 행렬을 영화명을 매핑하여 DataFrame으로 변환
item_sim_df = pd.DataFrame(data=item_sim, index=ratings_matrix.columns,
                          columns=ratings_matrix.columns)

print(item_sim_df.shape)
item_sim_df.head(3)

(193, 193)

# 유사 확인해보기
item_sim_df["고래사"].sort_values(ascending=False)[1:10]

Title
그라치에       1.000000
유일한식탁      1.000000
어바웃제이      1.000000
나이브브류어스    0.999934
베르크로스터스    0.999928
삼진어묵       0.999928
바로해장       0.999924
다케다야       0.999924
국이네낙지볶음    0.999737
Name: 고래사, dtype: float64

# 평점 벡터(행 벡터)와 유사도 벡터(열 벡터)를 내적(dot)해서 예측 평점을 계산하는 함수 정의
def predict_rating(ratings_arr, item_sim_arr):
    ratings_pred = ratings_arr.dot(item_sim_arr)/ np.array([np.abs(item_sim_arr).sum(axis=1)])
    return ratings_pred

ratings_pred = predict_rating(ratings_matrix.values , item_sim_df.values)
ratings_pred

array([[0.33567509, 0.33567509, 0.33567509, 0.35889099, 0.36302472,
        0.33567509, 0.33567509, 0.33567509, 0.36074923, 0.33567509,
        0.60970209, 0.36162415, 0.33567509, 0.33567509, 0.33567509,
        0.36159255, 0.33567509, 0.36074923, 0.36162415, 0.35943587,
        0.35741782, 0.33567509, 0.33567509, 0.62851446, 0.35803633,
        0.35905444, 0.36116826, 0.56840223, 0.35756435, 0.36165734,
        0.35939193, 0.35294904, 0.35814517, 0.36030291, 0.35487799,
        0.33567509, 0.35896125, 0.33567509, 0.3598722 , 0.35691247,
        0.58276719, 0.35355391, 0.35988228, 0.35936824, 0.33567509,
        0.62869203, 0.35845565, 0.33567509, 0.33567509, 0.36209664,
        0.35694639, 0.58349083, 0.33567509, 0.33567509, 0.35899382,
        0.56235501, 0.35371873, 0.33567509, 0.35741782, 0.35986162,
        0.33567509, 0.33567509, 0.36501339, 0.33567509, 0.36162415,
        0.33567509, 0.33567509, 0.33567509, 0.33567509, 0.36295771,
        0.33567509, 0.33567509, 0.33567509, 0.35896125, 0.33567509,
        0.36253006, 0.36030291, 0.35769672, 0.36209664, 0.35708585,
        0.35280292, 0.57801209, 0.33567509, 0.33567509, 0.33567509,
        0.36118961, 0.33567509, 0.36295771, 0.33567509, 0.33567509,
        0.59219754, 0.36402742, 0.35889099, 0.61496548, 0.33567509,
        0.3598919 , 0.35659007, 0.33567509, 0.33567509, 0.33567509,
        0.33567509, 0.33567509, 0.36118961, 0.33567509, 0.33567509,
        0.33567509, 0.33567509, 0.35936824, 0.33567509, 0.33567509,
        0.33567509, 0.35700144, 0.36201139, 0.33567509, 0.62834575,
        0.36205298, 0.33567509, 0.35741782, 0.35781688, 0.61496548,
        0.33567509, 0.33567509, 0.33567509, 0.33567509, 0.33567509,
        0.33567509, 0.35840693, 0.3607606 , 0.3559971 , 0.59463007,
        0.33567509, 0.35650285, 0.33567509, 0.35983884, 0.33567509,
        0.33567509, 0.64052246, 0.33567509, 0.35427113, 0.55738783,
        0.33567509, 0.3607606 , 0.35691247, 0.33567509, 0.33567509,
        0.35371873, 0.35947577, 0.35814517, 0.35854593, 0.63491314,
        0.33567509, 0.33567509, 0.35167319, 0.33567509, 0.58303518,
        0.35481525, 0.33567509, 0.33567509, 0.33567509, 0.33567509,
        0.33567509, 0.35791593, 0.33567509, 0.35775822, 0.35650285,
        0.36337972, 0.35896125, 0.33567509, 0.35896125, 0.33567509,
        0.36430377, 0.33567509, 0.56154835, 0.33567509, 0.33567509,
        0.33567509, 0.33567509, 0.33567509, 0.33567509, 0.33567509,
        0.33567509, 0.58200684, 0.56799497, 0.33567509, 0.33567509,
        0.33567509, 0.3543857 , 0.62851446, 0.35983884, 0.35423195,
        0.35409759, 0.35700144, 0.33567509],
       [1.50234426, 1.50234426, 1.50234426, 2.08292339, 2.18629908,
        1.50234426, 1.50234426, 1.50234426, 2.12939415, 1.50234426,
        1.53210213, 2.15127393, 1.50234426, 1.50234426, 1.50234426,
        2.15048367, 1.50234426, 2.12939415, 2.15127393, 2.09654983,
        2.04608281, 1.50234426, 1.50234426, 1.53414505, 2.06155035,
        2.0870111 , 2.13987315, 2.02845143, 2.0497472 , 2.1521039 ,
        2.09545085, 1.93432825, 2.06427225, 2.11823257, 1.98256697,
        1.50234426, 2.08468053, 1.50234426, 2.1074614 , 2.03344496,
        2.09529069, 1.94945468, 2.10771347, 2.09485849, 1.50234426,
        1.53416433, 2.07203648, 1.50234426, 1.50234426, 2.16308989,
        2.0342932 , 1.97473892, 1.50234426, 1.50234426, 2.08549511,
        2.05692442, 1.95357654, 1.50234426, 2.04608281, 2.10719694,
        1.50234426, 1.50234426, 2.23603143, 1.50234426, 2.15127393,
        1.50234426, 1.50234426, 1.50234426, 1.50234426, 2.18462344,
        1.50234426, 1.50234426, 1.50234426, 2.08468053, 1.50234426,
        2.17392876, 2.11823257, 2.05305736, 2.16308989, 2.03778096,
        1.93067398, 2.03793209, 1.50234426, 1.50234426, 1.50234426,
        2.14040695, 1.50234426, 2.18462344, 1.50234426, 1.50234426,
        2.0594169 , 2.21137438, 2.08292339, 1.5326737 , 1.50234426,
        2.10795402, 2.0253826 , 1.50234426, 1.50234426, 1.50234426,
        1.50234426, 1.50234426, 2.14040695, 1.50234426, 1.50234426,
        1.50234426, 1.50234426, 2.09485849, 1.50234426, 1.50234426,
        1.50234426, 2.03566985, 2.16095793, 1.50234426, 1.53412673,
        2.16199796, 1.50234426, 2.04608281, 2.05606229, 1.5326737 ,
        1.50234426, 1.50234426, 1.50234426, 1.50234426, 1.50234426,
        1.50234426, 2.0708182 , 2.12967838, 2.01055359, 2.0830747 ,
        1.50234426, 2.02320139, 1.50234426, 2.10662706, 1.50234426,
        1.50234426, 1.53544906, 1.50234426, 1.96739086, 2.04216679,
        1.50234426, 2.12967838, 2.03344496, 1.50234426, 1.50234426,
        1.95357654, 2.09754766, 2.06427225, 2.07429439, 1.53483991,
        1.50234426, 1.50234426, 1.90242192, 1.50234426, 2.09501473,
        1.98099798, 1.50234426, 1.50234426, 1.50234426, 1.50234426,
        1.50234426, 2.05853926, 1.50234426, 2.05459528, 2.02320139,
        2.19517679, 2.08468053, 1.50234426, 2.08468053, 1.50234426,
        2.21828542, 1.50234426, 2.06719049, 1.50234426, 1.50234426,
        1.50234426, 1.50234426, 1.50234426, 1.50234426, 1.50234426,
        1.50234426, 2.05096151, 2.04773167, 1.50234426, 1.50234426,
        1.50234426, 1.97025585, 1.53414505, 2.10662706, 1.96641105,
        1.96305105, 2.03566985, 1.50234426],
       [4.06516071, 4.06516071, 4.06516071, 4.06331156, 4.06298231,
        4.06516071, 4.06516071, 4.06516071, 4.06316355, 4.06516071,
        4.06437318, 4.06309386, 4.06516071, 4.06516071, 4.06516071,
        4.06309638, 4.06516071, 4.06316355, 4.06309386, 4.06326816,
        4.0634289 , 4.06516071, 4.06516071, 4.06431912, 4.06337963,
        4.06329854, 4.06313018, 4.06294756, 4.06341723, 4.06309122,
        4.06327166, 4.06378484, 4.06337097, 4.0631991 , 4.0636312 ,
        4.06516071, 4.06330596, 4.06516071, 4.06323341, 4.06346915,
        4.06270499, 4.06373666, 4.0632326 , 4.06327355, 4.06516071,
        4.06431861, 4.06334624, 4.06516071, 4.06516071, 4.06305623,
        4.06346645, 4.06307487, 4.06516071, 4.06516071, 4.06330337,
        4.06287512, 4.06372353, 4.06516071, 4.0634289 , 4.06323425,
        4.06516071, 4.06516071, 4.06282391, 4.06516071, 4.06309386,
        4.06516071, 4.06516071, 4.06516071, 4.06516071, 4.06298765,
        4.06516071, 4.06516071, 4.06516071, 4.06330596, 4.06516071,
        4.06302171, 4.0631991 , 4.06340668, 4.06305623, 4.06345534,
        4.06379648, 4.06289393, 4.06516071, 4.06516071, 4.06516071,
        4.06312848, 4.06516071, 4.06298765, 4.06516071, 4.06516071,
        4.06279166, 4.06290244, 4.06331156, 4.06435806, 4.06516071,
        4.06323184, 4.06349483, 4.06516071, 4.06516071, 4.06516071,
        4.06516071, 4.06516071, 4.06312848, 4.06516071, 4.06516071,
        4.06516071, 4.06516071, 4.06327355, 4.06516071, 4.06516071,
        4.06516071, 4.06346206, 4.06306302, 4.06516071, 4.0643196 ,
        4.06305971, 4.06516071, 4.0634289 , 4.06339711, 4.06435806,
        4.06516071, 4.06516071, 4.06516071, 4.06516071, 4.06516071,
        4.06516071, 4.06335012, 4.06316265, 4.06354206, 4.06271254,
        4.06516071, 4.06350178, 4.06516071, 4.06323606, 4.06516071,
        4.06516071, 4.06428461, 4.06516071, 4.06367953, 4.06293324,
        4.06516071, 4.06316265, 4.06346915, 4.06516071, 4.06516071,
        4.06372353, 4.06326498, 4.06337097, 4.06333904, 4.06430073,
        4.06516071, 4.06516071, 4.06388646, 4.06516071, 4.06270516,
        4.06363619, 4.06516071, 4.06516071, 4.06516071, 4.06516071,
        4.06516071, 4.06338923, 4.06516071, 4.06340179, 4.06350178,
        4.06295403, 4.06330596, 4.06516071, 4.06330596, 4.06516071,
        4.06288043, 4.06516071, 4.06284552, 4.06516071, 4.06516071,
        4.06516071, 4.06516071, 4.06516071, 4.06516071, 4.06516071,
        4.06516071, 4.06284361, 4.06288915, 4.06516071, 4.06516071,
        4.06516071, 4.06367041, 4.06431912, 4.06323606, 4.06368265,
        4.06369336, 4.06346206, 4.06516071]])

# 데이터프레임으로 변환
ratings_pred_matrix = pd.DataFrame(data=ratings_pred, index= ratings_matrix.index,
                                   columns = ratings_matrix.columns)
print(ratings_pred_matrix.shape)
ratings_pred_matrix.head(10)

(3, 193)

from sklearn.metrics import mean_squared_error

# 사용자가 평점을 부여한 영화에 대해서만 예측 성능 평가 MSE 를 구함. 
def get_mse(pred, actual):
    # Ignore nonzero terms.
    pred = pred[actual.nonzero()].flatten()
    actual = actual[actual.nonzero()].flatten()
    return mean_squared_error(pred, actual)

print('아이템 기반 모든 인접 이웃 MSE: ', get_mse(ratings_pred, ratings_matrix.values ))

아이템 기반 모든 인접 이웃 MSE:  1.755633561085019

top-n 유사도를 가진 데이터들에 대해서만 예측 평점 계산

def predict_rating_topsim(ratings_arr, item_sim_arr, n=20):
    # 사용자-아이템 평점 행렬 크기만큼 0으로 채운 예측 행렬 초기화
    pred = np.zeros(ratings_arr.shape)

    # 사용자-아이템 평점 행렬의 열 크기만큼 Loop 수행. 
    for col in range(ratings_arr.shape[1]):
        # 유사도 행렬에서 유사도가 큰 순으로 n개 데이터 행렬의 index 반환
        top_n_items = [np.argsort(item_sim_arr[:, col])[:-n-1:-1]]
        # 개인화된 예측 평점을 계산
        for row in range(ratings_arr.shape[0]):
            pred[row, col] = item_sim_arr[col, :][top_n_items].dot(ratings_arr[row, :][top_n_items].T) 
            pred[row, col] /= np.sum(np.abs(item_sim_arr[col, :][top_n_items]))        
    return pred

# 실행시간 2분 정도 걸림
ratings_pred = predict_rating_topsim(ratings_matrix.values , item_sim_df.values, n=20)
print('아이템 기반 인접 TOP-20 이웃 MSE: ', get_mse(ratings_pred, ratings_matrix.values ))

# 계산된 예측 평점 데이터는 DataFrame으로 재생성
ratings_pred_matrix = pd.DataFrame(data=ratings_pred, index= ratings_matrix.index,
                                   columns = ratings_matrix.columns)

아이템 기반 인접 TOP-20 이웃 MSE:  0.10567727070048756


C:\Users\dyj42\AppData\Local\Temp/ipykernel_9080/2382315358.py:11: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  pred[row, col] = item_sim_arr[col, :][top_n_items].dot(ratings_arr[row, :][top_n_items].T)
C:\Users\dyj42\AppData\Local\Temp/ipykernel_9080/2382315358.py:12: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  pred[row, col] /= np.sum(np.abs(item_sim_arr[col, :][top_n_items]))

user_rating_id = ratings_matrix.loc['kakao']
user_rating_id[user_rating_id > 0].sort_values(ascending=False)[:5]

Title
모루비         5.0
봉샌드         5.0
파크하얏트라운지    4.7
쿠루미과자점      4.6
문화공감 수정     4.5
Name: kakao, dtype: float64

평점이 매겨지지 않은 식당의 평점을 예측해보자

def get_unseen_movies(ratings_matrix, userId):
    # userId로 입력받은 사용자의 모든 식당정보 추출하여 Series로 반환함. 
    # 반환된 user_rating 은 식당명(title)을 index로 가지는 Series 객체임. 
    user_rating = ratings_matrix.loc[userId,:]
    
    # user_rating이 0보다 크면 기존에 있는 식당임. 대상 index를 추출하여 list 객체로 만듬
    already_seen = user_rating[ user_rating > 0].index.tolist()
    
    # 모든 식당명을 list 객체로 만듬. 
    movies_list = ratings_matrix.columns.tolist()
    
    # list comprehension으로 already_seen에 해당하는 movie는 movies_list에서 제외함. 
    unseen_list = [ movie for movie in movies_list if movie not in already_seen]
    
    return unseen_list

# pred_df : 앞서 계산된 영화 별 예측 평점
# unseen_list : 사용자가 보지 않은 영화들
# top_n : 상위 n개를 가져온다.

def recomm_movie_by_userid(pred_df, userId, unseen_list, top_n=10):
    # 예측 평점 DataFrame에서 사용자id index와 unseen_list로 들어온 영화명 컬럼을 추출하여
    # 가장 예측 평점이 높은 순으로 정렬함. 
    recomm_movies = pred_df.loc[userId, unseen_list].sort_values(ascending=False)[:top_n]
    return recomm_movies

# 평점이 매겨지지 않은 식당 추출   
unseen_list = get_unseen_movies(ratings_matrix, 'kakao')

# 아이템 기반의 인접 이웃 협업 필터링으로 식당 평점 예측 
recomm_movies = recomm_movie_by_userid(ratings_pred_matrix, 'kakao', unseen_list, top_n=5)

# 평점 데이타를 DataFrame으로 생성. 
recomm_movies = pd.DataFrame(data=recomm_movies.values,index=recomm_movies.index, columns=['pred_score'])
recomm_movies