<빅데이터 분석> 10. 기본-소비 데이터 분석과 승객 생존율 예측

정지인·2025년 6월 6일

✅ 1. 분석 준비

!pip install pandas

import pandas as pd
url='https://raw.githubusercontent.com/sehakflower/data/main/titanic.csv'
titanic_df= pd.read_csv(url,sep='\t')

# 컬럼명을 소문자로 바꾸기
new_columns=['passengerId', 'survived', 'pclass', 'name', 'sex', 'age', 'sibsp', 'parch',
'ticket', 'fare', 'cabin', 'embarked']
titanic_df.columns=new_columns

# 필요한 컬럼만 가져오기
titanic_df1=titanic_df[['survived', 'pclass', 'name', 'sex', 'age', 'sibsp', 'parch', 'fare']]

# 1. sex열을 gender로 바꾸고 여성은 0, 남성은 1로 바꾸기
tmp=[]
for each in titanic_df1['sex']:
  if each=='female':
    tmp.append(0)
  else:
    tmp.append(1)

titanic_df1['gender']=tmp
titanic_df1.drop(columns='sex', inplace=True) # sex 열 삭제하기

# 2. 호칭을 숫자형으로 바꾸기
condition=lambda x: x.split(',')[1].split('.')[0].strip()
titanic_df1['title']=titanic_df1['name'].map(condition) # name에서 호칭만 가져오기
Special=['Master', 'Don', 'Rev'] # 3개 호칭을 title 열의 Special로 대체
for each in Special:
  titanic_df1['title']=titanic_df1['title'].replace(each, 'Special')

titanic_df1=titanic_df1.drop('name', axis=1) # 사용하지 않는 name 열 삭제하기

# 숫자형으로 만드는 함수 만들기
def convert_title(x):
  if x=="Special":
    return 1
  else:
    return 0

titanic_df1["special_title"]=titanic_df1["title"].apply(convert_title)
# special_title 열 만들어서 special은 1, 나머지는 0으로 바꾸기
titanic_df1.drop('title', axis=1, inplace=True) # title 열 삭제하기
# 3. sibpar 열 만들어 동반자 수 더한 값 넣기
titanic_df1['sibpar']=titanic_df1['sibsp']+titanic_df1['parch']
titanic_df1.drop(['sibsp','parch'], axis =1, inplace=True)
# 사용하지 않는 sibsp, parch 열 삭제하기
# 4. avgfare 열 만들어 1인당 평균 탑승 요금 넣기
titanic_df1['avgfare']=titanic_df1['fare']/titanic_df1['sibpar']
titanic_df1['n_family']=titanic_df1['sibpar']+1
# n_family 열 만들어 동반자에 1을 더하여 0으로 나누어지지 않도록 함
titanic_df1['avgfare']=titanic_df1['fare']/titanic_df1['n_family'] # 1인당 평균 탑승 요금
titanic_df1=titanic_df1.drop(['fare','sibpar'], axis=1) # 필요 없는 열 삭제하기
# 5. 최종 분석 데이터 컬럼명만 추출하기
titanic_df1.rename(columns={'gender':'sex', 'special_title':'title', 'avgfare':'fare', 'n_family':'num_family'}, inplace=True) # 컬럼명 변경하기
titanic_df1=titanic_df1[['survived', 'pclass', 'sex', 'age', 'title', 'fare', 'num_family']]
# 컬럼 순서 다시 정하기
titanic_df1=titanic_df1.dropna()
# age에는 NaN 값이 있으므로 이를 삭제한 데이터를 분석에 사용
titanic_df1

raw=titanic_df1
np_raw=raw.values
type(np_raw)

train=np_raw[:100]
test=np_raw[100:]
y_train = [i[0] for i in train]
X_train = [j[1:] for j in train]
y_test = [i[0] for i in test]
X_test = [j[1:] for j in test]
len(X_train),len(y_train), len(y_test),len(X_test)

✅ 2. 의사결정나무

!pip3 install -U scikit-learn

from sklearn.tree import DecisionTreeClassifier
model=DecisionTreeClassifier(criterion='entropy',max_depth=3,min_samples_leaf=5).fit(X_train, y_train)
model.fit(X_train, y_train)
DecisionTreeClassifier(criterion='entropy', max_depth=3, min_samples_leaf=5)
print('Score:{}'.format(model.score(X_train, y_train)))
print('Score:{}'.format(model.score(X_test, y_test)))

!pip install graphviz

✅ 아래 코드가 실행되지 않는다면?

다음 경로에서 파일을 다운받고 설치하면 됩니다.
https://graphviz.org/download/
여기서는 graphviz-8.0.5 (64-bit) EXE installer [sha256] 항목을 설치하였다.

2.아나콘다설치후 주피터노트북을 실행한다면 아나콘다 프롬프트쉘에서 conda install python-graphviz 입력하여 실행한다.

환경변수에서 path도 함께 설정해주어야 합니다.
(1) 윈도우 검색란에 '환경'이라고 검색하고 Enter를 누르면 시스템 속성 대화상자가 나타난다.
(2) 하단의 [환경변수] 단추를 클릭하고 상단 목록에서 [path]항목을 선택한후 [편집] 단추를 클릭한다.
(3) 환경변수 편집 대화상자가 나타나면 [새로 만들기] 단추를 눌러 다음의 두 개의 경로를 추가한후 확인을 클릭한다.

pc에 대한 사용자 변수창 path변수에 아래를 추가한다.

시스템 변수창 path에 아래를 추가한다.

(4) 저장후 주피터 노트북을 재실행한 후 코드를 실행하면 오류 없이 잘 실행되는 것을 확인할수 있다.

from sklearn.tree import export_graphviz
export_graphviz(
 model,
 out_file="titanic.dot",
 feature_names=['pclass', 'sex', 'age', 'title', 'fare', 'num_family'],
 class_names=['0','1'],
 rounded=True,
 filled=True
)
import graphviz
with open("titanic.dot") as f:
 dot_graph=f.read()
dot=graphviz.Source(dot_graph)
dot.format='png'
dot.render(filename='titanic_tree', directory='image/decision_trees', cleanup=True)
dot

from sklearn.metrics import accuracy_score
y_pred=model.predict(X_test)
print("Test Accuracy is ", accuracy_score(y_test, y_pred)*100)

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_pred)

feature_names=['pclass', 'sex', 'age', 'title', 'fare', 'num_family']
Tom=[1,1,33,1,50,4]
Jane=[2,0,50,0,8,1]

model.predict_proba([Tom])

model.predict_proba([Jane])

✅ 3. 로지스틱 회귀분석

표준화하지 않은 데이터 이용하기

from sklearn.linear_model import LogisticRegression
log_reg=LogisticRegression(random_state=13, solver='liblinear', C=10.)
log_reg.fit(X_train, y_train)
LogisticRegression(C=10.0, random_state=13, solver='liblinear')
from sklearn.metrics import accuracy_score
pred=log_reg.predict(X_train)
accuracy_score(y_train, pred)

pred = log_reg.predict(X_test)
accuracy_score(y_test, pred)

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, pred)

✅ 표준화한 데이터 이용하기

!pip install seaborn

import seaborn as sns
sns.boxplot(data=titanic_df1[['pclass', 'sex', 'age', 'title', 'fare', 'num_family']])

X=titanic_df1
from sklearn.preprocessing import MinMaxScaler, StandardScaler
MMS=MinMaxScaler()
SS=StandardScaler()
SS.fit(X)
MMS.fit(X)
X_ss=SS.transform(X)
X_mms=MMS.transform(X)
X_ss_pd=pd.DataFrame(X_ss, columns=X.columns)
X_mms_pd=pd.DataFrame(X_mms, columns=X.columns)
X_ss_pd.head()

X_mms_pd.head()

sns.boxplot(data=X_mms_pd[['pclass','sex','age','title','fare','num_family']])

sns.boxplot(data=X_ss_pd[['pclass', 'sex', 'age', 'title', 'fare', 'num_family']])

y=raw['survived']
X=raw.drop(['survived'], axis=1)
X.head()

y.head()

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test=\
train_test_split(X, y, test_size=0.2, random_state=13)

import numpy as np
np.unique(y_train, return_counts=True)

X_out=X_mms_pd
X_train, X_test, y_train, y_test=\
train_test_split(X_out, y, test_size=0.2, random_state=13)

log_reg=LogisticRegression(random_state=13, solver='liblinear', C=10.)
log_reg.fit(X_train, y_train)

pred=log_reg.predict(X_test)
accuracy_score(y_test, pred)

X_out=X_ss_pd
X_train, X_test, y_train, y_test=\
train_test_split(X_out, y, test_size=0.2, random_state=13)
log_reg=LogisticRegression(random_state=13, solver='liblinear', C=10.)
log_reg.fit(X_train, y_train)
pred=log_reg.predict(X_test)
accuracy_score(y_test, pred)

log_reg.coef_

✅ 1. 머신러닝 기법에 적용하기

import pandas as pd
titanic_url='https://github.com/sehakflower/data/blob/main/titanic_1309.xlsx?raw=true'
titanic=pd.read_excel(titanic_url,sheet_name='total')
train_1000=titanic.iloc[:1000]
test_309=titanic.iloc[1000:]
train_1000.drop(['boat', 'body', 'home.dest'], axis=1, inplace=True)
test_309.drop(['boat', 'body', 'home.dest'], axis=1, inplace=True)
train_df=train_1000
test_df=test_309

# 1. name 열을 5개로 구분하고 숫자형 데이터로 바꾸기
total=[train_df, test_df]
titles={'Mr':1, 'Miss':2, 'Mrs':3, 'Master':4, 'Special':5} # 호칭에 따라 숫자로 표시하기
for dataset in total:
 dataset['title']=dataset.name.str.extract('([A-Za-z]+)\.', expand=False)
 dataset['title']=dataset['title'].replace(['Lady', 'Countess', 'Capt', 'Col',
'Don', 'Dr', 'Major', 'Rev', 'Sir',
'Jonkheer', 'Dona'], 'Special')
 dataset['title']=dataset['title'].replace('Mlle', 'Miss')
 dataset['title']=dataset['title'].replace('Ms', 'Miss')
 dataset['title']=dataset['title'].replace('Mme', 'Mrs')
 dataset['title']=dataset['title'].map(titles) # 타이틀을 숫자로 변경
 dataset['title']=dataset['title'].fillna(0) # NaN을 0으로 변경
train_df=train_df.drop(['name'], axis=1) # 필요 없는 name 열 삭제
test_df=test_df.drop(['name'], axis=1) # 필요 없는 name 열 삭제

# 2.age 열의 빈 값을 호칭별 중앙값으로 바꾸기
train_df['age'].fillna(train_df.groupby('title')['age'].transform('median'), inplace=True)
test_df['age'].fillna(test_df.groupby('title')['age'].transform('median'), inplace=True)
data=[train_df, test_df]
for dataset1 in data:
 dataset1['age']=dataset1['age'].astype(int)
 dataset1.loc[ dataset1['age']<=11, 'age']=0
 dataset1.loc[(dataset1['age']>11)&(dataset1['age']<=18), 'age']=1
 dataset1.loc[(dataset1['age']>18)&(dataset1['age']<=22), 'age']=2
 dataset1.loc[(dataset1['age']>22)&(dataset1['age']<=27), 'age']=3
 dataset1.loc[(dataset1['age']>27)&(dataset1['age']<=33), 'age']=4
 dataset1.loc[(dataset1['age']>33)&(dataset1['age']<=40), 'age']=5
 dataset1.loc[(dataset1['age']>40)&(dataset1['age']<=66), 'age']=6
 dataset1.loc[ dataset1['age']>66, 'age']=6
# 3. sex 열에서 'male':0, 'female':1로 바꾸기
data=[train_df, test_df]
# 일부 전처리된 train_df, test_df를 다시 업데이트해서 더한 후 숫자로 바꾸기 위함
sex_mapping={'male':0, 'female':1}
for dataset in data:
 dataset['sex']=dataset['sex'].map(sex_mapping)
# 4. embarked 열
for dataset in data:
 dataset['embarked']=dataset['embarked'].fillna('S')
embarked_mapping={'S':0, 'C':1, 'Q':2}
for dataset in data:
 dataset['embarked']=dataset['embarked'].map(embarked_mapping)
# 5. sibsp, parch 열
for dataset in data:
 dataset['sibpar']=dataset['sibsp']+dataset['parch']
 dataset.loc[dataset['sibpar']>0, 'n_alone']=0
 dataset.loc[dataset['sibpar']==0, 'n_alone']=1
 dataset['n_alone']=dataset['n_alone'].astype(int)
# 6. fare 열, 개인별 요금 나타내는 fare_person 열 추가하기
train_df["fare"].fillna(train_df.groupby('pclass')['fare'].transform('median'),
inplace=True)
test_df["fare"].fillna(test_df.groupby('pclass')['fare'].transform('median'),
inplace=True)

for dataset in data:
 dataset.loc[ dataset['fare']<=20, 'fare']=1
 dataset.loc[(dataset['fare']>20)&(dataset['fare']<=30), 'fare']=2
 dataset.loc[(dataset['fare']>30)&(dataset['fare']<=50), 'fare']=3
 dataset.loc[(dataset['fare']>50)&(dataset['fare']<=100), 'fare']=4
 dataset.loc[ dataset['fare']>100, 'fare']=5
for dataset1 in data:
 dataset1['fare_person']=dataset1['fare']/(dataset1['sibpar']+1)
 dataset1['fare_person']=dataset1['fare_person'].astype(int)
# 7. 최종 데이터
X_columns=['pclass', 'sex', 'age', 'embarked', 'title', 'sibpar', 'n_alone', 'fare_person']
y_column=['survived']
X_train=train_df[X_columns]
y_train=train_df[y_column]
X_test=test_df[X_columns]
y_test=test_df[y_column]

✅ 필요한 라이브러리

import numpy as np # 선형대수
import pandas as pd # 데이터 처리
# 머신러닝 알고리즘
from sklearn import linear_model
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.naive_bayes import GaussianNB
# 시각화
import seaborn as sns
%matplotlib inline
from matplotlib import pyplot as plt
from matplotlib import style

✅ 의사결정나무

decision_tree=DecisionTreeClassifier()
decision_tree.fit(X_train, y_train)
Y_pred=decision_tree.predict(X_test)
train_acc_decision_tree=round(decision_tree.score(X_train, y_train)*100, 2)
test_acc_decision_tree=round(decision_tree.score(X_test, y_test)*100, 2)
train_acc_decision_tree,test_acc_decision_tree

✅ 랜덤 포레스트

random_forest=RandomForestClassifier(n_estimators=100)
random_forest.fit(X_train, y_train)
Y_prediction=random_forest.predict(X_test)
random_forest.score(X_train, y_train)
train_acc_random_forest=round(random_forest.score(X_train, y_train)*100, 2)
test_acc_random_forest=round(random_forest.score(X_test, y_test)*100, 2)
train_acc_random_forest,test_acc_random_forest

✅ 로지스틱 회귀분석

log_reg=LogisticRegression()
log_reg.fit(X_train, y_train)
Y_pred=log_reg.predict(X_test)
train_acc_log=round(log_reg.score(X_train, y_train)*100, 2)
test_acc_log=round(log_reg.score(X_test, y_test)*100, 2)
train_acc_log, test_acc_log

✅ 나이브 베이지안

gaussian=GaussianNB()
gaussian.fit(X_train, y_train)
Y_pred=gaussian.predict(X_test)
train_acc_gaussian=round(gaussian.score(X_train, y_train)*100, 2)
test_acc_gaussian=round(gaussian.score(X_test, y_test)*100, 2)
train_acc_gaussian,test_acc_gaussian

✅ K-Nearest Neighbor

knn=KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)
Y_pred=knn.predict(X_test)
train_acc_knn=round(knn.score(X_train, y_train)*100, 2)
test_acc_knn=round(knn.score(X_test, y_test)*100, 2)
train_acc_knn,test_acc_knn

✅ SVM 모델

svc=LinearSVC()
svc.fit(X_train, y_train)
Y_pred=svc.predict(X_test)
train_acc_svc=round(svc.score(X_train, y_train)*100, 2)
test_acc_svc=round(svc.score(X_test, y_test)*100, 2)
train_acc_svc,test_acc_svc

✅ 퍼셉트론 모델

perceptron=Perceptron(max_iter=5)
perceptron.fit(X_train, y_train)
Y_pred=perceptron.predict(X_test)
train_acc_perceptron=round(perceptron.score(X_train, y_train)*100, 2)
test_acc_perceptron=round(perceptron.score(X_test, y_test)*100, 2)
train_acc_perceptron,test_acc_perceptron

✅ 확률적 경사하강법 방법

sgd=linear_model.SGDClassifier(max_iter=5, tol=None)
sgd.fit(X_train, y_train)
Y_pred=sgd.predict(X_test)
sgd.score(X_train, y_train)
train_acc_sgd=round(sgd.score(X_train, y_train)*100, 2)
test_acc_sgd=round(sgd.score(X_test, y_test)*100, 2)
train_acc_sgd,test_acc_sgd

✅ 2. 모델 비교

###모델의 정확도 비교

results = pd.DataFrame({
 'Model': ['Support Vector Machines', 'KNN', 'Logistic Regression', 'Random Forest','Naive Bayes', 'Perceptron', 'Stochastic Gradient Decent', 'Decision Tree'],
 'train_Score': [train_acc_svc, train_acc_knn, train_acc_log,train_acc_random_forest,
train_acc_gaussian, train_acc_perceptron, train_acc_sgd, train_acc_decision_tree],
 'test_Score': [test_acc_svc, test_acc_knn,test_acc_log, test_acc_random_forest, test_acc_gaussian, test_acc_perceptron, test_acc_sgd, test_acc_decision_tree]})
result_df=results.sort_values(by='train_Score', ascending=False)
result_df=result_df.set_index('Model')
result_df.head(10)

✅ 랜덤 포레스트 모델의 교차검증 실습

from sklearn.model_selection import cross_val_score
rf=RandomForestClassifier(n_estimators=100)
scores=cross_val_score(rf, X_train, y_train, cv=10, scoring="accuracy")
print("Scores:", scores)
print("Mean:", scores.mean())
print("Standard Deviation:", scores.std())

정지인

멋쟁이사자 13기 백엔드

이전 포스트

<빅데이터 분석> 9. 선형대수와 신경망 기초

다음 포스트