ML_cross_validation

교차검증

Straified KFold 교차검증

• 학습용 데이터를 3~5개로 나눈 후 각각의 데이터들을 검증한다
• 3~5개로 검증한 정확도(accuracy_score)의 평균을 최종 정확도로 한다
• 학습용 데이터를 나눌 때 각 클래스의 분포가 다르다면 이 분포 상태를 유지해야 된다
  

결과론적 학습/교차검증/테스트 과정

와인 데이터 불러오기 및 결정나무(DecisionTree) 생성 후 학습 및 결과 재확인

KFord로 확인

from sklearn.model_selection import KFold

kfold = KFold(n_splits=5)
wine_tree_cv = DecisionTreeClassifier(max_depth=2, random_state=13)

KFold는 index를 반환

for train_idx, validation_idx in kfold.split(X):
    print(len(train_idx), len(validation_idx))

5등분된(n_splits = 5로 설정된) 각각의 fold의 학습 후 accuracy_score

• 각 정확도(accuracy_score)의 평균을 대표값으로 한다(단, 분산이 크지 않다는 조건하에)

cv_accuracy = []

for train_idx, validation_idx in kfold.split(X):
    X_train, X_validation = X.iloc[train_idx], X.iloc[validation_idx]
    y_train, y_validation = y.iloc[train_idx], y.iloc[validation_idx]
    wine_tree_cv.fit(X_train, y_train)
    pred = wine_tree_cv.predict(X_validation)
    cv_accuracy.append(accuracy_score(y_validation, pred))

cv_accuracy

np.mean(cv_accuracy)

Straified KFold

from sklearn.model_selection import StratifiedKFold

skfold = StratifiedKFold(n_splits=5)
wine_tree_cv = DecisionTreeClassifier(max_depth=2, random_state=13)

cv_accuracy = []

for train_idx, validation_idx in skfold.split(X, y):
    X_train, X_validation = X.iloc[train_idx], X.iloc[validation_idx]
    y_train, y_validation = y.iloc[train_idx], y.iloc[validation_idx]
    wine_tree_cv.fit(X_train, y_train)
    pred = wine_tree_cv.predict(X_validation)
    cv_accuracy.append(accuracy_score(y_validation, pred))

cv_accuracy

np.mean(cv_accuracy)

교차검증 한번에 하기 : cross_val_score

• cross_val_score(estimator(여기선 결정나무), feature, label, scoring = , cv=)
• scoring = : 예측 성능 평가 지표 (ex. accurency)
• cv = : 데이터를 나눈 수, 교차 검증 폴드 수

from sklearn.model_selection import cross_val_score

X = wine.drop(['taste', 'quality'], axis=1) # feature
y = wine['taste'] # label

skfold = StratifiedKFold(n_splits=5)
wine_tree_cv = DecisionTreeClassifier(max_depth=2, random_state=13)

cross_val_score(wine_tree_cv, X, y, scoring=None, cv=skfold)

depth를 높인다고 무조건 ACC가 좋아지진 않는다

wine_tree_cv = DecisionTreeClassifier(max_depth=5, random_state=13)

cross_val_score(wine_tree_cv, X, y, scoring=None, cv=skfold)

train score와 함께 보기 : cross_validate

from sklearn.model_selection import cross_validate

X = wine.drop(['taste', 'quality'], axis=1) # feature
y = wine['taste'] # label

skfold = StratifiedKFold(n_splits=5)
wine_tree_cv = DecisionTreeClassifier(max_depth=2, random_state=13)

cross_validate(wine_tree_cv, X, y, scoring=None, cv=skfold, return_train_score=True)