ᄂ 😄 [13 일차] : Assignment - EX02. 손글씨 분류하기

백건·2022년 1월 21일

[AIFFEL] 아이펠 인공지능 전문가 과정

목록 보기

18/32

내용	상태	기타
손글씨분류하기	제출완료	깃허브

[E-02] 과제

손글씨 분류하기

데이터

데이터 준비

# 필요한 모듈 Import
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

digits = load_digits()  # 인스턴스 = 클래스()
digits_data = digits.data
digits_label = digits.target

train, Test 데이터 분리

X_train, X_test, y_train, y_test = train_test_split(digits_data, 
                                                    digits_label, 
                                                    test_size=0.2, 
                                                    random_state=13)

Model

Decision Tree Model

from sklearn.tree import DecisionTreeClassifier
decision_tree = DecisionTreeClassifier(random_state=13)
decision_tree.fit(X_train, y_train)
y_pred = decision_tree.predict(X_test)

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


# 정확도(Accuracy) 확인
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
accuracy

              precision    recall  f1-score   support

           0       0.97      0.93      0.95        30
           1       0.83      0.78      0.81        37
           2       0.87      0.87      0.87        38
           3       0.79      0.84      0.82        37
           4       0.89      0.87      0.88        38
           5       0.89      0.93      0.91        45
           6       0.95      0.93      0.94        42
           7       0.96      0.92      0.94        26
           8       0.93      0.79      0.86        34
           9       0.72      0.88      0.79        33

    accuracy                           0.88       360
   macro avg       0.88      0.87      0.88       360
weighted avg       0.88      0.88      0.88       360

[[28  0  1  0  0  0  0  0  0  1]
 [ 0 29  2  2  2  0  0  0  0  2]
 [ 0  1 33  1  1  0  1  0  1  0]
 [ 0  1  0 31  0  1  0  0  1  3]
 [ 0  0  1  1 33  1  0  1  0  1]
 [ 0  0  0  0  1 42  0  0  0  2]
 [ 0  0  0  2  0  1 39  0  0  0]
 [ 0  2  0  0  0  0  0 24  0  0]
 [ 0  2  0  1  0  1  1  0 27  2]
 [ 1  0  1  1  0  1  0  0  0 29]]





0.875

Random Forest 사용

from sklearn.ensemble import RandomForestClassifier

random_forest = RandomForestClassifier(random_state=32)
random_forest.fit(X_train, y_train)
y_pred = random_forest.predict(X_test)

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


# 정확도(Accuracy) 확인
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
accuracy

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        30
           1       0.90      1.00      0.95        37
           2       0.97      1.00      0.99        38
           3       1.00      0.95      0.97        37
           4       0.97      1.00      0.99        38
           5       1.00      0.98      0.99        45
           6       1.00      0.98      0.99        42
           7       0.96      0.96      0.96        26
           8       0.93      0.79      0.86        34
           9       0.89      0.97      0.93        33

    accuracy                           0.96       360
   macro avg       0.96      0.96      0.96       360
weighted avg       0.97      0.96      0.96       360

[[30  0  0  0  0  0  0  0  0  0]
 [ 0 37  0  0  0  0  0  0  0  0]
 [ 0  0 38  0  0  0  0  0  0  0]
 [ 0  0  0 35  0  0  0  0  1  1]
 [ 0  0  0  0 38  0  0  0  0  0]
 [ 0  0  0  0  0 44  0  0  0  1]
 [ 0  1  0  0  0  0 41  0  0  0]
 [ 0  0  0  0  0  0  0 25  0  1]
 [ 0  3  1  0  1  0  0  1 27  1]
 [ 0  0  0  0  0  0  0  0  1 32]]





0.9638888888888889

SVM Model

from sklearn import svm
svm_model = svm.SVC()
svm_model.fit(X_train, y_train)
y_pred = svm_model.predict(X_test)

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


# 정확도(Accuracy) 확인
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
accuracy

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        30
           1       0.93      1.00      0.96        37
           2       0.97      1.00      0.99        38
           3       1.00      0.97      0.99        37
           4       1.00      1.00      1.00        38
           5       1.00      0.98      0.99        45
           6       1.00      1.00      1.00        42
           7       1.00      0.96      0.98        26
           8       1.00      0.91      0.95        34
           9       0.94      1.00      0.97        33

    accuracy                           0.98       360
   macro avg       0.98      0.98      0.98       360
weighted avg       0.98      0.98      0.98       360

[[30  0  0  0  0  0  0  0  0  0]
 [ 0 37  0  0  0  0  0  0  0  0]
 [ 0  0 38  0  0  0  0  0  0  0]
 [ 0  0  1 36  0  0  0  0  0  0]
 [ 0  0  0  0 38  0  0  0  0  0]
 [ 0  0  0  0  0 44  0  0  0  1]
 [ 0  0  0  0  0  0 42  0  0  0]
 [ 0  0  0  0  0  0  0 25  0  1]
 [ 0  3  0  0  0  0  0  0 31  0]
 [ 0  0  0  0  0  0  0  0  0 33]]





0.9833333333333333

SGD Model 사용하기

from sklearn.linear_model import SGDClassifier
sgd_model = SGDClassifier()
sgd_model.fit(X_train, y_train)
y_pred = sgd_model.predict(X_test)

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


# 정확도(Accuracy) 확인
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
accuracy

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        30
           1       0.80      0.97      0.88        37
           2       1.00      0.95      0.97        38
           3       1.00      0.92      0.96        37
           4       1.00      1.00      1.00        38
           5       1.00      0.91      0.95        45
           6       0.98      1.00      0.99        42
           7       0.93      0.96      0.94        26
           8       0.78      0.85      0.82        34
           9       1.00      0.88      0.94        33

    accuracy                           0.94       360
   macro avg       0.95      0.94      0.94       360
weighted avg       0.95      0.94      0.95       360

[[30  0  0  0  0  0  0  0  0  0]
 [ 0 36  0  0  0  0  0  0  1  0]
 [ 0  2 36  0  0  0  0  0  0  0]
 [ 0  0  0 34  0  0  0  1  2  0]
 [ 0  0  0  0 38  0  0  0  0  0]
 [ 0  3  0  0  0 41  1  0  0  0]
 [ 0  0  0  0  0  0 42  0  0  0]
 [ 0  0  0  0  0  0  0 25  1  0]
 [ 0  4  0  0  0  0  0  1 29  0]
 [ 0  0  0  0  0  0  0  0  4 29]]





0.9444444444444444

Logistic Regression Model


from sklearn.linear_model import LogisticRegression
logistic_model = LogisticRegression()
logistic_model.fit(X_train, y_train)
y_pred = logistic_model.predict(X_test)

print(classification_report(y_test, y_pred))

print("######정답과 오답을 구분해서 확인#######")
print(confusion_matrix(y_test, y_pred))


# 정확도(Accuracy) 확인
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
accuracy

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        30
           1       0.85      0.92      0.88        37
           2       1.00      0.95      0.97        38
           3       0.97      0.92      0.94        37
           4       0.95      1.00      0.97        38
           5       0.98      0.96      0.97        45
           6       1.00      0.98      0.99        42
           7       0.96      0.92      0.94        26
           8       0.97      0.82      0.89        34
           9       0.82      1.00      0.90        33

    accuracy                           0.95       360
   macro avg       0.95      0.95      0.95       360
weighted avg       0.95      0.95      0.95       360

######정답과 오답을 구분해서 확인#######
[[30  0  0  0  0  0  0  0  0  0]
 [ 0 34  0  0  1  0  0  1  0  1]
 [ 0  2 36  0  0  0  0  0  0  0]
 [ 0  0  0 34  0  0  0  0  1  2]
 [ 0  0  0  0 38  0  0  0  0  0]
 [ 0  0  0  1  0 43  0  0  0  1]
 [ 0  1  0  0  0  0 41  0  0  0]
 [ 0  0  0  0  0  1  0 24  0  1]
 [ 0  3  0  0  1  0  0  0 28  2]
 [ 0  0  0  0  0  0  0  0  0 33]]


/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:818: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG,





0.9472222222222222

모델 선정

정리

Recall : 실제로 True인 데이터를 모델이 True 라고 인식한 데이터수
Precision : 모델이 True로 예측한 데이터 중 실제로 True인 데이터수
F1 Score : Recall 값과 Precision의 값과 차이가 적을 수록 노이즈가 적은 것.

SVM Model

Precision : 0.98
Recall : 0.98
F1 score : 0.98
Accuracy :0.983
우선 고려 항목 :

다른 모델에 비해 가장 높은 수치가 나왔으며 수치상 가장 뛰어난 성능을 보이는 것으로 판단됩니다.
정확도만 가지고 판단할 경우 SVM 모델 뿐만 아니라 Random Forest 역시 비슷한 정확도를 보여주었으나 정확도 만으로 판단하는 것은 불균형한 데이터로 인해 오류가 발생할 수 있기 때문에 Precision, Recall, F1-score 값을 모두 고려하였습니다.
이상의 내용에 따라 SVM 모델을 채택하였습니다.

와인 분류

데이터

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

wine = load_wine()  
wine_data = wine.data   
wine_label = wine.target  

X_train, X_test, y_train, y_test = train_test_split(wine_data, 
                                                    wine_label, 
                                                    test_size=0.2, 
                                                    random_state=7)

Model

# DecisionTreeClassifier
from sklearn.tree import DecisionTreeClassifier

decision_tree = DecisionTreeClassifier(random_state=32)
decision_tree.fit(X_train, y_train)
y_pred = decision_tree.predict(X_test)  # test 데이터로 예측

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         7
           1       0.89      1.00      0.94        17
           2       1.00      0.83      0.91        12

    accuracy                           0.94        36
   macro avg       0.96      0.94      0.95        36
weighted avg       0.95      0.94      0.94        36






array([[ 7,  0,  0],
       [ 0, 17,  0],
       [ 0,  2, 10]])

# Random Forest Classifier
from sklearn.ensemble import RandomForestClassifier

random_forest = RandomForestClassifier(random_state=32)
random_forest.fit(X_train, y_train)
y_pred = random_forest.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         7
           1       1.00      1.00      1.00        17
           2       1.00      1.00      1.00        12

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36






array([[ 7,  0,  0],
       [ 0, 17,  0],
       [ 0,  0, 12]])

# Support Vector Machine (SVM) 
from sklearn import svm
svm_model = svm.SVC()

svm_model.fit(X_train, y_train)
y_pred = svm_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       0.86      0.86      0.86         7
           1       0.58      0.88      0.70        17
           2       0.33      0.08      0.13        12

    accuracy                           0.61        36
   macro avg       0.59      0.61      0.56        36
weighted avg       0.55      0.61      0.54        36






array([[ 6,  0,  1],
       [ 1, 15,  1],
       [ 0, 11,  1]])

# SGD Classifier
from sklearn.linear_model import SGDClassifier
sgd_model = SGDClassifier()

sgd_model.fit(X_train, y_train)
y_pred = sgd_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       0.78      1.00      0.88         7
           1       0.59      0.94      0.73        17
           2       0.00      0.00      0.00        12

    accuracy                           0.64        36
   macro avg       0.46      0.65      0.53        36
weighted avg       0.43      0.64      0.51        36



/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))





array([[ 7,  0,  0],
       [ 1, 16,  0],
       [ 1, 11,  0]])

# Logistic Regression
from sklearn.linear_model import LogisticRegression
logistic_model = LogisticRegression()

logistic_model.fit(X_train, y_train)
y_pred = logistic_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      0.86      0.92         7
           1       0.94      1.00      0.97        17
           2       1.00      1.00      1.00        12

    accuracy                           0.97        36
   macro avg       0.98      0.95      0.96        36
weighted avg       0.97      0.97      0.97        36



/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:818: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG,





array([[ 6,  1,  0],
       [ 0, 17,  0],
       [ 0,  0, 12]])

모델 선정

분석 하고 나서도 찝찝한 기분이 드는 결과.
수치상만으로는 Random Forest가 맞는데 1이라는 값이 나올 수 있나라는 우려가 드는 부분이 있다.
1 이라는 결과를 오버피팅이라고 생각할 수 있는지 데이타를 세부적으로 살펴볼 필요가 있을 것 같다.
일단은 연습에 집중.

유방암 여부

데이터

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

breast_cancer = load_breast_cancer()  
breast_cancer_data = breast_cancer.data  
breast_cancer_label = breast_cancer.target

X_train, X_test, y_train, y_test = train_test_split(breast_cancer_data, 
                                                    breast_cancer_label, 
                                                    test_size=0.2, 
                                                    random_state=7)

Model

# DecisionTreeClassifier
from sklearn.tree import DecisionTreeClassifier

decision_tree = DecisionTreeClassifier(random_state=32)
decision_tree.fit(X_train, y_train)
y_pred = decision_tree.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       0.92      0.82      0.87        40
           1       0.91      0.96      0.93        74

    accuracy                           0.91       114
   macro avg       0.91      0.89      0.90       114
weighted avg       0.91      0.91      0.91       114






array([[33,  7],
       [ 3, 71]])

# Random Forest Classifier
from sklearn.ensemble import RandomForestClassifier

random_forest = RandomForestClassifier(random_state=32)
random_forest.fit(X_train, y_train)
y_pred = random_forest.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        40
           1       1.00      1.00      1.00        74

    accuracy                           1.00       114
   macro avg       1.00      1.00      1.00       114
weighted avg       1.00      1.00      1.00       114






array([[40,  0],
       [ 0, 74]])

# Support Vector Machine (SVM) 
from sklearn import svm
svm_model = svm.SVC()

svm_model.fit(X_train, y_train)
y_pred = svm_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      0.72      0.84        40
           1       0.87      1.00      0.93        74

    accuracy                           0.90       114
   macro avg       0.94      0.86      0.89       114
weighted avg       0.92      0.90      0.90       114






array([[29, 11],
       [ 0, 74]])

# SGD Classifier
from sklearn.linear_model import SGDClassifier
sgd_model = SGDClassifier()

sgd_model.fit(X_train, y_train)
y_pred = sgd_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       0.85      0.82      0.84        40
           1       0.91      0.92      0.91        74

    accuracy                           0.89       114
   macro avg       0.88      0.87      0.87       114
weighted avg       0.89      0.89      0.89       114






array([[33,  7],
       [ 6, 68]])

# Logistic Regression
from sklearn.linear_model import LogisticRegression
logistic_model = LogisticRegression()

logistic_model.fit(X_train, y_train)
y_pred = logistic_model.predict(X_test)

print(classification_report(y_test, y_pred))
confusion_matrix(y_test, y_pred)

              precision    recall  f1-score   support

           0       1.00      0.82      0.90        40
           1       0.91      1.00      0.95        74

    accuracy                           0.94       114
   macro avg       0.96      0.91      0.93       114
weighted avg       0.94      0.94      0.94       114



/Users/baekgun/opt/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:818: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG,





array([[33,  7],
       [ 0, 74]])

모델 선정

결과를 보고 살짝 한숨을 쉰^^;;;
도대체 1의 의미는 무엇일까.
- 찾아보았을 때에는 데이터를 확인해보아야 한다는데
- 사실 데이터의 어떤 부분을 보아야 할지 모르겠다는 것.
- 이미지를 분류하는 것에 있어서 잘 맞춘다는 것이니 좋은것?
여러가지 고민 결과 한가지 방안이 생각하는 것은 아래에서.

리뷰

데이터 평가 지표에 대한 막연한 감은 생겼지만 약간의 혼란스러움도 생김
1이라는 정확도에 있어서 가장 크게 고민이 되는 부분이지만
이것은 데이터셋을 나눠서 테스트해보면 해소될 수 있지 않을까 조심스럽게 추측해봄.
데이터 셋에서 Train을 줄이고 Test셋을 늘린 후 (적당한 비율로)
정확도를 확인했을 때 달라지는 부분이 있지 않을까 하는 것.
이 평가 지표에서 오버피팅이라면 어떻게 판단할 수 있는 사례에 대해 좀 더 알았으면 함.
- 수치가 어떻게 나왔을 떄 오버 피팅인지.
- 어떤 식의 그래프를 그리면 오버피팅을 의심해 봐야 하는지..
- 등에 대한 내용이 좀 더 필요하고 공부해야 할 듯.
사실 Random forest 수치가 잘 나왔다는 것은 극단적인 수치가 없다고 판단할 수 있는 것이고
이 부분에 대해 계속 오버피팅에 대한 의구심이 떠나가질 않음.
데이터 셋을 좀 더 나눠서 다시 테스트가 필요할 듯 함.

백건

마케팅을 위한 인공지능 설계와 스타트업 Log

이전 포스트

ᄂ 😄 [13 일차] : EXPLORATION 04. 인공지능 작사가

다음 포스트

ᄂ 😄 [13 일차] : Assignment - EX02. 손글씨 분류하기

[AIFFEL] 아이펠 인공지능 전문가 과정

[E-02] 과제

손글씨 분류하기

데이터

데이터 준비

train, Test 데이터 분리

Model

Decision Tree Model

Random Forest 사용

SVM Model

SGD Model 사용하기

Logistic Regression Model

모델 선정

정리

와인 분류

데이터

Model

모델 선정

유방암 여부

데이터

Model

모델 선정

리뷰

ᄂ 😄 [13 일차] : EXPLORATION 04. 인공지능 작사가

ᄂ 😄 [14 일차] : FUNDAMENTAL 15. 포켓몬

0개의 댓글