[python] PyTorch Project Structure 1

김보현·2024년 7월 22일

PyTorch

목록 보기

3/7

speedrun 4주차 목표: Make Custom Conv Model

Goal

OOP와 모듈의 개념을 통해 프로젝트를 구성
PyTorch 사용법과 함께 VSCode와 SSH로 연결
딥러닝 학습을 위한 환경 구축

Overview

Project Template

개발 초기 단계에서는 대화식 개발 과정이 유리하다.
왜냐하면 학습과정과 디버깅 등 지속적인 확인이 가능하기 때문이다.
Shift+Enter를 할 때 마다 얻어지는 정보
배포 및 공유 단계에서 어려움이 있다.
jupyter notebook에서도 DL,ML 기본적인 부분은 잘 되지만 개발과 배포과정으로 연결되는 쉬운 재현의 어려움, 실행순서 꼬임 등의 어려움이 존재한다.
DL 코드도 하나의 프로그램이다.
: 개발 용이성 확보와 유지보수의 향상이 필요하다.

OOP + 모듈
=> 하나의 project 단위로 사용자에게 제공해주어야 한다.

다양한 프로젝트 템플릿이 존재하기 때문에 필요에 따라 수정해서 사용하면 된다.
실행, 데이터, 모델, 설정, 로깅, 지표, 유틸리티 등 다양한 모듈들을 분리하여 프로젝트를 하나의 템플릿화한다.

PyTorch Template 추천 repository
가장 추천 추천1 추천2

Module

pytorch-template/
│
├── train.py - main script to start training
├── test.py - evaluation of trained model
│
├── config.json - holds configuration for training
├── parse_config.py - class to handle config file and cli options
│
├── new_project.py - initialize new project with template files
│
├── base/ - abstract base classes
│   ├── base_data_loader.py
│   ├── base_model.py
│   └── base_trainer.py
│
├── data_loader/ - anything about data loading goes here
│   └── data_loaders.py
│
├── data/ - default directory for storing input data
│
├── model/ - models, losses, and metrics
│   ├── model.py
│   ├── metric.py
│   └── loss.py
│
├── saved/
│   ├── models/ - trained models are saved here
│   └── log/ - default logdir for tensorboard and logging output
│
├── trainer/ - trainers
│   └── trainer.py
│
├── logger/ - module for tensorboard visualization and logging
│   ├── visualization.py
│   ├── logger.py
│   └── logger_config.json
│  
└── utils/ - small utility functions
    ├── util.py
    └── ...

PyTorch 프로젝트를 구성하는 템플릿 예시에서 디렉토리와 파일의 역할

실행

train.py: 이 스크립트는 모델 학습을 시작하는 메인 스크립트이다. 데이터 로딩, 모델 초기화, 학습 루프 등이 이곳에 정의된다.
test.py: 학습된 모델을 평가하는 스크립트이다. 모델을 로드하고 테스트 데이터셋에서 성능을 평가한다.

설정

config.json: 학습에 필요한 설정을 JSON 형식으로 저장하는 파일이다. 학습 파라미터, 경로, 하이퍼파라미터 등이 포함된다.
parse_config.py: config.json 파일과 CLI 옵션을 처리하는 클래스를 포함한다. 설정 파일을 파싱하고 커맨드 라인에서 받은 옵션을 처리한다.

base-abstract module

base/: 추상 베이스 클래스들을 포함하는 디렉토리이다. 각 파일은 다양한 컴포넌트의 베이스 클래스를 정의한다.
- base_data_loader.py: 데이터 로더의 베이스 클래스를 정의한다.
- base_model.py: 모델의 베이스 클래스를 정의한다.
- base_trainer.py: 트레이너의 베이스 클래스를 정의한다. 실행시켜주는 가장 기본적인 단계, trigger이다.

data

data_loader/: 데이터 로딩과 관련된 모든 것을 포함하는 디렉토리이다.
- data_loaders.py: 데이터 로더 클래스들이 정의되어 있다.
data/: 입력 데이터를 저장하는 기본 디렉토리이다.

model-architecture, loss, metric

model/: 모델, 손실 함수, 메트릭을 정의하는 디렉토리이다.
- model.py: 모델 아키텍처가 정의된 파일이다.
- metric.py: 성능 평가를 위한 메트릭 함수들이 정의된 파일이다.
- loss.py: 손실 함수들이 정의된 파일이다.

저장소 - 로그, 모델상태

saved/: 모델과 로그 파일을 저장하는 디렉토리이다.
- models/: 학습된 모델이 저장되는 디렉토리이다.
- log/: 텐서보드와 로그 출력을 위한 기본 로그 디렉토리이다.

학습수행

trainer/: 트레이너 클래스를 포함하는 디렉토리이다.
- trainer.py: 학습 루프와 관련된 로직이 포함된 트레이너 클래스가 정의된 파일이다.

로깅설정

logger/: 텐서보드 시각화 및 로깅을 위한 모듈이다.
- visualization.py: 텐서보드 시각화 기능을 포함한다.
- logger.py: 로깅 기능을 포함한다.
- logger_config.json: 로깅 설정을 포함한 JSON 파일이다.

유틸리티

utils/: 작은 유틸리티 함수들을 포함하는 디렉토리이다.
- util.py: 여러 가지 유틸리티 함수들이 정의된 파일이다.

SSH에 git clone

clone 해온 폴더 안에 MyProject를 만들어준다.

code .을 입력하면 새로운 VSCode 창이 뜨면서 코드 구조를 볼 수 있다.

config.json은 여러가지 설정에 들어가있는 정보이다.
python train.py -c config.json을 bash에 입력해주면 다음과 같이 train이 된다.

Train Epoch: 25 [22528/54000 (42%)] Loss: 0.103039
Train Epoch: 25 [23936/54000 (44%)] Loss: 0.033645
Train Epoch: 25 [25344/54000 (47%)] Loss: 0.050302
Train Epoch: 25 [26752/54000 (50%)] Loss: 0.082389
Train Epoch: 25 [28160/54000 (52%)] Loss: 0.121132
Train Epoch: 25 [29568/54000 (55%)] Loss: 0.076809
Train Epoch: 25 [30976/54000 (57%)] Loss: 0.094845
Train Epoch: 25 [32384/54000 (60%)] Loss: 0.064883
Train Epoch: 25 [33792/54000 (63%)] Loss: 0.048803
Train Epoch: 25 [35200/54000 (65%)] Loss: 0.061302
Train Epoch: 25 [36608/54000 (68%)] Loss: 0.131686
Train Epoch: 25 [38016/54000 (70%)] Loss: 0.141702
Train Epoch: 25 [39424/54000 (73%)] Loss: 0.054183
Train Epoch: 25 [40832/54000 (76%)] Loss: 0.119991
Train Epoch: 25 [42240/54000 (78%)] Loss: 0.102565
Train Epoch: 25 [43648/54000 (81%)] Loss: 0.064143
Train Epoch: 25 [45056/54000 (83%)] Loss: 0.049414
Train Epoch: 25 [46464/54000 (86%)] Loss: 0.100502
Train Epoch: 25 [47872/54000 (89%)] Loss: 0.153941
Train Epoch: 25 [49280/54000 (91%)] Loss: 0.097074
Train Epoch: 25 [50688/54000 (94%)] Loss: 0.103645
Train Epoch: 25 [52096/54000 (96%)] Loss: 0.119793
Train Epoch: 25 [53504/54000 (99%)] Loss: 0.093875
    epoch          : 25
    loss           : 0.10751288008580417
    accuracy       : 0.9685595802301963
    top_k_acc      : 0.9964825236966824
    val_loss       : 0.039838541930719736
    val_accuracy   : 0.9886730623100305
    val_top_k_acc  : 0.9991688829787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch25.pth ...
Train Epoch: 26 [0/54000 (0%)] Loss: 0.218775
Train Epoch: 26 [1408/54000 (3%)] Loss: 0.117433
Train Epoch: 26 [2816/54000 (5%)] Loss: 0.107088
Train Epoch: 26 [4224/54000 (8%)] Loss: 0.086084
Train Epoch: 26 [5632/54000 (10%)] Loss: 0.048478
Train Epoch: 26 [7040/54000 (13%)] Loss: 0.124609
Train Epoch: 26 [8448/54000 (16%)] Loss: 0.072788
Train Epoch: 26 [9856/54000 (18%)] Loss: 0.135024
Train Epoch: 26 [11264/54000 (21%)] Loss: 0.077583
Train Epoch: 26 [12672/54000 (23%)] Loss: 0.107356
Train Epoch: 26 [14080/54000 (26%)] Loss: 0.037458
Train Epoch: 26 [15488/54000 (29%)] Loss: 0.044388
Train Epoch: 26 [16896/54000 (31%)] Loss: 0.063734
Train Epoch: 26 [18304/54000 (34%)] Loss: 0.267027
Train Epoch: 26 [19712/54000 (37%)] Loss: 0.128427
Train Epoch: 26 [21120/54000 (39%)] Loss: 0.051455
Train Epoch: 26 [22528/54000 (42%)] Loss: 0.089729
Train Epoch: 26 [23936/54000 (44%)] Loss: 0.123832
Train Epoch: 26 [25344/54000 (47%)] Loss: 0.139797
Train Epoch: 26 [26752/54000 (50%)] Loss: 0.155027
Train Epoch: 26 [28160/54000 (52%)] Loss: 0.043367
Train Epoch: 26 [29568/54000 (55%)] Loss: 0.088288
Train Epoch: 26 [30976/54000 (57%)] Loss: 0.127634
Train Epoch: 26 [32384/54000 (60%)] Loss: 0.090415
Train Epoch: 26 [33792/54000 (63%)] Loss: 0.182064
Train Epoch: 26 [35200/54000 (65%)] Loss: 0.303926
Train Epoch: 26 [36608/54000 (68%)] Loss: 0.143217
Train Epoch: 26 [38016/54000 (70%)] Loss: 0.058583
Train Epoch: 26 [39424/54000 (73%)] Loss: 0.109315
Train Epoch: 26 [40832/54000 (76%)] Loss: 0.116083
Train Epoch: 26 [42240/54000 (78%)] Loss: 0.061569
Train Epoch: 26 [43648/54000 (81%)] Loss: 0.113060
Train Epoch: 26 [45056/54000 (83%)] Loss: 0.073252
Train Epoch: 26 [46464/54000 (86%)] Loss: 0.142844
Train Epoch: 26 [47872/54000 (89%)] Loss: 0.034814
Train Epoch: 26 [49280/54000 (91%)] Loss: 0.137386
Train Epoch: 26 [50688/54000 (94%)] Loss: 0.094655
Train Epoch: 26 [52096/54000 (96%)] Loss: 0.049345
Train Epoch: 26 [53504/54000 (99%)] Loss: 0.071435
    epoch          : 26
    loss           : 0.10394749072296501
    accuracy       : 0.9683559368652673
    top_k_acc      : 0.9966993906567365
    val_loss       : 0.0391211243868964
    val_accuracy   : 0.987295782674772
    val_top_k_acc  : 0.9993113601823709
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch26.pth ...
Train Epoch: 27 [0/54000 (0%)] Loss: 0.043722
Train Epoch: 27 [1408/54000 (3%)] Loss: 0.081133
Train Epoch: 27 [2816/54000 (5%)] Loss: 0.092307
Train Epoch: 27 [4224/54000 (8%)] Loss: 0.075349
Train Epoch: 27 [5632/54000 (10%)] Loss: 0.049910
Train Epoch: 27 [7040/54000 (13%)] Loss: 0.088892
Train Epoch: 27 [8448/54000 (16%)] Loss: 0.119868
Train Epoch: 27 [9856/54000 (18%)] Loss: 0.039580
Train Epoch: 27 [11264/54000 (21%)] Loss: 0.154280
Train Epoch: 27 [12672/54000 (23%)] Loss: 0.163594
Train Epoch: 27 [14080/54000 (26%)] Loss: 0.091927
Train Epoch: 27 [15488/54000 (29%)] Loss: 0.149581
Train Epoch: 27 [16896/54000 (31%)] Loss: 0.115137
Train Epoch: 27 [18304/54000 (34%)] Loss: 0.166628
Train Epoch: 27 [19712/54000 (37%)] Loss: 0.142237
Train Epoch: 27 [21120/54000 (39%)] Loss: 0.135696
Train Epoch: 27 [22528/54000 (42%)] Loss: 0.238346
Train Epoch: 27 [23936/54000 (44%)] Loss: 0.118776
Train Epoch: 27 [25344/54000 (47%)] Loss: 0.141135
Train Epoch: 27 [26752/54000 (50%)] Loss: 0.094542
Train Epoch: 27 [28160/54000 (52%)] Loss: 0.172510
Train Epoch: 27 [29568/54000 (55%)] Loss: 0.117066
Train Epoch: 27 [30976/54000 (57%)] Loss: 0.071727
Train Epoch: 27 [32384/54000 (60%)] Loss: 0.125582
Train Epoch: 27 [33792/54000 (63%)] Loss: 0.133626
Train Epoch: 27 [35200/54000 (65%)] Loss: 0.077264
Train Epoch: 27 [36608/54000 (68%)] Loss: 0.081816
Train Epoch: 27 [38016/54000 (70%)] Loss: 0.139331
Train Epoch: 27 [39424/54000 (73%)] Loss: 0.066412
Train Epoch: 27 [40832/54000 (76%)] Loss: 0.046289
Train Epoch: 27 [42240/54000 (78%)] Loss: 0.116054
Train Epoch: 27 [43648/54000 (81%)] Loss: 0.106013
Train Epoch: 27 [45056/54000 (83%)] Loss: 0.061179
Train Epoch: 27 [46464/54000 (86%)] Loss: 0.064731
Train Epoch: 27 [47872/54000 (89%)] Loss: 0.139106
Train Epoch: 27 [49280/54000 (91%)] Loss: 0.121178
Train Epoch: 27 [50688/54000 (94%)] Loss: 0.107683
Train Epoch: 27 [52096/54000 (96%)] Loss: 0.062962
Train Epoch: 27 [53504/54000 (99%)] Loss: 0.138280
    epoch          : 27
    loss           : 0.10824207426159162
    accuracy       : 0.9672530890318212
    top_k_acc      : 0.9967787322274881
    val_loss       : 0.03569327680111684
    val_accuracy   : 0.9895041793313071
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch27.pth ...
Saving current best: model_best.pth ...
Train Epoch: 28 [0/54000 (0%)] Loss: 0.071755
Train Epoch: 28 [1408/54000 (3%)] Loss: 0.106848
Train Epoch: 28 [2816/54000 (5%)] Loss: 0.143030
Train Epoch: 28 [4224/54000 (8%)] Loss: 0.076291
Train Epoch: 28 [5632/54000 (10%)] Loss: 0.200063
Train Epoch: 28 [7040/54000 (13%)] Loss: 0.047107
Train Epoch: 28 [8448/54000 (16%)] Loss: 0.116788
Train Epoch: 28 [9856/54000 (18%)] Loss: 0.113519
Train Epoch: 28 [11264/54000 (21%)] Loss: 0.210565
Train Epoch: 28 [12672/54000 (23%)] Loss: 0.060594
Train Epoch: 28 [14080/54000 (26%)] Loss: 0.094577
Train Epoch: 28 [15488/54000 (29%)] Loss: 0.159034
Train Epoch: 28 [16896/54000 (31%)] Loss: 0.080587
Train Epoch: 28 [18304/54000 (34%)] Loss: 0.067869
Train Epoch: 28 [19712/54000 (37%)] Loss: 0.074759
Train Epoch: 28 [21120/54000 (39%)] Loss: 0.141513
Train Epoch: 28 [22528/54000 (42%)] Loss: 0.034767
Train Epoch: 28 [23936/54000 (44%)] Loss: 0.046473
Train Epoch: 28 [25344/54000 (47%)] Loss: 0.132118
Train Epoch: 28 [26752/54000 (50%)] Loss: 0.120770
Train Epoch: 28 [28160/54000 (52%)] Loss: 0.073901
Train Epoch: 28 [29568/54000 (55%)] Loss: 0.099550
Train Epoch: 28 [30976/54000 (57%)] Loss: 0.095251
Train Epoch: 28 [32384/54000 (60%)] Loss: 0.020941
Train Epoch: 28 [33792/54000 (63%)] Loss: 0.175844
Train Epoch: 28 [35200/54000 (65%)] Loss: 0.172745
Train Epoch: 28 [36608/54000 (68%)] Loss: 0.078500
Train Epoch: 28 [38016/54000 (70%)] Loss: 0.223717
Train Epoch: 28 [39424/54000 (73%)] Loss: 0.109596
Train Epoch: 28 [40832/54000 (76%)] Loss: 0.046549
Train Epoch: 28 [42240/54000 (78%)] Loss: 0.185425
Train Epoch: 28 [43648/54000 (81%)] Loss: 0.099566
Train Epoch: 28 [45056/54000 (83%)] Loss: 0.068021
Train Epoch: 28 [46464/54000 (86%)] Loss: 0.108677
Train Epoch: 28 [47872/54000 (89%)] Loss: 0.070779
Train Epoch: 28 [49280/54000 (91%)] Loss: 0.278224
Train Epoch: 28 [50688/54000 (94%)] Loss: 0.125651
Train Epoch: 28 [52096/54000 (96%)] Loss: 0.078666
Train Epoch: 28 [53504/54000 (99%)] Loss: 0.045107
    epoch          : 28
    loss           : 0.10715693555401541
    accuracy       : 0.9690541426878808
    top_k_acc      : 0.9959641587677726
    val_loss       : 0.0385979158888036
    val_accuracy   : 0.9886255699088146
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch28.pth ...
Train Epoch: 29 [0/54000 (0%)] Loss: 0.050837
Train Epoch: 29 [1408/54000 (3%)] Loss: 0.052386
Train Epoch: 29 [2816/54000 (5%)] Loss: 0.061743
Train Epoch: 29 [4224/54000 (8%)] Loss: 0.109275
Train Epoch: 29 [5632/54000 (10%)] Loss: 0.043688
Train Epoch: 29 [7040/54000 (13%)] Loss: 0.298331
Train Epoch: 29 [8448/54000 (16%)] Loss: 0.191068
Train Epoch: 29 [9856/54000 (18%)] Loss: 0.254090
Train Epoch: 29 [11264/54000 (21%)] Loss: 0.086195
Train Epoch: 29 [12672/54000 (23%)] Loss: 0.065042
Train Epoch: 29 [14080/54000 (26%)] Loss: 0.122006
Train Epoch: 29 [15488/54000 (29%)] Loss: 0.108460
Train Epoch: 29 [16896/54000 (31%)] Loss: 0.104013
Train Epoch: 29 [18304/54000 (34%)] Loss: 0.193346
Train Epoch: 29 [19712/54000 (37%)] Loss: 0.054477
Train Epoch: 29 [21120/54000 (39%)] Loss: 0.071246
Train Epoch: 29 [22528/54000 (42%)] Loss: 0.114438
Train Epoch: 29 [23936/54000 (44%)] Loss: 0.136951
Train Epoch: 29 [25344/54000 (47%)] Loss: 0.095295
Train Epoch: 29 [26752/54000 (50%)] Loss: 0.122300
Train Epoch: 29 [28160/54000 (52%)] Loss: 0.056657
Train Epoch: 29 [29568/54000 (55%)] Loss: 0.092282
Train Epoch: 29 [30976/54000 (57%)] Loss: 0.068574
Train Epoch: 29 [32384/54000 (60%)] Loss: 0.074593
Train Epoch: 29 [33792/54000 (63%)] Loss: 0.052283
Train Epoch: 29 [35200/54000 (65%)] Loss: 0.093593
Train Epoch: 29 [36608/54000 (68%)] Loss: 0.054643
Train Epoch: 29 [38016/54000 (70%)] Loss: 0.085734
Train Epoch: 29 [39424/54000 (73%)] Loss: 0.114281
Train Epoch: 29 [40832/54000 (76%)] Loss: 0.121854
Train Epoch: 29 [42240/54000 (78%)] Loss: 0.151349
Train Epoch: 29 [43648/54000 (81%)] Loss: 0.067871
Train Epoch: 29 [45056/54000 (83%)] Loss: 0.057590
Train Epoch: 29 [46464/54000 (86%)] Loss: 0.098205
Train Epoch: 29 [47872/54000 (89%)] Loss: 0.100722
Train Epoch: 29 [49280/54000 (91%)] Loss: 0.053179
Train Epoch: 29 [50688/54000 (94%)] Loss: 0.105573
Train Epoch: 29 [52096/54000 (96%)] Loss: 0.073071
Train Epoch: 29 [53504/54000 (99%)] Loss: 0.107227
    epoch          : 29
    loss           : 0.10552937828304502
    accuracy       : 0.9684273442789438
    top_k_acc      : 0.9965195497630331
    val_loss       : 0.03602355396612845
    val_accuracy   : 0.9891954787234043
    val_top_k_acc  : 0.9996675531914894
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch29.pth ...
Train Epoch: 30 [0/54000 (0%)] Loss: 0.054464
Train Epoch: 30 [1408/54000 (3%)] Loss: 0.099514
Train Epoch: 30 [2816/54000 (5%)] Loss: 0.118238
Train Epoch: 30 [4224/54000 (8%)] Loss: 0.092277
Train Epoch: 30 [5632/54000 (10%)] Loss: 0.177417
Train Epoch: 30 [7040/54000 (13%)] Loss: 0.087548
Train Epoch: 30 [8448/54000 (16%)] Loss: 0.063869
Train Epoch: 30 [9856/54000 (18%)] Loss: 0.049474
Train Epoch: 30 [11264/54000 (21%)] Loss: 0.150585
Train Epoch: 30 [12672/54000 (23%)] Loss: 0.036277
Train Epoch: 30 [14080/54000 (26%)] Loss: 0.124688
Train Epoch: 30 [15488/54000 (29%)] Loss: 0.083651
Train Epoch: 30 [16896/54000 (31%)] Loss: 0.246006
Train Epoch: 30 [18304/54000 (34%)] Loss: 0.135900
Train Epoch: 30 [19712/54000 (37%)] Loss: 0.068475
Train Epoch: 30 [21120/54000 (39%)] Loss: 0.129598
Train Epoch: 30 [22528/54000 (42%)] Loss: 0.057585
Train Epoch: 30 [23936/54000 (44%)] Loss: 0.081213
Train Epoch: 30 [25344/54000 (47%)] Loss: 0.044985
Train Epoch: 30 [26752/54000 (50%)] Loss: 0.256656
Train Epoch: 30 [28160/54000 (52%)] Loss: 0.172870
Train Epoch: 30 [29568/54000 (55%)] Loss: 0.104113
Train Epoch: 30 [30976/54000 (57%)] Loss: 0.052581
Train Epoch: 30 [32384/54000 (60%)] Loss: 0.042269
Train Epoch: 30 [33792/54000 (63%)] Loss: 0.061408
Train Epoch: 30 [35200/54000 (65%)] Loss: 0.054798
Train Epoch: 30 [36608/54000 (68%)] Loss: 0.072385
Train Epoch: 30 [38016/54000 (70%)] Loss: 0.100783
Train Epoch: 30 [39424/54000 (73%)] Loss: 0.044973
Train Epoch: 30 [40832/54000 (76%)] Loss: 0.054644
Train Epoch: 30 [42240/54000 (78%)] Loss: 0.065359
Train Epoch: 30 [43648/54000 (81%)] Loss: 0.129444
Train Epoch: 30 [45056/54000 (83%)] Loss: 0.104685
Train Epoch: 30 [46464/54000 (86%)] Loss: 0.062659
Train Epoch: 30 [47872/54000 (89%)] Loss: 0.027979
Train Epoch: 30 [49280/54000 (91%)] Loss: 0.103071
Train Epoch: 30 [50688/54000 (94%)] Loss: 0.203892
Train Epoch: 30 [52096/54000 (96%)] Loss: 0.239221
Train Epoch: 30 [53504/54000 (99%)] Loss: 0.086577
    epoch          : 30
    loss           : 0.10546193615272995
    accuracy       : 0.9685331330399459
    top_k_acc      : 0.9965195497630331
    val_loss       : 0.03820485080731042
    val_accuracy   : 0.9881031534954408
    val_top_k_acc  : 0.9996675531914894
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch30.pth ...
Train Epoch: 31 [0/54000 (0%)] Loss: 0.052466
Train Epoch: 31 [1408/54000 (3%)] Loss: 0.056185
Train Epoch: 31 [2816/54000 (5%)] Loss: 0.106912
Train Epoch: 31 [4224/54000 (8%)] Loss: 0.159254
Train Epoch: 31 [5632/54000 (10%)] Loss: 0.067851
Train Epoch: 31 [7040/54000 (13%)] Loss: 0.079711
Train Epoch: 31 [8448/54000 (16%)] Loss: 0.118263
Train Epoch: 31 [9856/54000 (18%)] Loss: 0.051184
Train Epoch: 31 [11264/54000 (21%)] Loss: 0.051165
Train Epoch: 31 [12672/54000 (23%)] Loss: 0.098457
Train Epoch: 31 [14080/54000 (26%)] Loss: 0.041409
Train Epoch: 31 [15488/54000 (29%)] Loss: 0.121385
Train Epoch: 31 [16896/54000 (31%)] Loss: 0.100809
Train Epoch: 31 [18304/54000 (34%)] Loss: 0.049123
Train Epoch: 31 [19712/54000 (37%)] Loss: 0.117201
Train Epoch: 31 [21120/54000 (39%)] Loss: 0.191368
Train Epoch: 31 [22528/54000 (42%)] Loss: 0.072701
Train Epoch: 31 [23936/54000 (44%)] Loss: 0.036195
Train Epoch: 31 [25344/54000 (47%)] Loss: 0.140016
Train Epoch: 31 [26752/54000 (50%)] Loss: 0.158328
Train Epoch: 31 [28160/54000 (52%)] Loss: 0.074367
Train Epoch: 31 [29568/54000 (55%)] Loss: 0.172158
Train Epoch: 31 [30976/54000 (57%)] Loss: 0.151970
Train Epoch: 31 [32384/54000 (60%)] Loss: 0.075670
Train Epoch: 31 [33792/54000 (63%)] Loss: 0.129229
Train Epoch: 31 [35200/54000 (65%)] Loss: 0.084741
Train Epoch: 31 [36608/54000 (68%)] Loss: 0.164730
Train Epoch: 31 [38016/54000 (70%)] Loss: 0.064563
Train Epoch: 31 [39424/54000 (73%)] Loss: 0.080328
Train Epoch: 31 [40832/54000 (76%)] Loss: 0.041165
Train Epoch: 31 [42240/54000 (78%)] Loss: 0.097039
Train Epoch: 31 [43648/54000 (81%)] Loss: 0.170170
Train Epoch: 31 [45056/54000 (83%)] Loss: 0.146923
Train Epoch: 31 [46464/54000 (86%)] Loss: 0.066074
Train Epoch: 31 [47872/54000 (89%)] Loss: 0.115774
Train Epoch: 31 [49280/54000 (91%)] Loss: 0.196204
Train Epoch: 31 [50688/54000 (94%)] Loss: 0.085860
Train Epoch: 31 [52096/54000 (96%)] Loss: 0.093646
Train Epoch: 31 [53504/54000 (99%)] Loss: 0.071873
    epoch          : 31
    loss           : 0.10108021592668377
    accuracy       : 0.969162576167908
    top_k_acc      : 0.9970167569397428
    val_loss       : 0.03652940404680657
    val_accuracy   : 0.988981762917933
    val_top_k_acc  : 0.9993113601823709
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch31.pth ...
Train Epoch: 32 [0/54000 (0%)] Loss: 0.073793
Train Epoch: 32 [1408/54000 (3%)] Loss: 0.075022
Train Epoch: 32 [2816/54000 (5%)] Loss: 0.081306
Train Epoch: 32 [4224/54000 (8%)] Loss: 0.090030
Train Epoch: 32 [5632/54000 (10%)] Loss: 0.097269
Train Epoch: 32 [7040/54000 (13%)] Loss: 0.121223
Train Epoch: 32 [8448/54000 (16%)] Loss: 0.062559
Train Epoch: 32 [9856/54000 (18%)] Loss: 0.055144
Train Epoch: 32 [11264/54000 (21%)] Loss: 0.111191
Train Epoch: 32 [12672/54000 (23%)] Loss: 0.070774
Train Epoch: 32 [14080/54000 (26%)] Loss: 0.066266
Train Epoch: 32 [15488/54000 (29%)] Loss: 0.076401
Train Epoch: 32 [16896/54000 (31%)] Loss: 0.088867
Train Epoch: 32 [18304/54000 (34%)] Loss: 0.063385
Train Epoch: 32 [19712/54000 (37%)] Loss: 0.036653
Train Epoch: 32 [21120/54000 (39%)] Loss: 0.189151
Train Epoch: 32 [22528/54000 (42%)] Loss: 0.047996
Train Epoch: 32 [23936/54000 (44%)] Loss: 0.161894
Train Epoch: 32 [25344/54000 (47%)] Loss: 0.083547
Train Epoch: 32 [26752/54000 (50%)] Loss: 0.105474
Train Epoch: 32 [28160/54000 (52%)] Loss: 0.067311
Train Epoch: 32 [29568/54000 (55%)] Loss: 0.163670
Train Epoch: 32 [30976/54000 (57%)] Loss: 0.206910
Train Epoch: 32 [32384/54000 (60%)] Loss: 0.129203
Train Epoch: 32 [33792/54000 (63%)] Loss: 0.101921
Train Epoch: 32 [35200/54000 (65%)] Loss: 0.041896
Train Epoch: 32 [36608/54000 (68%)] Loss: 0.057178
Train Epoch: 32 [38016/54000 (70%)] Loss: 0.098332
Train Epoch: 32 [39424/54000 (73%)] Loss: 0.084896
Train Epoch: 32 [40832/54000 (76%)] Loss: 0.088086
Train Epoch: 32 [42240/54000 (78%)] Loss: 0.063353
Train Epoch: 32 [43648/54000 (81%)] Loss: 0.092034
Train Epoch: 32 [45056/54000 (83%)] Loss: 0.073708
Train Epoch: 32 [46464/54000 (86%)] Loss: 0.078747
Train Epoch: 32 [47872/54000 (89%)] Loss: 0.148245
Train Epoch: 32 [49280/54000 (91%)] Loss: 0.194452
Train Epoch: 32 [50688/54000 (94%)] Loss: 0.143861
Train Epoch: 32 [52096/54000 (96%)] Loss: 0.093417
Train Epoch: 32 [53504/54000 (99%)] Loss: 0.144855
    epoch          : 32
    loss           : 0.09921244060470595
    accuracy       : 0.9695777970548408
    top_k_acc      : 0.9970379146919431
    val_loss       : 0.040284532345236934
    val_accuracy   : 0.9879844224924011
    val_top_k_acc  : 0.9993351063829787
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch32.pth ...
Train Epoch: 33 [0/54000 (0%)] Loss: 0.117755
Train Epoch: 33 [1408/54000 (3%)] Loss: 0.110634
Train Epoch: 33 [2816/54000 (5%)] Loss: 0.095291
Train Epoch: 33 [4224/54000 (8%)] Loss: 0.045478
Train Epoch: 33 [5632/54000 (10%)] Loss: 0.037266
Train Epoch: 33 [7040/54000 (13%)] Loss: 0.070216
Train Epoch: 33 [8448/54000 (16%)] Loss: 0.128316
Train Epoch: 33 [9856/54000 (18%)] Loss: 0.307018
Train Epoch: 33 [11264/54000 (21%)] Loss: 0.122050
Train Epoch: 33 [12672/54000 (23%)] Loss: 0.049622
Train Epoch: 33 [14080/54000 (26%)] Loss: 0.073194
Train Epoch: 33 [15488/54000 (29%)] Loss: 0.133577
Train Epoch: 33 [16896/54000 (31%)] Loss: 0.130570
Train Epoch: 33 [18304/54000 (34%)] Loss: 0.098172
Train Epoch: 33 [19712/54000 (37%)] Loss: 0.127420
Train Epoch: 33 [21120/54000 (39%)] Loss: 0.188783
Train Epoch: 33 [22528/54000 (42%)] Loss: 0.019224
Train Epoch: 33 [23936/54000 (44%)] Loss: 0.098389
Train Epoch: 33 [25344/54000 (47%)] Loss: 0.091565
Train Epoch: 33 [26752/54000 (50%)] Loss: 0.096791
Train Epoch: 33 [28160/54000 (52%)] Loss: 0.079361
Train Epoch: 33 [29568/54000 (55%)] Loss: 0.157408
Train Epoch: 33 [30976/54000 (57%)] Loss: 0.125078
Train Epoch: 33 [32384/54000 (60%)] Loss: 0.088428
Train Epoch: 33 [33792/54000 (63%)] Loss: 0.090949
Train Epoch: 33 [35200/54000 (65%)] Loss: 0.032975
Train Epoch: 33 [36608/54000 (68%)] Loss: 0.079423
Train Epoch: 33 [38016/54000 (70%)] Loss: 0.124474
Train Epoch: 33 [39424/54000 (73%)] Loss: 0.146168
Train Epoch: 33 [40832/54000 (76%)] Loss: 0.079886
Train Epoch: 33 [42240/54000 (78%)] Loss: 0.122007
Train Epoch: 33 [43648/54000 (81%)] Loss: 0.033792
Train Epoch: 33 [45056/54000 (83%)] Loss: 0.193582
Train Epoch: 33 [46464/54000 (86%)] Loss: 0.087946
Train Epoch: 33 [47872/54000 (89%)] Loss: 0.142977
Train Epoch: 33 [49280/54000 (91%)] Loss: 0.130882
Train Epoch: 33 [50688/54000 (94%)] Loss: 0.131727
Train Epoch: 33 [52096/54000 (96%)] Loss: 0.092733
Train Epoch: 33 [53504/54000 (99%)] Loss: 0.143622
    epoch          : 33
    loss           : 0.10098831835357358
    accuracy       : 0.9699268999661476
    top_k_acc      : 0.9967205484089371
    val_loss       : 0.039045847427258466
    val_accuracy   : 0.9881506458966565
    val_top_k_acc  : 0.9993113601823709
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch33.pth ...
Train Epoch: 34 [0/54000 (0%)] Loss: 0.056494
Train Epoch: 34 [1408/54000 (3%)] Loss: 0.083461
Train Epoch: 34 [2816/54000 (5%)] Loss: 0.047392
Train Epoch: 34 [4224/54000 (8%)] Loss: 0.079874
Train Epoch: 34 [5632/54000 (10%)] Loss: 0.106087
Train Epoch: 34 [7040/54000 (13%)] Loss: 0.152799
Train Epoch: 34 [8448/54000 (16%)] Loss: 0.102737
Train Epoch: 34 [9856/54000 (18%)] Loss: 0.105465
Train Epoch: 34 [11264/54000 (21%)] Loss: 0.097985
Train Epoch: 34 [12672/54000 (23%)] Loss: 0.049451
Train Epoch: 34 [14080/54000 (26%)] Loss: 0.115870
Train Epoch: 34 [15488/54000 (29%)] Loss: 0.088090
Train Epoch: 34 [16896/54000 (31%)] Loss: 0.077561
Train Epoch: 34 [18304/54000 (34%)] Loss: 0.073033
Train Epoch: 34 [19712/54000 (37%)] Loss: 0.064860
Train Epoch: 34 [21120/54000 (39%)] Loss: 0.110995
Train Epoch: 34 [22528/54000 (42%)] Loss: 0.149479
Train Epoch: 34 [23936/54000 (44%)] Loss: 0.117649
Train Epoch: 34 [25344/54000 (47%)] Loss: 0.172732
Train Epoch: 34 [26752/54000 (50%)] Loss: 0.030049
Train Epoch: 34 [28160/54000 (52%)] Loss: 0.313914
Train Epoch: 34 [29568/54000 (55%)] Loss: 0.070358
Train Epoch: 34 [30976/54000 (57%)] Loss: 0.123293
Train Epoch: 34 [32384/54000 (60%)] Loss: 0.104521
Train Epoch: 34 [33792/54000 (63%)] Loss: 0.109352
Train Epoch: 34 [35200/54000 (65%)] Loss: 0.147304
Train Epoch: 34 [36608/54000 (68%)] Loss: 0.034360
Train Epoch: 34 [38016/54000 (70%)] Loss: 0.097994
Train Epoch: 34 [39424/54000 (73%)] Loss: 0.080687
Train Epoch: 34 [40832/54000 (76%)] Loss: 0.036257
Train Epoch: 34 [42240/54000 (78%)] Loss: 0.093420
Train Epoch: 34 [43648/54000 (81%)] Loss: 0.093650
Train Epoch: 34 [45056/54000 (83%)] Loss: 0.233469
Train Epoch: 34 [46464/54000 (86%)] Loss: 0.135906
Train Epoch: 34 [47872/54000 (89%)] Loss: 0.057028
Train Epoch: 34 [49280/54000 (91%)] Loss: 0.151237
Train Epoch: 34 [50688/54000 (94%)] Loss: 0.139648
Train Epoch: 34 [52096/54000 (96%)] Loss: 0.081431
Train Epoch: 34 [53504/54000 (99%)] Loss: 0.095010
    epoch          : 34
    loss           : 0.096699401038871
    accuracy       : 0.9695910206499662
    top_k_acc      : 0.9970564277251185
    val_loss       : 0.03542699578634285
    val_accuracy   : 0.989480433130699
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch34.pth ...
Saving current best: model_best.pth ...
Train Epoch: 35 [0/54000 (0%)] Loss: 0.046560
Train Epoch: 35 [1408/54000 (3%)] Loss: 0.078715
Train Epoch: 35 [2816/54000 (5%)] Loss: 0.178785
Train Epoch: 35 [4224/54000 (8%)] Loss: 0.040100
Train Epoch: 35 [5632/54000 (10%)] Loss: 0.055796
Train Epoch: 35 [7040/54000 (13%)] Loss: 0.127846
Train Epoch: 35 [8448/54000 (16%)] Loss: 0.088751
Train Epoch: 35 [9856/54000 (18%)] Loss: 0.077539
Train Epoch: 35 [11264/54000 (21%)] Loss: 0.097743
Train Epoch: 35 [12672/54000 (23%)] Loss: 0.137792
Train Epoch: 35 [14080/54000 (26%)] Loss: 0.050837
Train Epoch: 35 [15488/54000 (29%)] Loss: 0.089196
Train Epoch: 35 [16896/54000 (31%)] Loss: 0.110488
Train Epoch: 35 [18304/54000 (34%)] Loss: 0.064262
Train Epoch: 35 [19712/54000 (37%)] Loss: 0.058976
Train Epoch: 35 [21120/54000 (39%)] Loss: 0.102894
Train Epoch: 35 [22528/54000 (42%)] Loss: 0.173581
Train Epoch: 35 [23936/54000 (44%)] Loss: 0.062260
Train Epoch: 35 [25344/54000 (47%)] Loss: 0.283183
Train Epoch: 35 [26752/54000 (50%)] Loss: 0.017184
Train Epoch: 35 [28160/54000 (52%)] Loss: 0.160173
Train Epoch: 35 [29568/54000 (55%)] Loss: 0.116234
Train Epoch: 35 [30976/54000 (57%)] Loss: 0.247975
Train Epoch: 35 [32384/54000 (60%)] Loss: 0.144735
Train Epoch: 35 [33792/54000 (63%)] Loss: 0.073738
Train Epoch: 35 [35200/54000 (65%)] Loss: 0.174284
Train Epoch: 35 [36608/54000 (68%)] Loss: 0.046828
Train Epoch: 35 [38016/54000 (70%)] Loss: 0.156113
Train Epoch: 35 [39424/54000 (73%)] Loss: 0.122556
Train Epoch: 35 [40832/54000 (76%)] Loss: 0.220203
Train Epoch: 35 [42240/54000 (78%)] Loss: 0.042903
Train Epoch: 35 [43648/54000 (81%)] Loss: 0.163590
Train Epoch: 35 [45056/54000 (83%)] Loss: 0.105684
Train Epoch: 35 [46464/54000 (86%)] Loss: 0.092056
Train Epoch: 35 [47872/54000 (89%)] Loss: 0.065153
Train Epoch: 35 [49280/54000 (91%)] Loss: 0.132703
Train Epoch: 35 [50688/54000 (94%)] Loss: 0.065211
Train Epoch: 35 [52096/54000 (96%)] Loss: 0.048604
Train Epoch: 35 [53504/54000 (99%)] Loss: 0.084761
    epoch          : 35
    loss           : 0.09547909004260649
    accuracy       : 0.9699850837846986
    top_k_acc      : 0.9971304798578199
    val_loss       : 0.03338785989437886
    val_accuracy   : 0.9893617021276596
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch35.pth ...
Saving current best: model_best.pth ...
Train Epoch: 36 [0/54000 (0%)] Loss: 0.110239
Train Epoch: 36 [1408/54000 (3%)] Loss: 0.075448
Train Epoch: 36 [2816/54000 (5%)] Loss: 0.081101
Train Epoch: 36 [4224/54000 (8%)] Loss: 0.082518
Train Epoch: 36 [5632/54000 (10%)] Loss: 0.031230
Train Epoch: 36 [7040/54000 (13%)] Loss: 0.093411
Train Epoch: 36 [8448/54000 (16%)] Loss: 0.188358
Train Epoch: 36 [9856/54000 (18%)] Loss: 0.122625
Train Epoch: 36 [11264/54000 (21%)] Loss: 0.051547
Train Epoch: 36 [12672/54000 (23%)] Loss: 0.048227
Train Epoch: 36 [14080/54000 (26%)] Loss: 0.056212
Train Epoch: 36 [15488/54000 (29%)] Loss: 0.137457
Train Epoch: 36 [16896/54000 (31%)] Loss: 0.060233
Train Epoch: 36 [18304/54000 (34%)] Loss: 0.215974
Train Epoch: 36 [19712/54000 (37%)] Loss: 0.062715
Train Epoch: 36 [21120/54000 (39%)] Loss: 0.077740
Train Epoch: 36 [22528/54000 (42%)] Loss: 0.094538
Train Epoch: 36 [23936/54000 (44%)] Loss: 0.109965
Train Epoch: 36 [25344/54000 (47%)] Loss: 0.099188
Train Epoch: 36 [26752/54000 (50%)] Loss: 0.130187
Train Epoch: 36 [28160/54000 (52%)] Loss: 0.071677
Train Epoch: 36 [29568/54000 (55%)] Loss: 0.110488
Train Epoch: 36 [30976/54000 (57%)] Loss: 0.073976
Train Epoch: 36 [32384/54000 (60%)] Loss: 0.070490
Train Epoch: 36 [33792/54000 (63%)] Loss: 0.025605
Train Epoch: 36 [35200/54000 (65%)] Loss: 0.045893
Train Epoch: 36 [36608/54000 (68%)] Loss: 0.210975
Train Epoch: 36 [38016/54000 (70%)] Loss: 0.079554
Train Epoch: 36 [39424/54000 (73%)] Loss: 0.178145
Train Epoch: 36 [40832/54000 (76%)] Loss: 0.048682
Train Epoch: 36 [42240/54000 (78%)] Loss: 0.038427
Train Epoch: 36 [43648/54000 (81%)] Loss: 0.075224
Train Epoch: 36 [45056/54000 (83%)] Loss: 0.083520
Train Epoch: 36 [46464/54000 (86%)] Loss: 0.114444
Train Epoch: 36 [47872/54000 (89%)] Loss: 0.154557
Train Epoch: 36 [49280/54000 (91%)] Loss: 0.083652
Train Epoch: 36 [50688/54000 (94%)] Loss: 0.070300
Train Epoch: 36 [52096/54000 (96%)] Loss: 0.052837
Train Epoch: 36 [53504/54000 (99%)] Loss: 0.095903
    epoch          : 36
    loss           : 0.09915763002895349
    accuracy       : 0.97005384647935
    top_k_acc      : 0.996294748645904
    val_loss       : 0.03534735787035699
    val_accuracy   : 0.9896941489361702
    val_top_k_acc  : 0.9996675531914894
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch36.pth ...
Train Epoch: 37 [0/54000 (0%)] Loss: 0.113401
Train Epoch: 37 [1408/54000 (3%)] Loss: 0.073089
Train Epoch: 37 [2816/54000 (5%)] Loss: 0.057137
Train Epoch: 37 [4224/54000 (8%)] Loss: 0.055644
Train Epoch: 37 [5632/54000 (10%)] Loss: 0.094710
Train Epoch: 37 [7040/54000 (13%)] Loss: 0.042785
Train Epoch: 37 [8448/54000 (16%)] Loss: 0.101379
Train Epoch: 37 [9856/54000 (18%)] Loss: 0.083550
Train Epoch: 37 [11264/54000 (21%)] Loss: 0.048020
Train Epoch: 37 [12672/54000 (23%)] Loss: 0.078164
Train Epoch: 37 [14080/54000 (26%)] Loss: 0.010478
Train Epoch: 37 [15488/54000 (29%)] Loss: 0.133810
Train Epoch: 37 [16896/54000 (31%)] Loss: 0.142844
Train Epoch: 37 [18304/54000 (34%)] Loss: 0.120372
Train Epoch: 37 [19712/54000 (37%)] Loss: 0.061972
Train Epoch: 37 [21120/54000 (39%)] Loss: 0.219281
Train Epoch: 37 [22528/54000 (42%)] Loss: 0.156751
Train Epoch: 37 [23936/54000 (44%)] Loss: 0.149521
Train Epoch: 37 [25344/54000 (47%)] Loss: 0.083055
Train Epoch: 37 [26752/54000 (50%)] Loss: 0.119881
Train Epoch: 37 [28160/54000 (52%)] Loss: 0.065661
Train Epoch: 37 [29568/54000 (55%)] Loss: 0.052247
Train Epoch: 37 [30976/54000 (57%)] Loss: 0.251952
Train Epoch: 37 [32384/54000 (60%)] Loss: 0.053727
Train Epoch: 37 [33792/54000 (63%)] Loss: 0.054923
Train Epoch: 37 [35200/54000 (65%)] Loss: 0.052056
Train Epoch: 37 [36608/54000 (68%)] Loss: 0.196499
Train Epoch: 37 [38016/54000 (70%)] Loss: 0.097050
Train Epoch: 37 [39424/54000 (73%)] Loss: 0.047460
Train Epoch: 37 [40832/54000 (76%)] Loss: 0.044556
Train Epoch: 37 [42240/54000 (78%)] Loss: 0.043542
Train Epoch: 37 [43648/54000 (81%)] Loss: 0.069742
Train Epoch: 37 [45056/54000 (83%)] Loss: 0.090292
Train Epoch: 37 [46464/54000 (86%)] Loss: 0.084782
Train Epoch: 37 [47872/54000 (89%)] Loss: 0.071240
Train Epoch: 37 [49280/54000 (91%)] Loss: 0.141274
Train Epoch: 37 [50688/54000 (94%)] Loss: 0.148278
Train Epoch: 37 [52096/54000 (96%)] Loss: 0.087012
Train Epoch: 37 [53504/54000 (99%)] Loss: 0.084277
    epoch          : 37
    loss           : 0.094282587656919
    accuracy       : 0.9719130839539608
    top_k_acc      : 0.9971860189573459
    val_loss       : 0.03436503300141107
    val_accuracy   : 0.9886968085106383
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch37.pth ...
Train Epoch: 38 [0/54000 (0%)] Loss: 0.095586
Train Epoch: 38 [1408/54000 (3%)] Loss: 0.096110
Train Epoch: 38 [2816/54000 (5%)] Loss: 0.113169
Train Epoch: 38 [4224/54000 (8%)] Loss: 0.060112
Train Epoch: 38 [5632/54000 (10%)] Loss: 0.073031
Train Epoch: 38 [7040/54000 (13%)] Loss: 0.134203
Train Epoch: 38 [8448/54000 (16%)] Loss: 0.044021
Train Epoch: 38 [9856/54000 (18%)] Loss: 0.076188
Train Epoch: 38 [11264/54000 (21%)] Loss: 0.075099
Train Epoch: 38 [12672/54000 (23%)] Loss: 0.078441
Train Epoch: 38 [14080/54000 (26%)] Loss: 0.047802
Train Epoch: 38 [15488/54000 (29%)] Loss: 0.193711
Train Epoch: 38 [16896/54000 (31%)] Loss: 0.098749
Train Epoch: 38 [18304/54000 (34%)] Loss: 0.071644
Train Epoch: 38 [19712/54000 (37%)] Loss: 0.097200
Train Epoch: 38 [21120/54000 (39%)] Loss: 0.093836
Train Epoch: 38 [22528/54000 (42%)] Loss: 0.059156
Train Epoch: 38 [23936/54000 (44%)] Loss: 0.063883
Train Epoch: 38 [25344/54000 (47%)] Loss: 0.132185
Train Epoch: 38 [26752/54000 (50%)] Loss: 0.107593
Train Epoch: 38 [28160/54000 (52%)] Loss: 0.081784
Train Epoch: 38 [29568/54000 (55%)] Loss: 0.092768
Train Epoch: 38 [30976/54000 (57%)] Loss: 0.122817
Train Epoch: 38 [32384/54000 (60%)] Loss: 0.037802
Train Epoch: 38 [33792/54000 (63%)] Loss: 0.143304
Train Epoch: 38 [35200/54000 (65%)] Loss: 0.118462
Train Epoch: 38 [36608/54000 (68%)] Loss: 0.099895
Train Epoch: 38 [38016/54000 (70%)] Loss: 0.060148
Train Epoch: 38 [39424/54000 (73%)] Loss: 0.087494
Train Epoch: 38 [40832/54000 (76%)] Loss: 0.039721
Train Epoch: 38 [42240/54000 (78%)] Loss: 0.111765
Train Epoch: 38 [43648/54000 (81%)] Loss: 0.141694
Train Epoch: 38 [45056/54000 (83%)] Loss: 0.100822
Train Epoch: 38 [46464/54000 (86%)] Loss: 0.062999
Train Epoch: 38 [47872/54000 (89%)] Loss: 0.066024
Train Epoch: 38 [49280/54000 (91%)] Loss: 0.052156
Train Epoch: 38 [50688/54000 (94%)] Loss: 0.075910
Train Epoch: 38 [52096/54000 (96%)] Loss: 0.098019
Train Epoch: 38 [53504/54000 (99%)] Loss: 0.159874
    epoch          : 38
    loss           : 0.09224229881107383
    accuracy       : 0.971511086662153
    top_k_acc      : 0.9972759394041978
    val_loss       : 0.037849362394673396
    val_accuracy   : 0.9881981382978723
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch38.pth ...
Train Epoch: 39 [0/54000 (0%)] Loss: 0.105865
Train Epoch: 39 [1408/54000 (3%)] Loss: 0.076311
Train Epoch: 39 [2816/54000 (5%)] Loss: 0.077878
Train Epoch: 39 [4224/54000 (8%)] Loss: 0.079575
Train Epoch: 39 [5632/54000 (10%)] Loss: 0.240677
Train Epoch: 39 [7040/54000 (13%)] Loss: 0.072071
Train Epoch: 39 [8448/54000 (16%)] Loss: 0.108262
Train Epoch: 39 [9856/54000 (18%)] Loss: 0.101404
Train Epoch: 39 [11264/54000 (21%)] Loss: 0.157214
Train Epoch: 39 [12672/54000 (23%)] Loss: 0.143480
Train Epoch: 39 [14080/54000 (26%)] Loss: 0.127271
Train Epoch: 39 [15488/54000 (29%)] Loss: 0.132723
Train Epoch: 39 [16896/54000 (31%)] Loss: 0.175200
Train Epoch: 39 [18304/54000 (34%)] Loss: 0.040370
Train Epoch: 39 [19712/54000 (37%)] Loss: 0.241552
Train Epoch: 39 [21120/54000 (39%)] Loss: 0.156019
Train Epoch: 39 [22528/54000 (42%)] Loss: 0.092823
Train Epoch: 39 [23936/54000 (44%)] Loss: 0.121209
Train Epoch: 39 [25344/54000 (47%)] Loss: 0.029556
Train Epoch: 39 [26752/54000 (50%)] Loss: 0.071836
Train Epoch: 39 [28160/54000 (52%)] Loss: 0.065823
Train Epoch: 39 [29568/54000 (55%)] Loss: 0.055397
Train Epoch: 39 [30976/54000 (57%)] Loss: 0.057622
Train Epoch: 39 [32384/54000 (60%)] Loss: 0.089036
Train Epoch: 39 [33792/54000 (63%)] Loss: 0.090861
Train Epoch: 39 [35200/54000 (65%)] Loss: 0.090145
Train Epoch: 39 [36608/54000 (68%)] Loss: 0.075856
Train Epoch: 39 [38016/54000 (70%)] Loss: 0.104519
Train Epoch: 39 [39424/54000 (73%)] Loss: 0.048912
Train Epoch: 39 [40832/54000 (76%)] Loss: 0.086275
Train Epoch: 39 [42240/54000 (78%)] Loss: 0.188685
Train Epoch: 39 [43648/54000 (81%)] Loss: 0.073120
Train Epoch: 39 [45056/54000 (83%)] Loss: 0.057198
Train Epoch: 39 [46464/54000 (86%)] Loss: 0.181167
Train Epoch: 39 [47872/54000 (89%)] Loss: 0.195372
Train Epoch: 39 [49280/54000 (91%)] Loss: 0.059846
Train Epoch: 39 [50688/54000 (94%)] Loss: 0.117176
Train Epoch: 39 [52096/54000 (96%)] Loss: 0.112530
Train Epoch: 39 [53504/54000 (99%)] Loss: 0.152735
    epoch          : 39
    loss           : 0.09774254884800357
    accuracy       : 0.9694640741367637
    top_k_acc      : 0.9970749407582938
    val_loss       : 0.037271874793309796
    val_accuracy   : 0.9891717325227964
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch39.pth ...
Train Epoch: 40 [0/54000 (0%)] Loss: 0.095749
Train Epoch: 40 [1408/54000 (3%)] Loss: 0.088315
Train Epoch: 40 [2816/54000 (5%)] Loss: 0.039181
Train Epoch: 40 [4224/54000 (8%)] Loss: 0.122691
Train Epoch: 40 [5632/54000 (10%)] Loss: 0.214547
Train Epoch: 40 [7040/54000 (13%)] Loss: 0.188205
Train Epoch: 40 [8448/54000 (16%)] Loss: 0.031048
Train Epoch: 40 [9856/54000 (18%)] Loss: 0.059005
Train Epoch: 40 [11264/54000 (21%)] Loss: 0.119043
Train Epoch: 40 [12672/54000 (23%)] Loss: 0.125115
Train Epoch: 40 [14080/54000 (26%)] Loss: 0.100486
Train Epoch: 40 [15488/54000 (29%)] Loss: 0.099731
Train Epoch: 40 [16896/54000 (31%)] Loss: 0.046628
Train Epoch: 40 [18304/54000 (34%)] Loss: 0.061104
Train Epoch: 40 [19712/54000 (37%)] Loss: 0.040261
Train Epoch: 40 [21120/54000 (39%)] Loss: 0.109410
Train Epoch: 40 [22528/54000 (42%)] Loss: 0.104668
Train Epoch: 40 [23936/54000 (44%)] Loss: 0.056437
Train Epoch: 40 [25344/54000 (47%)] Loss: 0.074859
Train Epoch: 40 [26752/54000 (50%)] Loss: 0.102933
Train Epoch: 40 [28160/54000 (52%)] Loss: 0.094249
Train Epoch: 40 [29568/54000 (55%)] Loss: 0.082338
Train Epoch: 40 [30976/54000 (57%)] Loss: 0.105324
Train Epoch: 40 [32384/54000 (60%)] Loss: 0.088742
Train Epoch: 40 [33792/54000 (63%)] Loss: 0.104970
Train Epoch: 40 [35200/54000 (65%)] Loss: 0.018139
Train Epoch: 40 [36608/54000 (68%)] Loss: 0.057882
Train Epoch: 40 [38016/54000 (70%)] Loss: 0.067422
Train Epoch: 40 [39424/54000 (73%)] Loss: 0.114176
Train Epoch: 40 [40832/54000 (76%)] Loss: 0.137629
Train Epoch: 40 [42240/54000 (78%)] Loss: 0.064013
Train Epoch: 40 [43648/54000 (81%)] Loss: 0.055192
Train Epoch: 40 [45056/54000 (83%)] Loss: 0.106789
Train Epoch: 40 [46464/54000 (86%)] Loss: 0.072675
Train Epoch: 40 [47872/54000 (89%)] Loss: 0.088665
Train Epoch: 40 [49280/54000 (91%)] Loss: 0.079876
Train Epoch: 40 [50688/54000 (94%)] Loss: 0.049924
Train Epoch: 40 [52096/54000 (96%)] Loss: 0.055446
Train Epoch: 40 [53504/54000 (99%)] Loss: 0.048518
    epoch          : 40
    loss           : 0.09170501060360134
    accuracy       : 0.9719712677725119
    top_k_acc      : 0.9970167569397428
    val_loss       : 0.03763717375175552
    val_accuracy   : 0.9895041793313071
    val_top_k_acc  : 0.9993113601823709
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch40.pth ...
Train Epoch: 41 [0/54000 (0%)] Loss: 0.061825
Train Epoch: 41 [1408/54000 (3%)] Loss: 0.044266
Train Epoch: 41 [2816/54000 (5%)] Loss: 0.089720
Train Epoch: 41 [4224/54000 (8%)] Loss: 0.100966
Train Epoch: 41 [5632/54000 (10%)] Loss: 0.037384
Train Epoch: 41 [7040/54000 (13%)] Loss: 0.119024
Train Epoch: 41 [8448/54000 (16%)] Loss: 0.082317
Train Epoch: 41 [9856/54000 (18%)] Loss: 0.113641
Train Epoch: 41 [11264/54000 (21%)] Loss: 0.077739
Train Epoch: 41 [12672/54000 (23%)] Loss: 0.044483
Train Epoch: 41 [14080/54000 (26%)] Loss: 0.114967
Train Epoch: 41 [15488/54000 (29%)] Loss: 0.057102
Train Epoch: 41 [16896/54000 (31%)] Loss: 0.053922
Train Epoch: 41 [18304/54000 (34%)] Loss: 0.072728
Train Epoch: 41 [19712/54000 (37%)] Loss: 0.104610
Train Epoch: 41 [21120/54000 (39%)] Loss: 0.050922
Train Epoch: 41 [22528/54000 (42%)] Loss: 0.062473
Train Epoch: 41 [23936/54000 (44%)] Loss: 0.077329
Train Epoch: 41 [25344/54000 (47%)] Loss: 0.062833
Train Epoch: 41 [26752/54000 (50%)] Loss: 0.055186
Train Epoch: 41 [28160/54000 (52%)] Loss: 0.110490
Train Epoch: 41 [29568/54000 (55%)] Loss: 0.166576
Train Epoch: 41 [30976/54000 (57%)] Loss: 0.062634
Train Epoch: 41 [32384/54000 (60%)] Loss: 0.217256
Train Epoch: 41 [33792/54000 (63%)] Loss: 0.065789
Train Epoch: 41 [35200/54000 (65%)] Loss: 0.200196
Train Epoch: 41 [36608/54000 (68%)] Loss: 0.090136
Train Epoch: 41 [38016/54000 (70%)] Loss: 0.022803
Train Epoch: 41 [39424/54000 (73%)] Loss: 0.228338
Train Epoch: 41 [40832/54000 (76%)] Loss: 0.064105
Train Epoch: 41 [42240/54000 (78%)] Loss: 0.119741
Train Epoch: 41 [43648/54000 (81%)] Loss: 0.061061
Train Epoch: 41 [45056/54000 (83%)] Loss: 0.068955
Train Epoch: 41 [46464/54000 (86%)] Loss: 0.073667
Train Epoch: 41 [47872/54000 (89%)] Loss: 0.034911
Train Epoch: 41 [49280/54000 (91%)] Loss: 0.134841
Train Epoch: 41 [50688/54000 (94%)] Loss: 0.077341
Train Epoch: 41 [52096/54000 (96%)] Loss: 0.084659
Train Epoch: 41 [53504/54000 (99%)] Loss: 0.092938
    epoch          : 41
    loss           : 0.09513737439173517
    accuracy       : 0.9714158767772512
    top_k_acc      : 0.9972785841232228
    val_loss       : 0.03414595234604116
    val_accuracy   : 0.9895041793313071
    val_top_k_acc  : 0.9991688829787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch41.pth ...
Train Epoch: 42 [0/54000 (0%)] Loss: 0.066176
Train Epoch: 42 [1408/54000 (3%)] Loss: 0.117279
Train Epoch: 42 [2816/54000 (5%)] Loss: 0.020643
Train Epoch: 42 [4224/54000 (8%)] Loss: 0.028539
Train Epoch: 42 [5632/54000 (10%)] Loss: 0.135024
Train Epoch: 42 [7040/54000 (13%)] Loss: 0.058305
Train Epoch: 42 [8448/54000 (16%)] Loss: 0.102245
Train Epoch: 42 [9856/54000 (18%)] Loss: 0.058667
Train Epoch: 42 [11264/54000 (21%)] Loss: 0.067023
Train Epoch: 42 [12672/54000 (23%)] Loss: 0.049085
Train Epoch: 42 [14080/54000 (26%)] Loss: 0.059053
Train Epoch: 42 [15488/54000 (29%)] Loss: 0.101788
Train Epoch: 42 [16896/54000 (31%)] Loss: 0.044484
Train Epoch: 42 [18304/54000 (34%)] Loss: 0.073212
Train Epoch: 42 [19712/54000 (37%)] Loss: 0.081177
Train Epoch: 42 [21120/54000 (39%)] Loss: 0.142055
Train Epoch: 42 [22528/54000 (42%)] Loss: 0.033987
Train Epoch: 42 [23936/54000 (44%)] Loss: 0.188416
Train Epoch: 42 [25344/54000 (47%)] Loss: 0.093068
Train Epoch: 42 [26752/54000 (50%)] Loss: 0.066327
Train Epoch: 42 [28160/54000 (52%)] Loss: 0.053093
Train Epoch: 42 [29568/54000 (55%)] Loss: 0.082052
Train Epoch: 42 [30976/54000 (57%)] Loss: 0.056879
Train Epoch: 42 [32384/54000 (60%)] Loss: 0.094709
Train Epoch: 42 [33792/54000 (63%)] Loss: 0.053719
Train Epoch: 42 [35200/54000 (65%)] Loss: 0.123268
Train Epoch: 42 [36608/54000 (68%)] Loss: 0.118055
Train Epoch: 42 [38016/54000 (70%)] Loss: 0.068082
Train Epoch: 42 [39424/54000 (73%)] Loss: 0.062378
Train Epoch: 42 [40832/54000 (76%)] Loss: 0.043081
Train Epoch: 42 [42240/54000 (78%)] Loss: 0.053650
Train Epoch: 42 [43648/54000 (81%)] Loss: 0.014749
Train Epoch: 42 [45056/54000 (83%)] Loss: 0.099568
Train Epoch: 42 [46464/54000 (86%)] Loss: 0.114451
Train Epoch: 42 [47872/54000 (89%)] Loss: 0.071772
Train Epoch: 42 [49280/54000 (91%)] Loss: 0.131080
Train Epoch: 42 [50688/54000 (94%)] Loss: 0.144976
Train Epoch: 42 [52096/54000 (96%)] Loss: 0.139204
Train Epoch: 42 [53504/54000 (99%)] Loss: 0.051051
    epoch          : 42
    loss           : 0.09541039523929004
    accuracy       : 0.9715692704807042
    top_k_acc      : 0.9971833742383209
    val_loss       : 0.03534220296275267
    val_accuracy   : 0.988530585106383
    val_top_k_acc  : 0.999501329787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch42.pth ...
Train Epoch: 43 [0/54000 (0%)] Loss: 0.068082
Train Epoch: 43 [1408/54000 (3%)] Loss: 0.190127
Train Epoch: 43 [2816/54000 (5%)] Loss: 0.065486
Train Epoch: 43 [4224/54000 (8%)] Loss: 0.042941
Train Epoch: 43 [5632/54000 (10%)] Loss: 0.075803
Train Epoch: 43 [7040/54000 (13%)] Loss: 0.094999
Train Epoch: 43 [8448/54000 (16%)] Loss: 0.051589
Train Epoch: 43 [9856/54000 (18%)] Loss: 0.150701
Train Epoch: 43 [11264/54000 (21%)] Loss: 0.110641
Train Epoch: 43 [12672/54000 (23%)] Loss: 0.055810
Train Epoch: 43 [14080/54000 (26%)] Loss: 0.111280
Train Epoch: 43 [15488/54000 (29%)] Loss: 0.057315
Train Epoch: 43 [16896/54000 (31%)] Loss: 0.025387
Train Epoch: 43 [18304/54000 (34%)] Loss: 0.105587
Train Epoch: 43 [19712/54000 (37%)] Loss: 0.068636
Train Epoch: 43 [21120/54000 (39%)] Loss: 0.098570
Train Epoch: 43 [22528/54000 (42%)] Loss: 0.072477
Train Epoch: 43 [23936/54000 (44%)] Loss: 0.062320
Train Epoch: 43 [25344/54000 (47%)] Loss: 0.109872
Train Epoch: 43 [26752/54000 (50%)] Loss: 0.073085
Train Epoch: 43 [28160/54000 (52%)] Loss: 0.151497
Train Epoch: 43 [29568/54000 (55%)] Loss: 0.066213
Train Epoch: 43 [30976/54000 (57%)] Loss: 0.063629
Train Epoch: 43 [32384/54000 (60%)] Loss: 0.035868
Train Epoch: 43 [33792/54000 (63%)] Loss: 0.069645
Train Epoch: 43 [35200/54000 (65%)] Loss: 0.061967
Train Epoch: 43 [36608/54000 (68%)] Loss: 0.050763
Train Epoch: 43 [38016/54000 (70%)] Loss: 0.223181
Train Epoch: 43 [39424/54000 (73%)] Loss: 0.042183
Train Epoch: 43 [40832/54000 (76%)] Loss: 0.067208
Train Epoch: 43 [42240/54000 (78%)] Loss: 0.062035
Train Epoch: 43 [43648/54000 (81%)] Loss: 0.046409
Train Epoch: 43 [45056/54000 (83%)] Loss: 0.047210
Train Epoch: 43 [46464/54000 (86%)] Loss: 0.079769
Train Epoch: 43 [47872/54000 (89%)] Loss: 0.070106
Train Epoch: 43 [49280/54000 (91%)] Loss: 0.080252
Train Epoch: 43 [50688/54000 (94%)] Loss: 0.083506
Train Epoch: 43 [52096/54000 (96%)] Loss: 0.067707
Train Epoch: 43 [53504/54000 (99%)] Loss: 0.138547
    epoch          : 43
    loss           : 0.09548878068351561
    accuracy       : 0.9708551963439404
    top_k_acc      : 0.9970008886255924
    val_loss       : 0.03404288316155447
    val_accuracy   : 0.9881743920972644
    val_top_k_acc  : 0.9996675531914894
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch43.pth ...
Train Epoch: 44 [0/54000 (0%)] Loss: 0.131017
Train Epoch: 44 [1408/54000 (3%)] Loss: 0.078258
Train Epoch: 44 [2816/54000 (5%)] Loss: 0.154805
Train Epoch: 44 [4224/54000 (8%)] Loss: 0.076135
Train Epoch: 44 [5632/54000 (10%)] Loss: 0.082712
Train Epoch: 44 [7040/54000 (13%)] Loss: 0.137805
Train Epoch: 44 [8448/54000 (16%)] Loss: 0.085491
Train Epoch: 44 [9856/54000 (18%)] Loss: 0.079081
Train Epoch: 44 [11264/54000 (21%)] Loss: 0.042557
Train Epoch: 44 [12672/54000 (23%)] Loss: 0.097801
Train Epoch: 44 [14080/54000 (26%)] Loss: 0.074342
Train Epoch: 44 [15488/54000 (29%)] Loss: 0.125782
Train Epoch: 44 [16896/54000 (31%)] Loss: 0.086053
Train Epoch: 44 [18304/54000 (34%)] Loss: 0.176826
Train Epoch: 44 [19712/54000 (37%)] Loss: 0.111103
Train Epoch: 44 [21120/54000 (39%)] Loss: 0.060495
Train Epoch: 44 [22528/54000 (42%)] Loss: 0.086681
Train Epoch: 44 [23936/54000 (44%)] Loss: 0.066704
Train Epoch: 44 [25344/54000 (47%)] Loss: 0.092749
Train Epoch: 44 [26752/54000 (50%)] Loss: 0.125641
Train Epoch: 44 [28160/54000 (52%)] Loss: 0.089998
Train Epoch: 44 [29568/54000 (55%)] Loss: 0.077707
Train Epoch: 44 [30976/54000 (57%)] Loss: 0.054595
Train Epoch: 44 [32384/54000 (60%)] Loss: 0.072519
Train Epoch: 44 [33792/54000 (63%)] Loss: 0.159096
Train Epoch: 44 [35200/54000 (65%)] Loss: 0.064343
Train Epoch: 44 [36608/54000 (68%)] Loss: 0.119838
Train Epoch: 44 [38016/54000 (70%)] Loss: 0.065946
Train Epoch: 44 [39424/54000 (73%)] Loss: 0.083013
Train Epoch: 44 [40832/54000 (76%)] Loss: 0.115742
Train Epoch: 44 [42240/54000 (78%)] Loss: 0.074582
Train Epoch: 44 [43648/54000 (81%)] Loss: 0.082750
Train Epoch: 44 [45056/54000 (83%)] Loss: 0.083419
Train Epoch: 44 [46464/54000 (86%)] Loss: 0.066069
Train Epoch: 44 [47872/54000 (89%)] Loss: 0.043860
Train Epoch: 44 [49280/54000 (91%)] Loss: 0.066436
Train Epoch: 44 [50688/54000 (94%)] Loss: 0.041764
Train Epoch: 44 [52096/54000 (96%)] Loss: 0.099018
Train Epoch: 44 [53504/54000 (99%)] Loss: 0.030515
    epoch          : 44
    loss           : 0.09303460593039603
    accuracy       : 0.9727990648273527
    top_k_acc      : 0.9971675059241706
    val_loss       : 0.03665727877276058
    val_accuracy   : 0.9886968085106383
    val_top_k_acc  : 0.9993351063829787
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch44.pth ...
Train Epoch: 45 [0/54000 (0%)] Loss: 0.064786
Train Epoch: 45 [1408/54000 (3%)] Loss: 0.138503
Train Epoch: 45 [2816/54000 (5%)] Loss: 0.068605
Train Epoch: 45 [4224/54000 (8%)] Loss: 0.063553
Train Epoch: 45 [5632/54000 (10%)] Loss: 0.101821
Train Epoch: 45 [7040/54000 (13%)] Loss: 0.056002
Train Epoch: 45 [8448/54000 (16%)] Loss: 0.169862
Train Epoch: 45 [9856/54000 (18%)] Loss: 0.134792
Train Epoch: 45 [11264/54000 (21%)] Loss: 0.091166
Train Epoch: 45 [12672/54000 (23%)] Loss: 0.049179
Train Epoch: 45 [14080/54000 (26%)] Loss: 0.095279
Train Epoch: 45 [15488/54000 (29%)] Loss: 0.145587
Train Epoch: 45 [16896/54000 (31%)] Loss: 0.089605
Train Epoch: 45 [18304/54000 (34%)] Loss: 0.071702
Train Epoch: 45 [19712/54000 (37%)] Loss: 0.064612
Train Epoch: 45 [21120/54000 (39%)] Loss: 0.091519
Train Epoch: 45 [22528/54000 (42%)] Loss: 0.101064
Train Epoch: 45 [23936/54000 (44%)] Loss: 0.177253
Train Epoch: 45 [25344/54000 (47%)] Loss: 0.138424
Train Epoch: 45 [26752/54000 (50%)] Loss: 0.171499
Train Epoch: 45 [28160/54000 (52%)] Loss: 0.074019
Train Epoch: 45 [29568/54000 (55%)] Loss: 0.052505
Train Epoch: 45 [30976/54000 (57%)] Loss: 0.203400
Train Epoch: 45 [32384/54000 (60%)] Loss: 0.139977
Train Epoch: 45 [33792/54000 (63%)] Loss: 0.037859
Train Epoch: 45 [35200/54000 (65%)] Loss: 0.054618
Train Epoch: 45 [36608/54000 (68%)] Loss: 0.063929
Train Epoch: 45 [38016/54000 (70%)] Loss: 0.092166
Train Epoch: 45 [39424/54000 (73%)] Loss: 0.085069
Train Epoch: 45 [40832/54000 (76%)] Loss: 0.052635
Train Epoch: 45 [42240/54000 (78%)] Loss: 0.122251
Train Epoch: 45 [43648/54000 (81%)] Loss: 0.096386
Train Epoch: 45 [45056/54000 (83%)] Loss: 0.037219
Train Epoch: 45 [46464/54000 (86%)] Loss: 0.108541
Train Epoch: 45 [47872/54000 (89%)] Loss: 0.041960
Train Epoch: 45 [49280/54000 (91%)] Loss: 0.140661
Train Epoch: 45 [50688/54000 (94%)] Loss: 0.158312
Train Epoch: 45 [52096/54000 (96%)] Loss: 0.131670
Train Epoch: 45 [53504/54000 (99%)] Loss: 0.116118
    epoch          : 45
    loss           : 0.09171031364703207
    accuracy       : 0.9716486120514556
    top_k_acc      : 0.9972600710900474
    val_loss       : 0.03569985168213223
    val_accuracy   : 0.9893617021276596
    val_top_k_acc  : 0.9991688829787234
Saving checkpoint: saved/models/Mnist_LeNet/0722_140340/checkpoint-epoch45.pth ...
Train Epoch: 46 [0/54000 (0%)] Loss: 0.035129
Train Epoch: 46 [1408/54000 (3%)] Loss: 0.060531
Train Epoch: 46 [2816/54000 (5%)] Loss: 0.141322
Train Epoch: 46 [4224/54000 (8%)] Loss: 0.045136
Train Epoch: 46 [5632/54000 (10%)] Loss: 0.111405
Train Epoch: 46 [7040/54000 (13%)] Loss: 0.030723
Train Epoch: 46 [8448/54000 (16%)] Loss: 0.136971
Train Epoch: 46 [9856/54000 (18%)] Loss: 0.135877
Train Epoch: 46 [11264/54000 (21%)] Loss: 0.198417
Train Epoch: 46 [12672/54000 (23%)] Loss: 0.107843
Train Epoch: 46 [14080/54000 (26%)] Loss: 0.030834
Train Epoch: 46 [15488/54000 (29%)] Loss: 0.162332
Train Epoch: 46 [16896/54000 (31%)] Loss: 0.098182
Train Epoch: 46 [18304/54000 (34%)] Loss: 0.085969
Train Epoch: 46 [19712/54000 (37%)] Loss: 0.128704
Train Epoch: 46 [21120/54000 (39%)] Loss: 0.034047
Train Epoch: 46 [22528/54000 (42%)] Loss: 0.095470
Train Epoch: 46 [23936/54000 (44%)] Loss: 0.082977
Train Epoch: 46 [25344/54000 (47%)] Loss: 0.168601
Train Epoch: 46 [26752/54000 (50%)] Loss: 0.037510
Train Epoch: 46 [28160/54000 (52%)] Loss: 0.092670
Train Epoch: 46 [29568/54000 (55%)] Loss: 0.054018
Train Epoch: 46 [30976/54000 (57%)] Loss: 0.098995
Train Epoch: 46 [32384/54000 (60%)] Loss: 0.062006
Train Epoch: 46 [33792/54000 (63%)] Loss: 0.075248
Train Epoch: 46 [35200/54000 (65%)] Loss: 0.086751
Train Epoch: 46 [36608/54000 (68%)] Loss: 0.073657
Train Epoch: 46 [38016/54000 (70%)] Loss: 0.067035
Train Epoch: 46 [39424/54000 (73%)] Loss: 0.136418
Train Epoch: 46 [40832/54000 (76%)] Loss: 0.134104
Train Epoch: 46 [42240/54000 (78%)] Loss: 0.122963
Train Epoch: 46 [43648/54000 (81%)] Loss: 0.122107
Train Epoch: 46 [45056/54000 (83%)] Loss: 0.237209
Train Epoch: 46 [46464/54000 (86%)] Loss: 0.062584
Train Epoch: 46 [47872/54000 (89%)] Loss: 0.077283
Train Epoch: 46 [49280/54000 (91%)] Loss: 0.139206
Train Epoch: 46 [50688/54000 (94%)] Loss: 0.126840
Train Epoch: 46 [52096/54000 (96%)] Loss: 0.105001
Train Epoch: 46 [53504/54000 (99%)] Loss: 0.094605
    epoch          : 46
    loss           : 0.09230717961892697
    accuracy       : 0.9722463185511172
    top_k_acc      : 0.9971489928909952
    val_loss       : 0.03466828447206818
    val_accuracy   : 0.9893379559270518
    val_top_k_acc  : 0.999501329787234
Validation performance didn't improve for 10 epochs. Training stops.

git clone으로 불러온 data_loader/data_loaders.py 파일입니다.
저는 MNIST대신 CIFAR-10 데이터셋을 로드하도록 수정하겠습니다.

수정 전

from torchvision import datasets, transforms
from base import BaseDataLoader


class MnistDataLoader(BaseDataLoader):
    """
    MNIST data loading demo using BaseDataLoader
    """
    def __init__(self, data_dir, batch_size, shuffle=True, validation_split=0.0, num_workers=1, training=True):
        trsfm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,))
        ])
        self.data_dir = data_dir
        self.dataset = datasets.MNIST(self.data_dir, train=training, download=True, transform=trsfm)
        super().__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)

수정 후

from torchvision import datasets, transforms
from base import BaseDataLoader


class Cifar10DataLoader(BaseDataLoader):
    """
    CIFAR-10 data loading using BaseDataLoader
    """
    def __init__(self, data_dir, batch_size, shuffle=True, validation_split=0.0, num_workers=1, training=True):
        trsfm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
        ])
        self.data_dir = data_dir
        self.dataset = datasets.CIFAR10(self.data_dir, train=training, download=True, transform=trsfm)
        super().__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)

CIFAR-10 데이터셋을 불러오고, 이미지를 정규화하는 데 필요한 평균과 표준편차 값을 사용한다. BaseDataLoader 클래스를 상속받아 데이터셋을 초기화하고, 나머지 설정을 그대로 사용한다.

config.json파일 수정
수정 전

{
    "name": "Mnist_LeNet",
    "n_gpu": 1,

    "arch": {
        "type": "MnistModel",
        "args": {}
    },
    "data_loader": {
        "type": "MnistDataLoader",
        "args":{
            "data_dir": "data/",
            "batch_size": 128,
            "shuffle": true,
            "validation_split": 0.1,
            "num_workers": 2
        }
    },
    "optimizer": {
        "type": "Adam",
        "args":{
            "lr": 0.001,
            "weight_decay": 0,
            "amsgrad": true
        }
    },
    "loss": "nll_loss",
    "metrics": [
        "accuracy", "top_k_acc"
    ],
    "lr_scheduler": {
        "type": "StepLR",
        "args": {
            "step_size": 50,
            "gamma": 0.1
        }
    },
    "trainer": {
        "epochs": 100,

        "save_dir": "saved/",
        "save_period": 1,
        "verbosity": 2,
        
        "monitor": "min val_loss",
        "early_stop": 10,

        "tensorboard": true
    }
}

수정 후
config.json 파일을 CIFAR-10 데이터셋을 사용할 수 있도록 수정하려면, data_loader 항목에서 MnistDataLoader를 Cifar10DataLoader로 변경하고, 필요한 매개변수들을 CIFAR-10에 맞게 조정해야 한다.

{
    "name": "Cifar10_LeNet",
    "n_gpu": 1,

    "arch": {
        "type": "LeNet", 
        "args": {}
    },
    "data_loader": {
        "type": "Cifar10DataLoader",
        "args": {
            "data_dir": "data/",
            "batch_size": 128,
            "shuffle": true,
            "validation_split": 0.1,
            "num_workers": 2
        }
    },
    "optimizer": {
        "type": "Adam",
        "args": {
            "lr": 0.001,
            "weight_decay": 0,
            "amsgrad": true
        }
    },
    "loss": "nll_loss",
    "metrics": [
        "accuracy", "top_k_acc"
    ],
    "lr_scheduler": {
        "type": "StepLR",
        "args": {
            "step_size": 50,
            "gamma": 0.1
        }
    },
    "trainer": {
        "epochs": 100,

        "save_dir": "saved/",
        "save_period": 1,
        "verbosity": 2,
        
        "monitor": "min val_loss",
        "early_stop": 10,

        "tensorboard": true
    }
}

주요 변경 사항

name: "Cifar10_LeNet"으로 변경하였습니다. CIFAR-10 데이터셋에 맞게 업데이트한 것이다.
arch: "type"을 "LeNet"으로 설정했다. 이 부분은 모델의 아키텍처를 지정하는 것으로, CIFAR-10 데이터셋에 적합한 모델을 사용하도록 설정했다. 모델 이름은 사용하려는 실제 모델의 이름에 맞게 조정해야 한다.

import torch.nn as nn
import torch.nn.functional as F
from base import BaseModel


class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

data_loader: "type"을 "Cifar10DataLoader"로 변경하였습니다. 이 변경은 CIFAR-10 데이터셋을 사용하기 위한 것이다.
data_loader/data_loaders.py 파일을 열고, Cifar10DataLoader 클래스를 추가했다.
Cifar10DataLoader 클래스를 기반으로 학습하도록 변경한다.

from torchvision import datasets, transforms
from base import BaseDataLoader


class MnistDataLoader(BaseDataLoader):
    """
    MNIST data loading demo using BaseDataLoader
    """
    def __init__(self, data_dir, batch_size, shuffle=True, validation_split=0.0, num_workers=1, training=True):
        trsfm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,))
        ])
        self.data_dir = data_dir
        self.dataset = datasets.MNIST(self.data_dir, train=training, download=True, transform=trsfm)
        super().__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)


class Cifar10DataLoader(BaseDataLoader):
    """
    CIFAR-10 data loading demo using BaseDataLoader
    """
    def __init__(self, data_dir, batch_size, shuffle=True, validation_split=0.0, num_workers=1, training=True):
        trsfm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])
        self.data_dir = data_dir
        self.dataset = datasets.CIFAR10(self.data_dir, train=training, download=True, transform=trsfm)
        super().__init__(self.dataset, batch_size, shuffle, validation_split, num_workers)

args:
- data_dir: CIFAR-10 데이터셋을 저장할 디렉토리이다. 기본값은 "data/"로 설정하였다.
- batch_size: 배치 크기를 128로 설정하였다.
- shuffle: 데이터를 무작위로 섞도록 설정하였다.
- validation_split: 10%의 데이터를 검증 데이터로 사용할 수 있도록 설정하였다.
- num_workers: 데이터 로딩을 위한 워커 수를 2로 설정하였습니다.
trainer: "epochs"와 "save_period"는 CIFAR-10 데이터셋을 사용할 때 학습 및 저장 주기에 따라 조정될 수 있다. 필요한 경우 값을 조정할 수 있다.

수정된 config.json 파일을 사용하여 CIFAR-10 데이터셋으로 모델을 학습할 수 있다.

(bohyun) user@server:~/Desktop/BOHYUN/0726/speedrun/pytorch-template$ python train.py -c config.json
/home/user/miniconda3/envs/bohyun/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/user/miniconda3/envs/bohyun/lib/python3.8/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to data/cifar-10-python.tar.gz
 88%|██████████████████████████████████████████████▋      | 150208512/170498071 [00:12<00:04, 506

Loss 굉장히 엄청나..!
loss.py에 cross_entropy_loss를 정의했다.
train.py에서 직접 torch.nn.functional.cross_entropy를 사용하도록 변경했다.

def main(config):
    logger = config.get_logger('train')

    # setup data_loader instances
    data_loader = config.init_obj('data_loader', module_data)
    valid_data_loader = data_loader.split_validation()

    # build model architecture, then print to console
    model = config.init_obj('arch', module_arch)
    logger.info(model)

    # prepare for (multi-device) GPU training
    device, device_ids = prepare_device(config['n_gpu'])
    model = model.to(device)
    if len(device_ids) > 1:
        model = torch.nn.DataParallel(model, device_ids=device_ids)

    # get function handles of loss and metrics
    criterion = torch.nn.functional.cross_entropy

다행히 loss값이 줄어들었다!

(bohyun) user@server:~/Desktop/BOHYUN/0726/speedrun/pytorch-template$ python train.py -c config.json
/home/user/miniconda3/envs/bohyun/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/home/user/miniconda3/envs/bohyun/lib/python3.8/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
Files already downloaded and verified
LeNet(
  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)
Warning: visualization (Tensorboard) is configured to use, but currently not installed on this machine. Please install TensorboardX with 'pip install tensorboardx', upgrade PyTorch to version >= 1.1 to use 'torch.utils.tensorboard' or turn off the option in the 'config.json' file.
Train Epoch: 1 [0/45000 (0%)] Loss: 2.311082
Train Epoch: 1 [1408/45000 (3%)] Loss: 2.284087
Train Epoch: 1 [2816/45000 (6%)] Loss: 2.129331
Train Epoch: 1 [4224/45000 (9%)] Loss: 2.028467
Train Epoch: 1 [5632/45000 (13%)] Loss: 2.004406
Train Epoch: 1 [7040/45000 (16%)] Loss: 2.095008
Train Epoch: 1 [8448/45000 (19%)] Loss: 1.890626
Train Epoch: 1 [9856/45000 (22%)] Loss: 2.010740
Train Epoch: 1 [11264/45000 (25%)] Loss: 1.741392
Train Epoch: 1 [12672/45000 (28%)] Loss: 1.962260
Train Epoch: 1 [14080/45000 (31%)] Loss: 1.826351
Train Epoch: 1 [15488/45000 (34%)] Loss: 1.859895
Train Epoch: 1 [16896/45000 (38%)] Loss: 1.777300
Train Epoch: 1 [18304/45000 (41%)] Loss: 1.714267
Train Epoch: 1 [19712/45000 (44%)] Loss: 1.648159
Train Epoch: 1 [21120/45000 (47%)] Loss: 1.852577
Train Epoch: 1 [22528/45000 (50%)] Loss: 1.649575
Train Epoch: 1 [23936/45000 (53%)] Loss: 1.578230
Train Epoch: 1 [25344/45000 (56%)] Loss: 1.633049
Train Epoch: 1 [26752/45000 (59%)] Loss: 1.593858
Train Epoch: 1 [28160/45000 (63%)] Loss: 1.544563
Train Epoch: 1 [29568/45000 (66%)] Loss: 1.594563
Train Epoch: 1 [30976/45000 (69%)] Loss: 1.618854
Train Epoch: 1 [32384/45000 (72%)] Loss: 1.571820
Train Epoch: 1 [33792/45000 (75%)] Loss: 1.673366
Train Epoch: 1 [35200/45000 (78%)] Loss: 1.637884
Train Epoch: 1 [36608/45000 (81%)] Loss: 1.585719
Train Epoch: 1 [38016/45000 (84%)] Loss: 1.590082
Train Epoch: 1 [39424/45000 (88%)] Loss: 1.654451
Train Epoch: 1 [40832/45000 (91%)] Loss: 1.532288
Train Epoch: 1 [42240/45000 (94%)] Loss: 1.667048
Train Epoch: 1 [43648/45000 (97%)] Loss: 1.455048
    epoch          : 1
    loss           : 1.757788048549132
    accuracy       : 0.356060606060606
    top_k_acc      : 0.6880080097853536
    val_loss       : 1.5926437616348266
    val_accuracy   : 0.4203125
    val_top_k_acc  : 0.764453125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch1.pth ...
Saving current best: model_best.pth ...
Train Epoch: 2 [0/45000 (0%)] Loss: 1.693297
Train Epoch: 2 [1408/45000 (3%)] Loss: 1.573273
Train Epoch: 2 [2816/45000 (6%)] Loss: 1.532989
Train Epoch: 2 [4224/45000 (9%)] Loss: 1.577287
Train Epoch: 2 [5632/45000 (13%)] Loss: 1.485909
Train Epoch: 2 [7040/45000 (16%)] Loss: 1.660465
Train Epoch: 2 [8448/45000 (19%)] Loss: 1.294579
Train Epoch: 2 [9856/45000 (22%)] Loss: 1.613634
Train Epoch: 2 [11264/45000 (25%)] Loss: 1.494236
Train Epoch: 2 [12672/45000 (28%)] Loss: 1.447646
Train Epoch: 2 [14080/45000 (31%)] Loss: 1.498989
Train Epoch: 2 [15488/45000 (34%)] Loss: 1.645624
Train Epoch: 2 [16896/45000 (38%)] Loss: 1.599334
Train Epoch: 2 [18304/45000 (41%)] Loss: 1.331498
Train Epoch: 2 [19712/45000 (44%)] Loss: 1.591256
Train Epoch: 2 [21120/45000 (47%)] Loss: 1.482763
Train Epoch: 2 [22528/45000 (50%)] Loss: 1.626119
Train Epoch: 2 [23936/45000 (53%)] Loss: 1.544034
Train Epoch: 2 [25344/45000 (56%)] Loss: 1.468094
Train Epoch: 2 [26752/45000 (59%)] Loss: 1.371051
Train Epoch: 2 [28160/45000 (63%)] Loss: 1.347986
Train Epoch: 2 [29568/45000 (66%)] Loss: 1.424763
Train Epoch: 2 [30976/45000 (69%)] Loss: 1.502793
Train Epoch: 2 [32384/45000 (72%)] Loss: 1.316469
Train Epoch: 2 [33792/45000 (75%)] Loss: 1.351766
Train Epoch: 2 [35200/45000 (78%)] Loss: 1.363080
Train Epoch: 2 [36608/45000 (81%)] Loss: 1.270391
Train Epoch: 2 [38016/45000 (84%)] Loss: 1.305497
Train Epoch: 2 [39424/45000 (88%)] Loss: 1.392212
Train Epoch: 2 [40832/45000 (91%)] Loss: 1.240124
Train Epoch: 2 [42240/45000 (94%)] Loss: 1.439730
Train Epoch: 2 [43648/45000 (97%)] Loss: 1.357497
    epoch          : 2
    loss           : 1.451602590693669
    accuracy       : 0.4710557725694445
    top_k_acc      : 0.8025888770517677
    val_loss       : 1.3521079152822495
    val_accuracy   : 0.5091796875
    val_top_k_acc  : 0.8333984375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch2.pth ...
Saving current best: model_best.pth ...
Train Epoch: 3 [0/45000 (0%)] Loss: 1.332903
Train Epoch: 3 [1408/45000 (3%)] Loss: 1.376957
Train Epoch: 3 [2816/45000 (6%)] Loss: 1.301742
Train Epoch: 3 [4224/45000 (9%)] Loss: 1.348263
Train Epoch: 3 [5632/45000 (13%)] Loss: 1.305164
Train Epoch: 3 [7040/45000 (16%)] Loss: 1.245183
Train Epoch: 3 [8448/45000 (19%)] Loss: 1.446791
Train Epoch: 3 [9856/45000 (22%)] Loss: 1.421561
Train Epoch: 3 [11264/45000 (25%)] Loss: 1.309925
Train Epoch: 3 [12672/45000 (28%)] Loss: 1.438207
Train Epoch: 3 [14080/45000 (31%)] Loss: 1.328497
Train Epoch: 3 [15488/45000 (34%)] Loss: 1.378746
Train Epoch: 3 [16896/45000 (38%)] Loss: 1.464753
Train Epoch: 3 [18304/45000 (41%)] Loss: 1.312119
Train Epoch: 3 [19712/45000 (44%)] Loss: 1.512890
Train Epoch: 3 [21120/45000 (47%)] Loss: 1.186520
Train Epoch: 3 [22528/45000 (50%)] Loss: 1.473404
Train Epoch: 3 [23936/45000 (53%)] Loss: 1.400741
Train Epoch: 3 [25344/45000 (56%)] Loss: 1.394369
Train Epoch: 3 [26752/45000 (59%)] Loss: 1.279838
Train Epoch: 3 [28160/45000 (63%)] Loss: 1.397544
Train Epoch: 3 [29568/45000 (66%)] Loss: 1.374417
Train Epoch: 3 [30976/45000 (69%)] Loss: 1.332261
Train Epoch: 3 [32384/45000 (72%)] Loss: 1.299266
Train Epoch: 3 [33792/45000 (75%)] Loss: 1.220224
Train Epoch: 3 [35200/45000 (78%)] Loss: 1.534112
Train Epoch: 3 [36608/45000 (81%)] Loss: 1.189254
Train Epoch: 3 [38016/45000 (84%)] Loss: 1.279112
Train Epoch: 3 [39424/45000 (88%)] Loss: 1.299281
Train Epoch: 3 [40832/45000 (91%)] Loss: 1.354111
Train Epoch: 3 [42240/45000 (94%)] Loss: 1.204156
Train Epoch: 3 [43648/45000 (97%)] Loss: 1.083748
    epoch          : 3
    loss           : 1.3260643976655873
    accuracy       : 0.5230281328914141
    top_k_acc      : 0.835757477114899
    val_loss       : 1.3033548563718795
    val_accuracy   : 0.528125
    val_top_k_acc  : 0.8384765625
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch3.pth ...
Saving current best: model_best.pth ...
Train Epoch: 4 [0/45000 (0%)] Loss: 1.401881
Train Epoch: 4 [1408/45000 (3%)] Loss: 1.211562
Train Epoch: 4 [2816/45000 (6%)] Loss: 1.421231
Train Epoch: 4 [4224/45000 (9%)] Loss: 1.296725
Train Epoch: 4 [5632/45000 (13%)] Loss: 1.401259
Train Epoch: 4 [7040/45000 (16%)] Loss: 1.240167
Train Epoch: 4 [8448/45000 (19%)] Loss: 1.114549
Train Epoch: 4 [9856/45000 (22%)] Loss: 1.244109
Train Epoch: 4 [11264/45000 (25%)] Loss: 1.413304
Train Epoch: 4 [12672/45000 (28%)] Loss: 1.238069
Train Epoch: 4 [14080/45000 (31%)] Loss: 1.096353
Train Epoch: 4 [15488/45000 (34%)] Loss: 1.267534
Train Epoch: 4 [16896/45000 (38%)] Loss: 1.267906
Train Epoch: 4 [18304/45000 (41%)] Loss: 1.086545
Train Epoch: 4 [19712/45000 (44%)] Loss: 1.292011
Train Epoch: 4 [21120/45000 (47%)] Loss: 1.258958
Train Epoch: 4 [22528/45000 (50%)] Loss: 1.248809
Train Epoch: 4 [23936/45000 (53%)] Loss: 1.209560
Train Epoch: 4 [25344/45000 (56%)] Loss: 1.212018
Train Epoch: 4 [26752/45000 (59%)] Loss: 1.285514
Train Epoch: 4 [28160/45000 (63%)] Loss: 1.302564
Train Epoch: 4 [29568/45000 (66%)] Loss: 1.115664
Train Epoch: 4 [30976/45000 (69%)] Loss: 1.185650
Train Epoch: 4 [32384/45000 (72%)] Loss: 1.303093
Train Epoch: 4 [33792/45000 (75%)] Loss: 1.173501
Train Epoch: 4 [35200/45000 (78%)] Loss: 1.071488
Train Epoch: 4 [36608/45000 (81%)] Loss: 1.204483
Train Epoch: 4 [38016/45000 (84%)] Loss: 1.487452
Train Epoch: 4 [39424/45000 (88%)] Loss: 1.201912
Train Epoch: 4 [40832/45000 (91%)] Loss: 1.192295
Train Epoch: 4 [42240/45000 (94%)] Loss: 1.334574
Train Epoch: 4 [43648/45000 (97%)] Loss: 1.413898
    epoch          : 4
    loss           : 1.246184879405932
    accuracy       : 0.5540413904671717
    top_k_acc      : 0.852423157354798
    val_loss       : 1.212526473402977
    val_accuracy   : 0.5671875
    val_top_k_acc  : 0.8599609375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch4.pth ...
Saving current best: model_best.pth ...
Train Epoch: 5 [0/45000 (0%)] Loss: 1.299487
Train Epoch: 5 [1408/45000 (3%)] Loss: 1.276864
Train Epoch: 5 [2816/45000 (6%)] Loss: 1.293610
Train Epoch: 5 [4224/45000 (9%)] Loss: 1.346122
Train Epoch: 5 [5632/45000 (13%)] Loss: 1.176972
Train Epoch: 5 [7040/45000 (16%)] Loss: 1.201090
Train Epoch: 5 [8448/45000 (19%)] Loss: 1.242998
Train Epoch: 5 [9856/45000 (22%)] Loss: 1.107223
Train Epoch: 5 [11264/45000 (25%)] Loss: 1.138589
Train Epoch: 5 [12672/45000 (28%)] Loss: 1.071263
Train Epoch: 5 [14080/45000 (31%)] Loss: 1.253372
Train Epoch: 5 [15488/45000 (34%)] Loss: 1.306817
Train Epoch: 5 [16896/45000 (38%)] Loss: 1.265868
Train Epoch: 5 [18304/45000 (41%)] Loss: 1.349330
Train Epoch: 5 [19712/45000 (44%)] Loss: 1.186573
Train Epoch: 5 [21120/45000 (47%)] Loss: 1.174410
Train Epoch: 5 [22528/45000 (50%)] Loss: 1.258422
Train Epoch: 5 [23936/45000 (53%)] Loss: 1.248507
Train Epoch: 5 [25344/45000 (56%)] Loss: 1.253944
Train Epoch: 5 [26752/45000 (59%)] Loss: 1.204013
Train Epoch: 5 [28160/45000 (63%)] Loss: 1.084910
Train Epoch: 5 [29568/45000 (66%)] Loss: 1.184014
Train Epoch: 5 [30976/45000 (69%)] Loss: 1.123324
Train Epoch: 5 [32384/45000 (72%)] Loss: 1.325830
Train Epoch: 5 [33792/45000 (75%)] Loss: 1.175442
Train Epoch: 5 [35200/45000 (78%)] Loss: 1.222975
Train Epoch: 5 [36608/45000 (81%)] Loss: 1.255943
Train Epoch: 5 [38016/45000 (84%)] Loss: 1.252389
Train Epoch: 5 [39424/45000 (88%)] Loss: 1.280669
Train Epoch: 5 [40832/45000 (91%)] Loss: 1.130400
Train Epoch: 5 [42240/45000 (94%)] Loss: 1.149628
Train Epoch: 5 [43648/45000 (97%)] Loss: 1.141840
    epoch          : 5
    loss           : 1.1801500501619144
    accuracy       : 0.5779277146464646
    top_k_acc      : 0.8649384469696969
    val_loss       : 1.148858542740345
    val_accuracy   : 0.59453125
    val_top_k_acc  : 0.8716796875
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch5.pth ...
Saving current best: model_best.pth ...
Train Epoch: 6 [0/45000 (0%)] Loss: 1.088257
Train Epoch: 6 [1408/45000 (3%)] Loss: 1.004750
Train Epoch: 6 [2816/45000 (6%)] Loss: 1.235136
Train Epoch: 6 [4224/45000 (9%)] Loss: 1.079967
Train Epoch: 6 [5632/45000 (13%)] Loss: 0.989684
Train Epoch: 6 [7040/45000 (16%)] Loss: 0.980146
Train Epoch: 6 [8448/45000 (19%)] Loss: 1.181001
Train Epoch: 6 [9856/45000 (22%)] Loss: 1.066919
Train Epoch: 6 [11264/45000 (25%)] Loss: 1.290936
Train Epoch: 6 [12672/45000 (28%)] Loss: 1.005111
Train Epoch: 6 [14080/45000 (31%)] Loss: 1.035926
Train Epoch: 6 [15488/45000 (34%)] Loss: 1.205193
Train Epoch: 6 [16896/45000 (38%)] Loss: 1.148724
Train Epoch: 6 [18304/45000 (41%)] Loss: 1.161606
Train Epoch: 6 [19712/45000 (44%)] Loss: 1.158653
Train Epoch: 6 [21120/45000 (47%)] Loss: 0.959053
Train Epoch: 6 [22528/45000 (50%)] Loss: 1.135890
Train Epoch: 6 [23936/45000 (53%)] Loss: 1.204551
Train Epoch: 6 [25344/45000 (56%)] Loss: 1.005802
Train Epoch: 6 [26752/45000 (59%)] Loss: 1.253474
Train Epoch: 6 [28160/45000 (63%)] Loss: 1.216064
Train Epoch: 6 [29568/45000 (66%)] Loss: 1.091174
Train Epoch: 6 [30976/45000 (69%)] Loss: 1.033377
Train Epoch: 6 [32384/45000 (72%)] Loss: 1.200705
Train Epoch: 6 [33792/45000 (75%)] Loss: 1.045272
Train Epoch: 6 [35200/45000 (78%)] Loss: 1.102125
Train Epoch: 6 [36608/45000 (81%)] Loss: 1.232696
Train Epoch: 6 [38016/45000 (84%)] Loss: 1.061720
Train Epoch: 6 [39424/45000 (88%)] Loss: 1.314298
Train Epoch: 6 [40832/45000 (91%)] Loss: 0.987156
Train Epoch: 6 [42240/45000 (94%)] Loss: 1.027130
Train Epoch: 6 [43648/45000 (97%)] Loss: 0.981405
    epoch          : 6
    loss           : 1.1255753971636295
    accuracy       : 0.6007215711805555
    top_k_acc      : 0.8752589370265151
    val_loss       : 1.1360471621155739
    val_accuracy   : 0.59765625
    val_top_k_acc  : 0.880078125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch6.pth ...
Saving current best: model_best.pth ...
Train Epoch: 7 [0/45000 (0%)] Loss: 1.055499
Train Epoch: 7 [1408/45000 (3%)] Loss: 1.090695
Train Epoch: 7 [2816/45000 (6%)] Loss: 1.075405
Train Epoch: 7 [4224/45000 (9%)] Loss: 1.113984
Train Epoch: 7 [5632/45000 (13%)] Loss: 1.025546
Train Epoch: 7 [7040/45000 (16%)] Loss: 1.084318
Train Epoch: 7 [8448/45000 (19%)] Loss: 0.957558
Train Epoch: 7 [9856/45000 (22%)] Loss: 1.175414
Train Epoch: 7 [11264/45000 (25%)] Loss: 0.993390
Train Epoch: 7 [12672/45000 (28%)] Loss: 1.099832
Train Epoch: 7 [14080/45000 (31%)] Loss: 1.105725
Train Epoch: 7 [15488/45000 (34%)] Loss: 1.126402
Train Epoch: 7 [16896/45000 (38%)] Loss: 1.316167
Train Epoch: 7 [18304/45000 (41%)] Loss: 0.904510
Train Epoch: 7 [19712/45000 (44%)] Loss: 0.989934
Train Epoch: 7 [21120/45000 (47%)] Loss: 0.849800
Train Epoch: 7 [22528/45000 (50%)] Loss: 1.096114
Train Epoch: 7 [23936/45000 (53%)] Loss: 1.041114
Train Epoch: 7 [25344/45000 (56%)] Loss: 1.059751
Train Epoch: 7 [26752/45000 (59%)] Loss: 1.312766
Train Epoch: 7 [28160/45000 (63%)] Loss: 1.166769
Train Epoch: 7 [29568/45000 (66%)] Loss: 1.102752
Train Epoch: 7 [30976/45000 (69%)] Loss: 1.084084
Train Epoch: 7 [32384/45000 (72%)] Loss: 1.028466
Train Epoch: 7 [33792/45000 (75%)] Loss: 0.796235
Train Epoch: 7 [35200/45000 (78%)] Loss: 1.232450
Train Epoch: 7 [36608/45000 (81%)] Loss: 1.081826
Train Epoch: 7 [38016/45000 (84%)] Loss: 1.116175
Train Epoch: 7 [39424/45000 (88%)] Loss: 1.049885
Train Epoch: 7 [40832/45000 (91%)] Loss: 1.019670
Train Epoch: 7 [42240/45000 (94%)] Loss: 0.986338
Train Epoch: 7 [43648/45000 (97%)] Loss: 1.055743
    epoch          : 7
    loss           : 1.0725817074152557
    accuracy       : 0.6205487492108586
    top_k_acc      : 0.886114563604798
    val_loss       : 1.125757098197937
    val_accuracy   : 0.601953125
    val_top_k_acc  : 0.876953125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch7.pth ...
Saving current best: model_best.pth ...
Train Epoch: 8 [0/45000 (0%)] Loss: 1.182363
Train Epoch: 8 [1408/45000 (3%)] Loss: 1.053009
Train Epoch: 8 [2816/45000 (6%)] Loss: 1.087380
Train Epoch: 8 [4224/45000 (9%)] Loss: 1.250609
Train Epoch: 8 [5632/45000 (13%)] Loss: 1.107539
Train Epoch: 8 [7040/45000 (16%)] Loss: 1.265630
Train Epoch: 8 [8448/45000 (19%)] Loss: 1.102556
Train Epoch: 8 [9856/45000 (22%)] Loss: 1.254344
Train Epoch: 8 [11264/45000 (25%)] Loss: 0.924904
Train Epoch: 8 [12672/45000 (28%)] Loss: 0.897804
Train Epoch: 8 [14080/45000 (31%)] Loss: 1.052867
Train Epoch: 8 [15488/45000 (34%)] Loss: 1.009198
Train Epoch: 8 [16896/45000 (38%)] Loss: 1.209001
Train Epoch: 8 [18304/45000 (41%)] Loss: 1.093536
Train Epoch: 8 [19712/45000 (44%)] Loss: 1.048473
Train Epoch: 8 [21120/45000 (47%)] Loss: 0.968676
Train Epoch: 8 [22528/45000 (50%)] Loss: 1.119528
Train Epoch: 8 [23936/45000 (53%)] Loss: 1.299663
Train Epoch: 8 [25344/45000 (56%)] Loss: 1.070257
Train Epoch: 8 [26752/45000 (59%)] Loss: 1.005109
Train Epoch: 8 [28160/45000 (63%)] Loss: 0.964205
Train Epoch: 8 [29568/45000 (66%)] Loss: 0.846506
Train Epoch: 8 [30976/45000 (69%)] Loss: 0.938006
Train Epoch: 8 [32384/45000 (72%)] Loss: 1.038977
Train Epoch: 8 [33792/45000 (75%)] Loss: 1.129711
Train Epoch: 8 [35200/45000 (78%)] Loss: 0.992284
Train Epoch: 8 [36608/45000 (81%)] Loss: 1.109119
Train Epoch: 8 [38016/45000 (84%)] Loss: 1.091673
Train Epoch: 8 [39424/45000 (88%)] Loss: 1.147746
Train Epoch: 8 [40832/45000 (91%)] Loss: 0.826579
Train Epoch: 8 [42240/45000 (94%)] Loss: 0.880384
Train Epoch: 8 [43648/45000 (97%)] Loss: 1.066597
    epoch          : 8
    loss           : 1.0299025618217208
    accuracy       : 0.6349900370896464
    top_k_acc      : 0.8934930358270201
    val_loss       : 1.0598522379994393
    val_accuracy   : 0.620703125
    val_top_k_acc  : 0.8947265625
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch8.pth ...
Saving current best: model_best.pth ...
Train Epoch: 9 [0/45000 (0%)] Loss: 0.901991
Train Epoch: 9 [1408/45000 (3%)] Loss: 0.812038
Train Epoch: 9 [2816/45000 (6%)] Loss: 0.713308
Train Epoch: 9 [4224/45000 (9%)] Loss: 0.967145
Train Epoch: 9 [5632/45000 (13%)] Loss: 1.053719
Train Epoch: 9 [7040/45000 (16%)] Loss: 1.236303
Train Epoch: 9 [8448/45000 (19%)] Loss: 1.049692
Train Epoch: 9 [9856/45000 (22%)] Loss: 1.068757
Train Epoch: 9 [11264/45000 (25%)] Loss: 0.938932
Train Epoch: 9 [12672/45000 (28%)] Loss: 0.791045
Train Epoch: 9 [14080/45000 (31%)] Loss: 1.027412
Train Epoch: 9 [15488/45000 (34%)] Loss: 1.011786
Train Epoch: 9 [16896/45000 (38%)] Loss: 1.080511
Train Epoch: 9 [18304/45000 (41%)] Loss: 0.953819
Train Epoch: 9 [19712/45000 (44%)] Loss: 0.955995
Train Epoch: 9 [21120/45000 (47%)] Loss: 0.828439
Train Epoch: 9 [22528/45000 (50%)] Loss: 1.005574
Train Epoch: 9 [23936/45000 (53%)] Loss: 0.874705
Train Epoch: 9 [25344/45000 (56%)] Loss: 0.978828
Train Epoch: 9 [26752/45000 (59%)] Loss: 1.121823
Train Epoch: 9 [28160/45000 (63%)] Loss: 0.981903
Train Epoch: 9 [29568/45000 (66%)] Loss: 0.901816
Train Epoch: 9 [30976/45000 (69%)] Loss: 1.005196
Train Epoch: 9 [32384/45000 (72%)] Loss: 0.818558
Train Epoch: 9 [33792/45000 (75%)] Loss: 1.000438
Train Epoch: 9 [35200/45000 (78%)] Loss: 1.074451
Train Epoch: 9 [36608/45000 (81%)] Loss: 0.841017
Train Epoch: 9 [38016/45000 (84%)] Loss: 0.930088
Train Epoch: 9 [39424/45000 (88%)] Loss: 0.789353
Train Epoch: 9 [40832/45000 (91%)] Loss: 0.829898
Train Epoch: 9 [42240/45000 (94%)] Loss: 0.941184
Train Epoch: 9 [43648/45000 (97%)] Loss: 0.847386
    epoch          : 9
    loss           : 0.990502804517746
    accuracy       : 0.6500872987689393
    top_k_acc      : 0.9014756944444445
    val_loss       : 1.0505714237689971
    val_accuracy   : 0.625390625
    val_top_k_acc  : 0.89765625
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch9.pth ...
Saving current best: model_best.pth ...
Train Epoch: 10 [0/45000 (0%)] Loss: 0.750488
Train Epoch: 10 [1408/45000 (3%)] Loss: 0.888372
Train Epoch: 10 [2816/45000 (6%)] Loss: 0.973849
Train Epoch: 10 [4224/45000 (9%)] Loss: 0.917334
Train Epoch: 10 [5632/45000 (13%)] Loss: 1.035519
Train Epoch: 10 [7040/45000 (16%)] Loss: 0.947066
Train Epoch: 10 [8448/45000 (19%)] Loss: 0.981028
Train Epoch: 10 [9856/45000 (22%)] Loss: 1.064674
Train Epoch: 10 [11264/45000 (25%)] Loss: 0.882197
Train Epoch: 10 [12672/45000 (28%)] Loss: 0.996694
Train Epoch: 10 [14080/45000 (31%)] Loss: 0.877907
Train Epoch: 10 [15488/45000 (34%)] Loss: 0.979022
Train Epoch: 10 [16896/45000 (38%)] Loss: 0.739088
Train Epoch: 10 [18304/45000 (41%)] Loss: 1.061241
Train Epoch: 10 [19712/45000 (44%)] Loss: 0.919300
Train Epoch: 10 [21120/45000 (47%)] Loss: 1.058437
Train Epoch: 10 [22528/45000 (50%)] Loss: 0.871295
Train Epoch: 10 [23936/45000 (53%)] Loss: 0.822586
Train Epoch: 10 [25344/45000 (56%)] Loss: 0.946040
Train Epoch: 10 [26752/45000 (59%)] Loss: 1.021629
Train Epoch: 10 [28160/45000 (63%)] Loss: 1.013809
Train Epoch: 10 [29568/45000 (66%)] Loss: 1.003351
Train Epoch: 10 [30976/45000 (69%)] Loss: 0.666750
Train Epoch: 10 [32384/45000 (72%)] Loss: 0.801211
Train Epoch: 10 [33792/45000 (75%)] Loss: 0.972062
Train Epoch: 10 [35200/45000 (78%)] Loss: 1.013361
Train Epoch: 10 [36608/45000 (81%)] Loss: 1.013442
Train Epoch: 10 [38016/45000 (84%)] Loss: 0.960849
Train Epoch: 10 [39424/45000 (88%)] Loss: 1.005431
Train Epoch: 10 [40832/45000 (91%)] Loss: 0.983348
Train Epoch: 10 [42240/45000 (94%)] Loss: 1.059026
Train Epoch: 10 [43648/45000 (97%)] Loss: 0.887484
    epoch          : 10
    loss           : 0.9610089971260591
    accuracy       : 0.6623781762941919
    top_k_acc      : 0.905889954229798
    val_loss       : 1.0409195333719254
    val_accuracy   : 0.62734375
    val_top_k_acc  : 0.8953125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch10.pth ...
Saving current best: model_best.pth ...
Train Epoch: 11 [0/45000 (0%)] Loss: 0.923823
Train Epoch: 11 [1408/45000 (3%)] Loss: 0.823638
Train Epoch: 11 [2816/45000 (6%)] Loss: 0.989438
Train Epoch: 11 [4224/45000 (9%)] Loss: 0.892790
Train Epoch: 11 [5632/45000 (13%)] Loss: 0.966308
Train Epoch: 11 [7040/45000 (16%)] Loss: 1.036240
Train Epoch: 11 [8448/45000 (19%)] Loss: 1.010116
Train Epoch: 11 [9856/45000 (22%)] Loss: 0.859646
Train Epoch: 11 [11264/45000 (25%)] Loss: 0.889599
Train Epoch: 11 [12672/45000 (28%)] Loss: 1.056447
Train Epoch: 11 [14080/45000 (31%)] Loss: 0.782600
Train Epoch: 11 [15488/45000 (34%)] Loss: 0.925699
Train Epoch: 11 [16896/45000 (38%)] Loss: 1.013393
Train Epoch: 11 [18304/45000 (41%)] Loss: 0.996419
Train Epoch: 11 [19712/45000 (44%)] Loss: 0.882355
Train Epoch: 11 [21120/45000 (47%)] Loss: 0.827101
Train Epoch: 11 [22528/45000 (50%)] Loss: 0.833380
Train Epoch: 11 [23936/45000 (53%)] Loss: 0.978863
Train Epoch: 11 [25344/45000 (56%)] Loss: 1.026799
Train Epoch: 11 [26752/45000 (59%)] Loss: 0.869467
Train Epoch: 11 [28160/45000 (63%)] Loss: 0.853738
Train Epoch: 11 [29568/45000 (66%)] Loss: 0.857987
Train Epoch: 11 [30976/45000 (69%)] Loss: 0.755217
Train Epoch: 11 [32384/45000 (72%)] Loss: 0.840597
Train Epoch: 11 [33792/45000 (75%)] Loss: 1.001999
Train Epoch: 11 [35200/45000 (78%)] Loss: 0.803402
Train Epoch: 11 [36608/45000 (81%)] Loss: 1.015158
Train Epoch: 11 [38016/45000 (84%)] Loss: 0.960375
Train Epoch: 11 [39424/45000 (88%)] Loss: 0.842795
Train Epoch: 11 [40832/45000 (91%)] Loss: 0.999473
Train Epoch: 11 [42240/45000 (94%)] Loss: 0.853421
Train Epoch: 11 [43648/45000 (97%)] Loss: 0.926417
    epoch          : 11
    loss           : 0.9223284745080904
    accuracy       : 0.6748441445707071
    top_k_acc      : 0.9122104837436869
    val_loss       : 1.0483410030603408
    val_accuracy   : 0.62265625
    val_top_k_acc  : 0.898828125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch11.pth ...
Train Epoch: 12 [0/45000 (0%)] Loss: 0.839948
Train Epoch: 12 [1408/45000 (3%)] Loss: 0.781067
Train Epoch: 12 [2816/45000 (6%)] Loss: 0.928835
Train Epoch: 12 [4224/45000 (9%)] Loss: 0.919579
Train Epoch: 12 [5632/45000 (13%)] Loss: 0.888303
Train Epoch: 12 [7040/45000 (16%)] Loss: 0.764395
Train Epoch: 12 [8448/45000 (19%)] Loss: 0.672037
Train Epoch: 12 [9856/45000 (22%)] Loss: 0.849778
Train Epoch: 12 [11264/45000 (25%)] Loss: 1.016248
Train Epoch: 12 [12672/45000 (28%)] Loss: 0.922121
Train Epoch: 12 [14080/45000 (31%)] Loss: 0.832104
Train Epoch: 12 [15488/45000 (34%)] Loss: 0.878688
Train Epoch: 12 [16896/45000 (38%)] Loss: 1.011967
Train Epoch: 12 [18304/45000 (41%)] Loss: 0.970751
Train Epoch: 12 [19712/45000 (44%)] Loss: 0.848732
Train Epoch: 12 [21120/45000 (47%)] Loss: 0.927162
Train Epoch: 12 [22528/45000 (50%)] Loss: 0.822467
Train Epoch: 12 [23936/45000 (53%)] Loss: 0.902060
Train Epoch: 12 [25344/45000 (56%)] Loss: 0.997272
Train Epoch: 12 [26752/45000 (59%)] Loss: 0.873540
Train Epoch: 12 [28160/45000 (63%)] Loss: 0.978591
Train Epoch: 12 [29568/45000 (66%)] Loss: 0.839814
Train Epoch: 12 [30976/45000 (69%)] Loss: 0.852791
Train Epoch: 12 [32384/45000 (72%)] Loss: 0.969526
Train Epoch: 12 [33792/45000 (75%)] Loss: 1.062154
Train Epoch: 12 [35200/45000 (78%)] Loss: 0.700287
Train Epoch: 12 [36608/45000 (81%)] Loss: 1.009396
Train Epoch: 12 [38016/45000 (84%)] Loss: 0.782711
Train Epoch: 12 [39424/45000 (88%)] Loss: 0.893738
Train Epoch: 12 [40832/45000 (91%)] Loss: 1.108445
Train Epoch: 12 [42240/45000 (94%)] Loss: 0.850492
Train Epoch: 12 [43648/45000 (97%)] Loss: 0.831905
    epoch          : 12
    loss           : 0.8954865581948649
    accuracy       : 0.6842028685290404
    top_k_acc      : 0.9181438407512627
    val_loss       : 1.020134025812149
    val_accuracy   : 0.6408203125
    val_top_k_acc  : 0.9033203125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch12.pth ...
Saving current best: model_best.pth ...
Train Epoch: 13 [0/45000 (0%)] Loss: 0.830100
Train Epoch: 13 [1408/45000 (3%)] Loss: 0.756679
Train Epoch: 13 [2816/45000 (6%)] Loss: 0.671197
Train Epoch: 13 [4224/45000 (9%)] Loss: 0.971551
Train Epoch: 13 [5632/45000 (13%)] Loss: 0.999658
Train Epoch: 13 [7040/45000 (16%)] Loss: 1.020469
Train Epoch: 13 [8448/45000 (19%)] Loss: 0.766345
Train Epoch: 13 [9856/45000 (22%)] Loss: 0.935873
Train Epoch: 13 [11264/45000 (25%)] Loss: 0.796107
Train Epoch: 13 [12672/45000 (28%)] Loss: 0.958403
Train Epoch: 13 [14080/45000 (31%)] Loss: 0.901687
Train Epoch: 13 [15488/45000 (34%)] Loss: 0.711622
Train Epoch: 13 [16896/45000 (38%)] Loss: 0.999816
Train Epoch: 13 [18304/45000 (41%)] Loss: 0.800690
Train Epoch: 13 [19712/45000 (44%)] Loss: 0.932063
Train Epoch: 13 [21120/45000 (47%)] Loss: 1.055891
Train Epoch: 13 [22528/45000 (50%)] Loss: 0.895391
Train Epoch: 13 [23936/45000 (53%)] Loss: 0.843843
Train Epoch: 13 [25344/45000 (56%)] Loss: 0.908437
Train Epoch: 13 [26752/45000 (59%)] Loss: 0.902339
Train Epoch: 13 [28160/45000 (63%)] Loss: 1.041674
Train Epoch: 13 [29568/45000 (66%)] Loss: 0.979390
Train Epoch: 13 [30976/45000 (69%)] Loss: 0.932182
Train Epoch: 13 [32384/45000 (72%)] Loss: 0.851873
Train Epoch: 13 [33792/45000 (75%)] Loss: 0.870563
Train Epoch: 13 [35200/45000 (78%)] Loss: 0.942896
Train Epoch: 13 [36608/45000 (81%)] Loss: 0.784553
Train Epoch: 13 [38016/45000 (84%)] Loss: 0.917073
Train Epoch: 13 [39424/45000 (88%)] Loss: 1.054311
Train Epoch: 13 [40832/45000 (91%)] Loss: 0.909091
Train Epoch: 13 [42240/45000 (94%)] Loss: 0.827222
Train Epoch: 13 [43648/45000 (97%)] Loss: 0.953323
    epoch          : 13
    loss           : 0.8648704830557108
    accuracy       : 0.6942372948232323
    top_k_acc      : 0.9212486190025253
    val_loss       : 1.0032038912177086
    val_accuracy   : 0.6447265625
    val_top_k_acc  : 0.90859375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch13.pth ...
Saving current best: model_best.pth ...
Train Epoch: 14 [0/45000 (0%)] Loss: 0.815349
Train Epoch: 14 [1408/45000 (3%)] Loss: 0.857148
Train Epoch: 14 [2816/45000 (6%)] Loss: 0.935290
Train Epoch: 14 [4224/45000 (9%)] Loss: 0.855447
Train Epoch: 14 [5632/45000 (13%)] Loss: 0.871716
Train Epoch: 14 [7040/45000 (16%)] Loss: 0.673660
Train Epoch: 14 [8448/45000 (19%)] Loss: 0.880691
Train Epoch: 14 [9856/45000 (22%)] Loss: 0.975428
Train Epoch: 14 [11264/45000 (25%)] Loss: 0.751729
Train Epoch: 14 [12672/45000 (28%)] Loss: 0.704253
Train Epoch: 14 [14080/45000 (31%)] Loss: 0.827796
Train Epoch: 14 [15488/45000 (34%)] Loss: 0.850868
Train Epoch: 14 [16896/45000 (38%)] Loss: 0.887877
Train Epoch: 14 [18304/45000 (41%)] Loss: 0.876028
Train Epoch: 14 [19712/45000 (44%)] Loss: 0.885767
Train Epoch: 14 [21120/45000 (47%)] Loss: 0.852582
Train Epoch: 14 [22528/45000 (50%)] Loss: 0.886271
Train Epoch: 14 [23936/45000 (53%)] Loss: 0.917076
Train Epoch: 14 [25344/45000 (56%)] Loss: 0.771606
Train Epoch: 14 [26752/45000 (59%)] Loss: 0.930203
Train Epoch: 14 [28160/45000 (63%)] Loss: 1.021493
Train Epoch: 14 [29568/45000 (66%)] Loss: 0.748006
Train Epoch: 14 [30976/45000 (69%)] Loss: 0.867918
Train Epoch: 14 [32384/45000 (72%)] Loss: 0.945301
Train Epoch: 14 [33792/45000 (75%)] Loss: 0.818264
Train Epoch: 14 [35200/45000 (78%)] Loss: 0.948010
Train Epoch: 14 [36608/45000 (81%)] Loss: 0.931059
Train Epoch: 14 [38016/45000 (84%)] Loss: 0.756923
Train Epoch: 14 [39424/45000 (88%)] Loss: 0.921435
Train Epoch: 14 [40832/45000 (91%)] Loss: 0.704261
Train Epoch: 14 [42240/45000 (94%)] Loss: 0.722232
Train Epoch: 14 [43648/45000 (97%)] Loss: 0.744458
    epoch          : 14
    loss           : 0.8458949383348227
    accuracy       : 0.7009721235795454
    top_k_acc      : 0.9247874250315656
    val_loss       : 1.0027408182621003
    val_accuracy   : 0.6515625
    val_top_k_acc  : 0.909375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch14.pth ...
Saving current best: model_best.pth ...
Train Epoch: 15 [0/45000 (0%)] Loss: 0.926461
Train Epoch: 15 [1408/45000 (3%)] Loss: 0.754750
Train Epoch: 15 [2816/45000 (6%)] Loss: 0.720997
Train Epoch: 15 [4224/45000 (9%)] Loss: 0.745690
Train Epoch: 15 [5632/45000 (13%)] Loss: 0.999385
Train Epoch: 15 [7040/45000 (16%)] Loss: 0.763259
Train Epoch: 15 [8448/45000 (19%)] Loss: 0.823969
Train Epoch: 15 [9856/45000 (22%)] Loss: 0.793698
Train Epoch: 15 [11264/45000 (25%)] Loss: 0.782717
Train Epoch: 15 [12672/45000 (28%)] Loss: 0.770053
Train Epoch: 15 [14080/45000 (31%)] Loss: 0.722096
Train Epoch: 15 [15488/45000 (34%)] Loss: 0.752565
Train Epoch: 15 [16896/45000 (38%)] Loss: 0.879226
Train Epoch: 15 [18304/45000 (41%)] Loss: 0.763195
Train Epoch: 15 [19712/45000 (44%)] Loss: 0.650786
Train Epoch: 15 [21120/45000 (47%)] Loss: 0.704479
Train Epoch: 15 [22528/45000 (50%)] Loss: 0.767152
Train Epoch: 15 [23936/45000 (53%)] Loss: 0.879763
Train Epoch: 15 [25344/45000 (56%)] Loss: 0.831756
Train Epoch: 15 [26752/45000 (59%)] Loss: 0.861238
Train Epoch: 15 [28160/45000 (63%)] Loss: 0.696502
Train Epoch: 15 [29568/45000 (66%)] Loss: 0.849680
Train Epoch: 15 [30976/45000 (69%)] Loss: 1.039806
Train Epoch: 15 [32384/45000 (72%)] Loss: 0.780041
Train Epoch: 15 [33792/45000 (75%)] Loss: 0.580751
Train Epoch: 15 [35200/45000 (78%)] Loss: 0.891863
Train Epoch: 15 [36608/45000 (81%)] Loss: 0.779546
Train Epoch: 15 [38016/45000 (84%)] Loss: 0.827319
Train Epoch: 15 [39424/45000 (88%)] Loss: 0.883568
Train Epoch: 15 [40832/45000 (91%)] Loss: 0.795430
Train Epoch: 15 [42240/45000 (94%)] Loss: 0.749381
Train Epoch: 15 [43648/45000 (97%)] Loss: 0.896578
    epoch          : 15
    loss           : 0.8149280143393711
    accuracy       : 0.7112210976957071
    top_k_acc      : 0.9290783814709596
    val_loss       : 0.9873785026371479
    val_accuracy   : 0.6560546875
    val_top_k_acc  : 0.9080078125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch15.pth ...
Saving current best: model_best.pth ...
Train Epoch: 16 [0/45000 (0%)] Loss: 0.770628
Train Epoch: 16 [1408/45000 (3%)] Loss: 0.740320
Train Epoch: 16 [2816/45000 (6%)] Loss: 0.999864
Train Epoch: 16 [4224/45000 (9%)] Loss: 0.731030
Train Epoch: 16 [5632/45000 (13%)] Loss: 0.635134
Train Epoch: 16 [7040/45000 (16%)] Loss: 0.806698
Train Epoch: 16 [8448/45000 (19%)] Loss: 0.816584
Train Epoch: 16 [9856/45000 (22%)] Loss: 0.813603
Train Epoch: 16 [11264/45000 (25%)] Loss: 0.757479
Train Epoch: 16 [12672/45000 (28%)] Loss: 0.826807
Train Epoch: 16 [14080/45000 (31%)] Loss: 0.663840
Train Epoch: 16 [15488/45000 (34%)] Loss: 0.920332
Train Epoch: 16 [16896/45000 (38%)] Loss: 0.820315
Train Epoch: 16 [18304/45000 (41%)] Loss: 0.829886
Train Epoch: 16 [19712/45000 (44%)] Loss: 0.774563
Train Epoch: 16 [21120/45000 (47%)] Loss: 0.714753
Train Epoch: 16 [22528/45000 (50%)] Loss: 1.024034
Train Epoch: 16 [23936/45000 (53%)] Loss: 0.618661
Train Epoch: 16 [25344/45000 (56%)] Loss: 0.774184
Train Epoch: 16 [26752/45000 (59%)] Loss: 0.906607
Train Epoch: 16 [28160/45000 (63%)] Loss: 0.516756
Train Epoch: 16 [29568/45000 (66%)] Loss: 0.787273
Train Epoch: 16 [30976/45000 (69%)] Loss: 0.647561
Train Epoch: 16 [32384/45000 (72%)] Loss: 0.678976
Train Epoch: 16 [33792/45000 (75%)] Loss: 0.799528
Train Epoch: 16 [35200/45000 (78%)] Loss: 0.775636
Train Epoch: 16 [36608/45000 (81%)] Loss: 0.737521
Train Epoch: 16 [38016/45000 (84%)] Loss: 0.743080
Train Epoch: 16 [39424/45000 (88%)] Loss: 0.871715
Train Epoch: 16 [40832/45000 (91%)] Loss: 0.852877
Train Epoch: 16 [42240/45000 (94%)] Loss: 0.837181
Train Epoch: 16 [43648/45000 (97%)] Loss: 0.898900
    epoch          : 16
    loss           : 0.7917350547557528
    accuracy       : 0.7215169270833334
    top_k_acc      : 0.9331523240214646
    val_loss       : 1.0241489231586456
    val_accuracy   : 0.641015625
    val_top_k_acc  : 0.9021484375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch16.pth ...
Train Epoch: 17 [0/45000 (0%)] Loss: 0.632721
Train Epoch: 17 [1408/45000 (3%)] Loss: 0.836995
Train Epoch: 17 [2816/45000 (6%)] Loss: 0.800756
Train Epoch: 17 [4224/45000 (9%)] Loss: 0.797079
Train Epoch: 17 [5632/45000 (13%)] Loss: 0.794427
Train Epoch: 17 [7040/45000 (16%)] Loss: 0.749777
Train Epoch: 17 [8448/45000 (19%)] Loss: 0.649248
Train Epoch: 17 [9856/45000 (22%)] Loss: 0.827842
Train Epoch: 17 [11264/45000 (25%)] Loss: 0.780937
Train Epoch: 17 [12672/45000 (28%)] Loss: 0.835085
Train Epoch: 17 [14080/45000 (31%)] Loss: 0.830586
Train Epoch: 17 [15488/45000 (34%)] Loss: 0.647384
Train Epoch: 17 [16896/45000 (38%)] Loss: 0.749322
Train Epoch: 17 [18304/45000 (41%)] Loss: 0.746269
Train Epoch: 17 [19712/45000 (44%)] Loss: 0.780225
Train Epoch: 17 [21120/45000 (47%)] Loss: 0.710672
Train Epoch: 17 [22528/45000 (50%)] Loss: 0.697672
Train Epoch: 17 [23936/45000 (53%)] Loss: 0.984214
Train Epoch: 17 [25344/45000 (56%)] Loss: 0.924689
Train Epoch: 17 [26752/45000 (59%)] Loss: 0.864405
Train Epoch: 17 [28160/45000 (63%)] Loss: 0.724198
Train Epoch: 17 [29568/45000 (66%)] Loss: 0.782862
Train Epoch: 17 [30976/45000 (69%)] Loss: 0.870514
Train Epoch: 17 [32384/45000 (72%)] Loss: 0.813521
Train Epoch: 17 [33792/45000 (75%)] Loss: 0.702031
Train Epoch: 17 [35200/45000 (78%)] Loss: 0.745122
Train Epoch: 17 [36608/45000 (81%)] Loss: 0.643037
Train Epoch: 17 [38016/45000 (84%)] Loss: 0.613040
Train Epoch: 17 [39424/45000 (88%)] Loss: 0.761461
Train Epoch: 17 [40832/45000 (91%)] Loss: 0.785930
Train Epoch: 17 [42240/45000 (94%)] Loss: 0.804363
Train Epoch: 17 [43648/45000 (97%)] Loss: 0.789039
    epoch          : 17
    loss           : 0.7751436091282151
    accuracy       : 0.7275859177714646
    top_k_acc      : 0.9355172821969696
    val_loss       : 1.0094095513224601
    val_accuracy   : 0.655859375
    val_top_k_acc  : 0.9048828125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch17.pth ...
Train Epoch: 18 [0/45000 (0%)] Loss: 0.912798
Train Epoch: 18 [1408/45000 (3%)] Loss: 0.646165
Train Epoch: 18 [2816/45000 (6%)] Loss: 0.610245
Train Epoch: 18 [4224/45000 (9%)] Loss: 0.712968
Train Epoch: 18 [5632/45000 (13%)] Loss: 0.919746
Train Epoch: 18 [7040/45000 (16%)] Loss: 0.738489
Train Epoch: 18 [8448/45000 (19%)] Loss: 0.964826
Train Epoch: 18 [9856/45000 (22%)] Loss: 0.665661
Train Epoch: 18 [11264/45000 (25%)] Loss: 0.820208
Train Epoch: 18 [12672/45000 (28%)] Loss: 0.774737
Train Epoch: 18 [14080/45000 (31%)] Loss: 0.606422
Train Epoch: 18 [15488/45000 (34%)] Loss: 0.634152
Train Epoch: 18 [16896/45000 (38%)] Loss: 0.665806
Train Epoch: 18 [18304/45000 (41%)] Loss: 0.703845
Train Epoch: 18 [19712/45000 (44%)] Loss: 0.724115
Train Epoch: 18 [21120/45000 (47%)] Loss: 0.858767
Train Epoch: 18 [22528/45000 (50%)] Loss: 0.914971
Train Epoch: 18 [23936/45000 (53%)] Loss: 0.899234
Train Epoch: 18 [25344/45000 (56%)] Loss: 0.896950
Train Epoch: 18 [26752/45000 (59%)] Loss: 0.598256
Train Epoch: 18 [28160/45000 (63%)] Loss: 0.797579
Train Epoch: 18 [29568/45000 (66%)] Loss: 0.724742
Train Epoch: 18 [30976/45000 (69%)] Loss: 0.711451
Train Epoch: 18 [32384/45000 (72%)] Loss: 0.794877
Train Epoch: 18 [33792/45000 (75%)] Loss: 0.700822
Train Epoch: 18 [35200/45000 (78%)] Loss: 0.730092
Train Epoch: 18 [36608/45000 (81%)] Loss: 0.764895
Train Epoch: 18 [38016/45000 (84%)] Loss: 0.758665
Train Epoch: 18 [39424/45000 (88%)] Loss: 0.876395
Train Epoch: 18 [40832/45000 (91%)] Loss: 0.751065
Train Epoch: 18 [42240/45000 (94%)] Loss: 0.751348
Train Epoch: 18 [43648/45000 (97%)] Loss: 0.693681
    epoch          : 18
    loss           : 0.7511896978725087
    accuracy       : 0.7355291193181818
    top_k_acc      : 0.9389056581439394
    val_loss       : 1.0292415291070938
    val_accuracy   : 0.648046875
    val_top_k_acc  : 0.9037109375
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch18.pth ...
Train Epoch: 19 [0/45000 (0%)] Loss: 0.667245
Train Epoch: 19 [1408/45000 (3%)] Loss: 0.633884
Train Epoch: 19 [2816/45000 (6%)] Loss: 0.668805
Train Epoch: 19 [4224/45000 (9%)] Loss: 0.638836
Train Epoch: 19 [5632/45000 (13%)] Loss: 0.627658
Train Epoch: 19 [7040/45000 (16%)] Loss: 0.765687
Train Epoch: 19 [8448/45000 (19%)] Loss: 0.648248
Train Epoch: 19 [9856/45000 (22%)] Loss: 0.786415
Train Epoch: 19 [11264/45000 (25%)] Loss: 0.601976
Train Epoch: 19 [12672/45000 (28%)] Loss: 0.657138
Train Epoch: 19 [14080/45000 (31%)] Loss: 0.897736
Train Epoch: 19 [15488/45000 (34%)] Loss: 0.696369
Train Epoch: 19 [16896/45000 (38%)] Loss: 0.813039
Train Epoch: 19 [18304/45000 (41%)] Loss: 0.716361
Train Epoch: 19 [19712/45000 (44%)] Loss: 0.786678
Train Epoch: 19 [21120/45000 (47%)] Loss: 0.764095
Train Epoch: 19 [22528/45000 (50%)] Loss: 0.636790
Train Epoch: 19 [23936/45000 (53%)] Loss: 0.742417
Train Epoch: 19 [25344/45000 (56%)] Loss: 0.659951
Train Epoch: 19 [26752/45000 (59%)] Loss: 0.724423
Train Epoch: 19 [28160/45000 (63%)] Loss: 0.869024
Train Epoch: 19 [29568/45000 (66%)] Loss: 0.774333
Train Epoch: 19 [30976/45000 (69%)] Loss: 0.886113
Train Epoch: 19 [32384/45000 (72%)] Loss: 0.792295
Train Epoch: 19 [33792/45000 (75%)] Loss: 0.665857
Train Epoch: 19 [35200/45000 (78%)] Loss: 0.838052
Train Epoch: 19 [36608/45000 (81%)] Loss: 0.761542
Train Epoch: 19 [38016/45000 (84%)] Loss: 0.717784
Train Epoch: 19 [39424/45000 (88%)] Loss: 0.855918
Train Epoch: 19 [40832/45000 (91%)] Loss: 0.712209
Train Epoch: 19 [42240/45000 (94%)] Loss: 0.653968
Train Epoch: 19 [43648/45000 (97%)] Loss: 0.673014
    epoch          : 19
    loss           : 0.7279951828108593
    accuracy       : 0.7415389244002526
    top_k_acc      : 0.9420400291982323
    val_loss       : 1.0607590571045875
    val_accuracy   : 0.648046875
    val_top_k_acc  : 0.903515625
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch19.pth ...
Train Epoch: 20 [0/45000 (0%)] Loss: 0.862416
Train Epoch: 20 [1408/45000 (3%)] Loss: 0.733674
Train Epoch: 20 [2816/45000 (6%)] Loss: 0.746174
Train Epoch: 20 [4224/45000 (9%)] Loss: 0.781364
Train Epoch: 20 [5632/45000 (13%)] Loss: 0.605555
Train Epoch: 20 [7040/45000 (16%)] Loss: 0.692463
Train Epoch: 20 [8448/45000 (19%)] Loss: 0.894890
Train Epoch: 20 [9856/45000 (22%)] Loss: 0.727676
Train Epoch: 20 [11264/45000 (25%)] Loss: 0.641883
Train Epoch: 20 [12672/45000 (28%)] Loss: 0.773392
Train Epoch: 20 [14080/45000 (31%)] Loss: 0.625053
Train Epoch: 20 [15488/45000 (34%)] Loss: 0.528645
Train Epoch: 20 [16896/45000 (38%)] Loss: 0.639930
Train Epoch: 20 [18304/45000 (41%)] Loss: 0.544965
Train Epoch: 20 [19712/45000 (44%)] Loss: 0.537010
Train Epoch: 20 [21120/45000 (47%)] Loss: 0.639223
Train Epoch: 20 [22528/45000 (50%)] Loss: 0.670522
Train Epoch: 20 [23936/45000 (53%)] Loss: 0.612434
Train Epoch: 20 [25344/45000 (56%)] Loss: 0.572076
Train Epoch: 20 [26752/45000 (59%)] Loss: 0.718023
Train Epoch: 20 [28160/45000 (63%)] Loss: 0.868828
Train Epoch: 20 [29568/45000 (66%)] Loss: 0.809501
Train Epoch: 20 [30976/45000 (69%)] Loss: 0.688319
Train Epoch: 20 [32384/45000 (72%)] Loss: 0.799011
Train Epoch: 20 [33792/45000 (75%)] Loss: 0.994791
Train Epoch: 20 [35200/45000 (78%)] Loss: 0.708890
Train Epoch: 20 [36608/45000 (81%)] Loss: 0.604731
Train Epoch: 20 [38016/45000 (84%)] Loss: 0.561342
Train Epoch: 20 [39424/45000 (88%)] Loss: 0.637584
Train Epoch: 20 [40832/45000 (91%)] Loss: 0.687308
Train Epoch: 20 [42240/45000 (94%)] Loss: 0.620674
Train Epoch: 20 [43648/45000 (97%)] Loss: 0.782604
    epoch          : 20
    loss           : 0.715023057704622
    accuracy       : 0.7463132299558081
    top_k_acc      : 0.9448612097537878
    val_loss       : 1.0520759254693985
    val_accuracy   : 0.64140625
    val_top_k_acc  : 0.8986328125
Saving checkpoint: saved/models/Cifar10_LeNet/0722_145243/checkpoint-epoch20.pth ...
Train Epoch: 21 [0/45000 (0%)] Loss: 0.592278
Train Epoch: 21 [1408/45000 (3%)] Loss: 0.522283
Train Epoch: 21 [2816/45000 (6%)] Loss: 0.685153
Train Epoch: 21 [4224/45000 (9%)] Loss: 0.617217
Train Epoch: 21 [5632/45000 (13%)] Loss: 0.614216
Train Epoch: 21 [7040/45000 (16%)] Loss: 0.702992
Train Epoch: 21 [8448/45000 (19%)] Loss: 0.648547
Train Epoch: 21 [9856/45000 (22%)] Loss: 0.726128
Train Epoch: 21 [11264/45000 (25%)] Loss: 0.706113
Train Epoch: 21 [12672/45000 (28%)] Loss: 0.671948