Since Albumentation on mmDetection was appliable with provided 'albu_example' in config folder, thought it must be appliable on mmSegmentation as well.
However, mmSegmentation officialy mentioned that they are not into developing albumentation on mmseg. Luckily I found a github branch of mmseg about appling Albumantation (https://github.com/vitrox-technologies/mmsegmentation) and went straigt into it.
Try1. Checkout branch
I simply checkout the branch and tried to run train.py with custom dataset inserting 'Albu' in the pipeline but failed. By looking logs saying, " 'Albu' not registered in custom dataset" and searching inside of mmseg/datasets/piplines, I noticed that, even though I checkout the albumentation git branch, what was installed in my server has not chainged. So I additionally installed the albumentation enabling files with 'pip install setup.py -e .' .
Try2. Setting with setup.py
After setting the branch, tried the run again and failed. The branch has not been updated recent models, so the model SegFormer, what I was tring to use, was not developed. So I duplicated mmseg/models and mmseg/core to update recent models.
Try3. Duplicating recent models
Failed againg with
This error is solved after adding flip augmentation in pipeline. Why couldn't I run without a flip? Will figure out later.
Try4. adding flip augmentation
Failed again with AssertionError: Default process group is not initialized..
By checking the error log, figure out that the Error started with batchnormalization. Solved by switching norm_cfg from SyncBN to BN.
Try5. Switching norm_cfg from SyncBN to BN
Finally works!
..
No it was not the end.
At evaluation, not expecting arg, 'pre_eval' issue came up. I guess this is muti-gpu kind of problem,, solved it with eleminating the argument.
Try6.
ERROR again.OSError: [Errno 12] Cannot allocate memory came up.
Since there is 21G memory left, tried to reduce worker per gpu to 1 but failed.
Thought it might be the augmentation 'CropNonEmptyMaskIfExists' why it requires such memory, changed to simple 'crop' augmentation but failed.
Furthemore, made albumentation pipeline empty, to find out whether albu-branch it self is the problem.
Checking memory usage during evaluation, it was abundant. Can't figure out what is the problem.
Try7.
Rather adding the recent model to the albumentation branch, tried to add albumentation transform to the recent mmsegmentation. Copied mmseg/datasets/pipelines/transforms.py Albu register module part, pasted. And it's working. Hallellujah.
Further Questions
About AssertionError: Default process group is not initialized. why BN solved it.
