🔎 Stanford CS231n 강의를 듣고 기록용도로 정리하는 게시글입니다.

Semantic Segmentation

• input an image and output a decision of a category for every pixel in that image

• only labeling pixels independently for what the category of that pixel is
  • in the cow picture - doesn't distinguish between those 2 cows

Idea: Sliding Window

• break up the input image into many many small, tiny local crops
  • take each of those crops and treat them as a classification problem
  • ex) for this crop, what is the category of the central pixel of the crop?
• might work to some extent, but not a good idea
• problems
  • very inefficient - need a separate crop for every pixel in that image to label every pixel in the image and this would be very expensive to run forward and backward
  • patches that are next to each other and they overlap → conv features of those patches end up going through the same conv layer → can share
• nobody does this, don't do this

Idea: Fully Convolutional

• make the network to be a whole stack of Conv layers, without Fully Connected layers
  • final conv layer output a tensor of CxHxW (C is the number of categories) → giving classification scores for every pixel in the input image
  • train by putting a classification loss at every pixel of this output, take an average over those pixels in space
• since we're keeping the Convs that are keeping the spatial size of the input image → very expensive
  • take a ton of memory
• in practice, this architecture doesn't get used

• have downsampling and upsampling of the feature map inside the image
  • rather than doing Conv of the full spatial resolution of the image, maybe do the small number of conv layers at the original resolution, and downsample the feature map using e.g. max pooling, strided convolutions
  • then, rather than transitioning to a FC layer, increase the spatial resolution of the predictions in the 2nd half of the network → output image is the same size as the input image
• computationally efficient - can make network deep, work at a lower spatial resolution

In-Network UpSampling

Unpooling

• Nearest Neighbor unpooling
  • duplicate the element for every point in the 2x2 receptive field of the unpooling region
• Bed of Nails unpooling
  • make all 0s except for 1 element of the unpooling region

Max Unpooling

• a lot of networks tend to be symmetrical; have a downsampling portion of the network and upsampling portion of the network with a symmetry between those 2 portions
• for each upsampling layer, it's associated with one of the pooling layers in the first half of the network
• in the downsampling, when we do max pooling, we'll remember which element of the receptive field was used to do the max pooling
  • do something like bed of nails, but instead of putting the element in the same position, put it in the position that was used in the corresponding max pooling stop earlir in the network (upsampling 할 때는 max pooling 때 그 숫자가 있던 자리에 그대로 다시 넣는다)
  • fill in the rest with 0

Q: Why is this a good idea?
A: When we do semantic segmentation, we want it to be pixel perfect. We want to get sharp boundaries. When doing max pooling, you sort of lose the spatial information b/c you don't know where that feature vector came from in the local receptive field after max pooling. Putting the vector in the same slot when unpooling → help us find little details better

Upsampling: Transpose Convolution

• unpooling methods above are fixed function; they're not learning how to do the upsampling

  • ex) strided convolution; it's a learnable layer that learns the way that the network wants to perform downsampling at that layer
  • Normal 3x3 convolution, stride 1, pad 1
    
    
  • Normal 3x3 convolution, stride 2, pad 1
    
    
    • stride of 2 gives us the ratio between how much we move in the input and how much we move in the output
    • with stride 2, end up downsampling the feature map by a factor of 2 in a learnable way

• transpose convolution lets you do learnable upsampling

  • upsample the feature map and learn some weights about how it wants to do that upsampling

• Transpose 3x3 convolution, stride 2 pad 1

  • input is 2x2, output is 4x4
    
  • take the value of the input feature map at the upper left corner → mutliply the filter by that value → copy those values over to this 3x3 region in the output
  • rather than taking an inner product with the filter and the input, input gives weights that we'll use to weight the filter, and the output will be the copies of the filter that are weighted by the values in the input
  • 여기서 filter라는 게 뭐지?????
    
  • now, plop the filter down 2 pixels away in the output
  • blue pixel in the input is a scalar value → multiply it by the values in the filter → copy those weighted filter values into the new region in the output
  • receptive field of the output might overlap → sum the results in the output

Convolution as Matrix Multiplication (1D Ex.)

• Where does that name "transpose" come from..?

• You can always write convolution as matrix multiplication
  • take the convolution kernel $x$ and turn it into a matrix $X$, which contains copies of that convolutional kernel that are offset by different regions
  • now we can take this giant weight matrix $X$ and do a matrix vector multiplication between $X$ and input $a$
  • this produces the same result as convolution

• multiply the same weight matrix by the transpose of it

• in case of stride 1, stride 1 transpose convolution ends up being a stride 1 normal convolution ????????

• in case of stride 2,
  
  - take a stride 2 convolution and write out this stride 2 convolution as a matrix multiplication
  

  • now, corresponding transpose convolution is no longer convolution
  • this transposed matrix for stride 2 convolution is something fundamentally different from the original normal convolution operation 어차피 행렬 곱 하는 건 같은데 뭐가 다른건가?

As a whole

• a giant convolutional network
• downsampling and upsampling inside the network
• train it end to end with backprop using this cross entropy loss over every pixel

Classification + Localization

• in addition to predicting what the category is, know where the object in the image is (draw a bounding box around the region of that object)
  • difference between this and object detection - in localization, you assume that there's exactly 1 object in the image that you're looking for

• have an input image
• feed the input image into a giant conv network, which gives us final vector summarizing the content of the image
• have FC layer that goes from that vector to the class scores
• have ANOTHER FC layer that goes from that vector to 4 numbers; height, width, x, and y positions of the bounding box
• produces 2 different outputs; set of class scores and 4 numbers giving coordinates of the bounding box in the input image
• during training, we have 2 losses
  • Softmax loss - using the ground truth category label and the predicted class scores
  • L2 loss - measuring dissimilarity between predicted coordinates for the bounding box and the actual coordinates for the bounding box

Aside: Human Pose Estimation

• input: image of a person

• output: the position of the joints for that person

• allows the network to predict the pose of the human
  

• one parameterization: define a person's pose by 14 joint positions

  • when training, output 14 numbers, giving the $x$ and $y$ coordinates for each of those 14 joints
  • apply regression loss(L2, L1 loss..) on each of those 14 different predicted joints
  • then do backpropagation

• if output is

  • categorical, cross entropy loss, Softmax loss, SVM margin type losses
  • continuous, L2, L1 loss

Object detection

• start with some fixed set of categories (ex.cat, dog, fish)
• given input image, when one of those categories appears in the image, draw a box around it and predict the category of that box

• progression of object detection on a dataset called PASCAL VOC

• this is different from localization, b/c there might be differing numbers of objects in the image
• tricky to think of detection as a regression problem

Idea: Sliding window

• object detection as classification
• take different crops from the input image, feed them through conv network, then convnet does a classification decision
  
  
  
• problem: how do you choose the crops?
  • since the object can appear at any size, location, aspect ratio in the image,
  • would end up having to test many many different crops
  • this would be computationally intractable
• in practice, people don't do this brute force sliding window approach

Idea: Region Proposals

• not using deep learning, slightly more traditional computer vision technique
  • uses more traditional image processing type things
• given an input image, region proposal network will output thousand boxes where an object might be present
  • look for bloby regions and give some set of candidate proposal regions
• now, rather than applying classification network to every possible location,
  • apply region proposal networks to get some set of proposal regions where objects are likely located
  • and apply convolutional network for classification to each of those proposal regions

R-CNN

• given input image, run region proposal network and get regions of interest

  • however, these regions might have different sizes
  • but when running through a conv network for classification, input image size should be the same (b/c of FC layers..)

• warp(휘게 만들다) each proposal to a fixed square size

• run each of them through convnet

  • use SVM to predict categories for each of those crops

• in addtion, do a multi-task loss? (do a regression for bounding box)

• Problems

  • Ad hoc(즉석) training objectives
    • Fine-tune network with softmax classifier (log loss)
    • Train post-hoc linear SVMs (hinge loss)
    • Train post-hoc bounding-box regressions (least squares)
  • Training is slow (84h), takes a lot of disk space
  • Inference (detection) is slow
    • 47s/image with VGG16 [Simonyan & Zisserman. ICLR15]
    • Fixed by SPP-net [He et al. ECCV14]

Fast R-CNN

• run the entire image through convolutional layers all at once
  • to give high resolution convolutional feature map
• use region proposal
  • but rather than taking crops directly from the image,
  • project those region proposals onto the conv feature map, then take crops from the conv feature map
  • this allows us to reuse a lot of conv computation when we have a lot of crops per image
• ROI pooling layer - reshape the crops from the convolutional feature map
  • b/c if we have FC layers downstream, they expect fixed-size input
• FC layer - predict classification scores and linear regression offsets to the bounding boxes

Training

• when training, we have a multi-task loss that trades off between these 2 constraints
• during backprop, backprop this entire thing and learn jointly

RoI Pooling

• looks like max pooling
  • no more detail

Speed Comparison

• at training time, Fast R-CNN is 10x faster
  • b/c it shares all the computation between different feature maps
• at test time, Fast R-CNN is fast
  • too fast that its computation time is dominated by computing region proposals
  • computing 2000 regions proposals takes 2 seconds
    • b/c we're processing them in a shared way (by sharing expensive convolutions), we can process all of these region proposals in less than a 1s altogether
  • Fast R-CNN ends up being bottlenecked by computing region proposals
    ⇒ solved this problem w/ Faster R-CNN

Faster R-CNN

• make the network to predict its own region proposals
  • run entire input image through convolutional layers → get convolutional feature map representing the high resolution image
  • separate region proposal network that works on top of convolutional features → predict its own region proposals
  • take crops from region proposals from convolutional features, pass them up to the rest of the network
• now need to multi-task training networks to do 4 things at once

• b/c we eliminated computing region proposals outside the network, faster R-CNN ends up begin more fast than other methods

Detection without Proposals: YOLO/SSD

• YOLO: You Only Look Once
• SSD: Single Shot Detection
• feed forward in a single pass
  • rather than doing independent processing for each of these potential regions,
  • treat this like a regression problem and make all these predictions all at once with big convolutional network
• given an input image, divide that image into coarse(굵은) grid
• within each of grid cells, imagine some set of base bounding boxes
• now, for each of these grid cells and base bounding boxes, predict several things
  • 1) predict an offset off the base bounding box
    • what is the true location of the object off this base bounding box?
  • 2) predict classification scores
    • classification score for each of the base bounding boxes
• finally, predict a giant tensor of 7x7 grid by 5B+C
  • $B$ base bounding boxes
  • $C$ classification scores for $C$ categories

Aside: Dense Captioning

• Object Detection + Captioning = Dense Captioning
• rather than predicting a fixed category label for each region, write a caption for each region

Instance Segmentation

• given an input image, predict the locations and identities of objects in that image, but rather than bounding box, predict a whole segmentation mask for each objects

Mask R-CNN

• multi-stage processing approach

  • take whole input image into CNN and region proposal network (same as faster R-CNN)
  • project region proposals onto convolutional feature map
  • do RoI aligning to warp features corresponding to the region proposal into the right shape
  • now, 2 branches
    • predict classification scores and bounding box coordinates that regressed off the region proposal coordinates
    • classify for each pixel in that input region proposal whether or not it's an object

• very good results
  

• also does pose
  

  • for each of the region proposals, add additional branch that predicts the coordinates of the joints for the instance of the current region proposals