iOS 필터 효과 있는 카메라 앱 만들기

shintwl·2024년 2월 22일
AVFoundationCameraMetalfilteriOS

카메라 프리뷰 이미지에 필터를 적용하는 앱입니다

애플 공식 문서에서 제공하는 가이드와 예제 코드 중 최소한으로 필요한 것만 골라 구성했습니다

3줄 요약

  1. AVCaptureSession을 통해 SampleBuffer를 받아온다
  2. SampleBuffer에 CIFilter를 적용한다
  3. CIFilter를 적용한 이미지로 MTKView에 그린다

UI 구성

카메라를 통해 들어오는 프레임 이미지가 화면 전체를 채우고, 필터 토글 버튼이 좌측 하단에 위치합니다

MetalKit에서 제공하는 MTKView를 사용합니다
카메라 구현시 주로 사용하는 AVCaptureVideoPreviewLayer는 AVCaptureSession으로부터 프레임을 바로 받기 때문에 여기에 효과를 적용할 수 없다고 합니다 (애플)

private var mtkView: MTKView = MTKView()

private var filterChangeButton: UIButton = {
	let button = UIButton(type: .system)
    button.setImage(UIImage(systemName: "camera.filters"), for: .normal)
    button.backgroundColor = .white
    button.layer.cornerRadius = 20
    return button
}()

private func configureUI() {
	self.view.addSubview(self.mtkView)
    self.view.addSubview(self.filterChangeButton)

    self.filterChangeButton.addTarget(self, action: #selector(filterChangeButtonTapped), for: .touchUpInside)
}

private func configureAutoLayout() {
	self.mtkView.translatesAutoresizingMaskIntoConstraints = false
    self.filterChangeButton.translatesAutoresizingMaskIntoConstraints = false

    NSLayoutConstraint.activate([
    	self.mtkView.topAnchor.constraint(equalTo: self.view.topAnchor),
        self.mtkView.leadingAnchor.constraint(equalTo: self.view.leadingAnchor),
        self.mtkView.trailingAnchor.constraint(equalTo: self.view.trailingAnchor),
        self.mtkView.bottomAnchor.constraint(equalTo: self.view.bottomAnchor),

        self.filterChangeButton.leadingAnchor.constraint(equalTo: self.view.leadingAnchor, constant: 15),
        self.filterChangeButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor, constant: -15),
         self.filterChangeButton.widthAnchor.constraint(equalToConstant: 40),
         self.filterChangeButton.heightAnchor.constraint(equalToConstant: 40)
	])
}

Metal 설정

MTKView

private var mtkView: MTKView = MTKView()

self.mtkView.isPaused = true
self.mtkView.enableSetNeedsDisplay = false

MTLDevice 설정

디바이스의 그래픽 하드웨어와 연결한 인터페이스를 선정합니다

self.metalDevice = MTLCreateSystemDefaultDevice()

self.mtkView.device = self.metalDevice

MTLCommandQueue 설정

그래픽 작업을 처리할 전용 queue입니다

self.metalCommandQueue = metalDevice.makeCommandQueue()

AVFoundation 설정

queue 생성

카메라 제어 및 sampleBuffer를 제어할 serial queue를 생성합니다

private let cameraQueue = DispatchQueue(label: "cameraQueue")
private let videoQueue = DispatchQueue(label: "videoQueue")

AVCaptureSession 객체 생성

private let session = AVCaptureSession()

AVCaptureDeviceInput 설정

카메라 하드웨어와 연결하는 과정입니다

카메라 탐색

설정한 조건에 맞는 카메라를 OS에서 반환해줍니다

private func configureCamera() -> AVCaptureDevice {
	let discoverySession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInWideAngleCamera,.builtInUltraWideCamera], mediaType: .video, position: .back)

    guard let cameraDevice = discoverySession.devices.first else {
    	fatalError("no camera device is available")
	}

    return cameraDevice
}

input 연결

AVCaptureSession와 카메라 하드웨어를 연결합니다

private var deviceInput: AVCaptureDeviceInput!

let cameraDevice: AVCaptureDevice = configureCamera()

self.deviceInput = try AVCaptureDeviceInput(device: cameraDevice)
self.session.addInput(self.deviceInput)

AVCaptureVideoDataOutput 설정

카메라 하드웨어로부터 받아온 프레임 및 메타정보를 사용할 곳을 기존에 생성한 AVCaptureSession으로 지정합니다

private var videoOutput: AVCaptureVideoDataOutput!

self.videoOutput = AVCaptureVideoDataOutput()
self.videoOutput.connections.first?.videoOrientation = .portrait

self.session.addOutput(self.videoOutput)

CIFilter 적용

CIContext 설정

그래픽 작업을 할 컨텍스트입니다

private var ciContext: CIContext!

self.ciContext = CIContext(mtlDevice: self.metalDevice)

AVCaptureVideoDataOutputSampleBufferDelegate 구현

self.videoOutput.setSampleBufferDelegate(self, queue: self.videoQueue)

받아온 SampleBuffer를 CIImage로 변환하여 CIFilter를 적용합니다
이후 MTKView의 draw 메서드를 호출하여 MTKView가 해당 CIImage를 렌더링하게 합니다

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
	guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    	return
	}

    let ciImage = CIImage(cvImageBuffer: cvBuffer)

    if self.filterApplied {
    	guard let filteredImage = applyFilter(inputImage: ciImage) else {
        	return
		}

        self.currentCIImage = filteredImage

	} else {
    	self.currentCIImage = ciImage
	}

    self.mtkView.draw()
}

func applyFilter(inputImage image: CIImage) -> CIImage? {
	var filteredImage: CIImage?

    self.sepiaFilter.setValue(image, forKey: kCIInputImageKey)
    filteredImage = self.sepiaFilter.outputImage

    return filteredImage
}

MTKViewDelegate 구현

self.mtkView.delegate = self
func draw(in view: MTKView) {
	guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
    	return
	}

    guard let ciImage = self.currentCIImage else {
    	return
	}

    guard let currentDrawable = view.currentDrawable else {
    	return
	}

    self.ciContext.render(ciImage,
                          to: currentDrawable.texture,
                          commandBuffer: commandBuffer,
                          bounds: CGRect(origin: .zero, size: view.drawableSize),
                          colorSpace: CGColorSpaceCreateDeviceRGB())

	commandBuffer.present(currentDrawable)
    commandBuffer.commit()
}

완성본

이미지

import UIKit
import MetalKit
import AVFoundation
import CoreImage

class ViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        configureUI()
        configureAutoLayout()

        configureMetal()
        configureCoreImage()

        configureCaptureSession()
    }

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        cameraQueue.async {
            self.session.startRunning()
        }
    }

    override func viewDidDisappear(_ animated: Bool) {
        super.viewDidDisappear(animated)

        cameraQueue.async {
            if self.session.isRunning {
                self.session.stopRunning()
            }
        }
    }

    // MARK: - UI
    private var mtkView: MTKView = MTKView()

    private var filterChangeButton: UIButton = {
        let button = UIButton(type: .system)
        button.setImage(UIImage(systemName: "camera.filters"), for: .normal)
        button.backgroundColor = .white
        button.layer.cornerRadius = 20
        return button
    }()

    private func configureUI() {
        self.view.addSubview(self.mtkView)
        self.view.addSubview(self.filterChangeButton)

        self.filterChangeButton.addTarget(self, action: #selector(filterChangeButtonTapped), for: .touchUpInside)
    }

    private func configureAutoLayout() {
        self.mtkView.translatesAutoresizingMaskIntoConstraints = false
        self.filterChangeButton.translatesAutoresizingMaskIntoConstraints = false

        NSLayoutConstraint.activate([
            self.mtkView.topAnchor.constraint(equalTo: self.view.topAnchor),
            self.mtkView.leadingAnchor.constraint(equalTo: self.view.leadingAnchor),
            self.mtkView.trailingAnchor.constraint(equalTo: self.view.trailingAnchor),
            self.mtkView.bottomAnchor.constraint(equalTo: self.view.bottomAnchor),

            self.filterChangeButton.leadingAnchor.constraint(equalTo: self.view.leadingAnchor, constant: 15),
            self.filterChangeButton.bottomAnchor.constraint(equalTo: self.view.bottomAnchor, constant: -15),
            self.filterChangeButton.widthAnchor.constraint(equalToConstant: 40),
            self.filterChangeButton.heightAnchor.constraint(equalToConstant: 40)
        ])
    }

    // MARK: - configure Metal

    private var metalDevice: MTLDevice!
    private var metalCommandQueue: MTLCommandQueue!

    private func configureMetal() {
        self.metalDevice = MTLCreateSystemDefaultDevice()

        self.mtkView.device = self.metalDevice

        self.mtkView.isPaused = true
        self.mtkView.enableSetNeedsDisplay = false

        self.metalCommandQueue = metalDevice.makeCommandQueue()

        self.mtkView.delegate = self

        self.mtkView.framebufferOnly = false
    }

    // MARK: - camera control
    private let cameraQueue = DispatchQueue(label: "cameraQueue")
    private let videoQueue = DispatchQueue(label: "videoQueue")

    private let session = AVCaptureSession()

    private var deviceInput: AVCaptureDeviceInput!
    private var videoOutput: AVCaptureVideoDataOutput!

    private func configureCaptureSession() {
        let cameraDevice: AVCaptureDevice = configureCamera()
        do {
            self.deviceInput = try AVCaptureDeviceInput(device: cameraDevice)

            self.videoOutput = AVCaptureVideoDataOutput()
            self.videoOutput.setSampleBufferDelegate(self, queue: self.videoQueue)

            self.session.addInput(self.deviceInput)
            self.session.addOutput(self.videoOutput)

            self.videoOutput.connections.first?.videoOrientation = .portrait
        } catch {
            print("error = \(error.localizedDescription)")
        }
    }

    private func configureCamera() -> AVCaptureDevice {
        let discoverySession = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInWideAngleCamera,.builtInUltraWideCamera], mediaType: .video, position: .back)

        guard let cameraDevice = discoverySession.devices.first else {
            fatalError("no camera device is available")
        }

        return cameraDevice
    }

    // MARK: - configure core image
    private var ciContext: CIContext!
    private var currentCIImage: CIImage?

    private func configureCoreImage() {
        self.ciContext = CIContext(mtlDevice: self.metalDevice)
    }

    // MARK: - private methods
    private var filterApplied: Bool = false

    private let sepiaFilter:CIFilter = {
        let filter = CIFilter(name: "CISepiaTone")!
        filter.setValue(NSNumber(value: 1), forKeyPath: "inputIntensity")
        return filter
    }()

    @objc private func filterChangeButtonTapped(_ button:UIButton) {
        self.filterApplied.toggle()
    }

}

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
	func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
		guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
			return
		}

        let ciImage = CIImage(cvImageBuffer: cvBuffer)

        if self.filterApplied {
            guard let filteredImage = applyFilter(inputImage: ciImage) else {
                return
            }

            self.currentCIImage = filteredImage

        } else {
            self.currentCIImage = ciImage
        }

        self.mtkView.draw()
    }

    func applyFilter(inputImage image: CIImage) -> CIImage? {
        var filteredImage: CIImage?

        self.sepiaFilter.setValue(image, forKey: kCIInputImageKey)
        filteredImage = self.sepiaFilter.outputImage

        return filteredImage
    }

}

extension ViewController: MTKViewDelegate {
	func mtkView(\_ view: MTKView, drawableSizeWillChange size: CGSize) {
		// do nothing
	}

    func draw(in view: MTKView) {
        guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
            return
        }

        guard let ciImage = self.currentCIImage else {
            return
        }

        guard let currentDrawable = view.currentDrawable else {
            return
        }

        self.ciContext.render(ciImage,
                              to: currentDrawable.texture,
                              commandBuffer: commandBuffer,
                              bounds: CGRect(origin: .zero, size: view.drawableSize),
                              colorSpace: CGColorSpaceCreateDeviceRGB())

        commandBuffer.present(currentDrawable)
        commandBuffer.commit()
    }

}

참고

https://developer.apple.com/documentation/avfoundation/additional_data_capture/avcamfilter_applying_filters_to_a_capture_stream

profile
shintwl
주 iOS, 부 Android
이전 포스트

Hugging vs Compression Priority

다음 포스트

UICollectionView.register 졸업하기

3개의 댓글

comment-user-thumbnail
Youth
2024년 2월 23일

GCD관련해서 공부를할때 "serial queue는 순서가 중요한 상황에서 사용한다"라고 배웠는데 대체 어떨때 순서가 중요한거지? 동시큐로 사용하는게 나은상황밖에떠오르지않았던거같아요 그런데 글을보면서 video같은 frame단위의 작업은 순서가 중요하기에 이럴때는 직렬큐를 사용해야하는구나라는걸 알게되었습니다! 좋은 레퍼런스네요

한가지 궁금한게있습니다
카메라관련큐와 비디오관련큐를 따로 생성해서 task를 분배하는 이유가 있을까요

1개의 답글
comment-user-thumbnail
Horang
2024년 2월 24일

그래픽스의 관심을 갖으면서 Metal에 대해 알게 되었는데
글로 맛보기만 했는데도 아직 제가 접근하기에는 다소 어렵게 느껴지네요 😭
좋은 글 감사합니다 !

답글 달기