How to draw a camera video as background on iOS using Swift scenekit?

Question

I am trying to develop an augmented reality application using swift and scenekit on ios. Is there a way to make the video shot by the device’s camera as the background of the scene?

Answer 1

It worked for me

I used AVFoundationto capture the video input of the device’s camera:

   let captureSession = AVCaptureSession()
   let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
    previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill

    if let videoDevice = AVCaptureDevice.defaultDeviceWithMediaType(AVMediaTypeVideo) {
        var err: NSError? = nil
        if let videoIn : AVCaptureDeviceInput = AVCaptureDeviceInput.deviceInputWithDevice(videoDevice, error: &err) as? AVCaptureDeviceInput {
            if(err == nil){
                if (captureSession.canAddInput(videoIn as AVCaptureInput)){
                    captureSession.addInput(videoIn as AVCaptureDeviceInput)
                }
                else {
                    println("Failed to add video input.")
                }
            }
            else {
                println("Failed to create video input.")
            }
        }
        else {
            println("Failed to create video capture device.")
        }
    }
    captureSession.startRunning()

At this point, based on Apple's documentation of the backgroundproperty SCNScene, I expected to add an instance AVCaptureVideoPreviewLayerto SCNScene background.contents, with something like:

   previewLayer.frame = sceneView.bounds
   sceneView.scene.background.contents = previewLayer 

, . iOS?. , B. "AVCaptureVideoPreviewLayer" UIView:

 previewLayer.frame = self.view.bounds
 self.view.layer.addSublayer(previewLayer)

SCNView UIView, SCNView :

    let sceneView = SCNView()
    sceneView.frame = self.view.bounds
    sceneView.backgroundColor = UIColor.clearColor()
    self.view.addSubview(sceneView)

.

.

Answer 2

, AVCaptureVideoPreviewLayer contents ( , ).

, ( ).

var background: SCNMaterialProperty! { get }

Answer 3

Starting with iOS 11, you can now use AVCaptureDevice as material on the object or in the background of the scene (see "Using animated content" here )

Example:

 
    let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)!
    scnScene.background.contents = captureDevice