Native Frame Processor Plugins

Native Frame Processor Plugins are essentially native Swift/Kotlin functions that can be called from a JS Frame Processor (see "The Frame Output").

const myNativePlugin = ...
const frameOutput = useFrameOutput({
  onFrame(frame) {
    'worklet'
    myNativePlugin.call(frame)
    frame.dispose()
  }
})

Implementation

Native Frame Processor Plugins are Nitro Modules.

1. Bootstrap a Nitro Module

To get started, follow the "Nitro: How to build a Nitro Module" guide, and create a Swift/Kotlin Hybrid Object.

2. Add a dependency on react-native-vision-camera

Then, in your Nitro Module, add a dependency on react-native-vision-camera in your package.json:

{
  // ...
  "devDependencies": {
    // ...
    "react-native-vision-camera": "*",
  },
  "peerDependencies": {
    // ...
    "react-native-vision-camera": "*"
  },
}

..as well as to your native iOS/Android dependencies:

Pod::Spec.new do |s|
  # ...
  s.dependency 'VisionCamera'
end

// ...
dependencies {
  // ...
  implementation project(":react-native-vision-camera")

// ...
find_package(react-native-vision-camera REQUIRED)

# Link all libraries together
target_link_libraries(
        ${PACKAGE_NAME}
        // ...
        react-native-vision-camera::VisionCamera
)

3. Using the Frame in the specs

Now, define a Nitro HybridObject that takes a Frame from react-native-vision-camera:

import type { HybridObject } from 'react-native-nitro-modules'
import type { Frame } from 'react-native-vision-camera'

export interface MyNativePlugin extends HybridObject<{
  ios: 'swift',
  android: 'kotlin'
}> {
  call(frame: Frame): void
}

4. Implement the native plugin

Then, generate your Nitro specs via nitrogen. In your native iOS/Android project, you can now implement MyNativePlugin and its call(...) function:

import VisionCamera
import NitroModules

class HybridMyNativePlugin: HybridMyNativePluginSpec {
  func call(frame: any HybridFrameSpec) {
    // TODO: Implementation
  }
}

package com.margelo.nitro.mynativeplugin
import com.margelo.nitro.camera.HybridFrameSpec

class HybridMyNativePlugin: HybridMyNativePluginSpec() {
  fun call(frame: HybridFrameSpec) {
    // TODO: Implementation
  }
}

To unwrap the native CMSampleBuffer/Image from the HybridFrameSpec, cast it to NativeFrame - a public type exposed by VisionCamera:

import VisionCamera
import AVFoundation
import NitroModules

class HybridMyNativePlugin: HybridMyNativePluginSpec {
  func call(frame: any HybridFrameSpec) {
    guard let frame = frame as? NativeFrame else { return }
    let buffer = frame.sampleBuffer // CMSampleBuffer
  }
}

package com.margelo.nitro.mynativeplugin
import com.margelo.nitro.camera.HybridFrameSpec
import com.margelo.nitro.camera.public.NativeFrame

class HybridMyNativePlugin: HybridMyNativePluginSpec() {
  fun call(frame: HybridFrameSpec) {
    val frame = frame as? NativeFrame ?: return
    val image = frame.image // ImageProxy
  }
}

5. Call it from JS

In JS, you can now create an instance of your Nitro HybridObject and use it in a Frame Processor:

import { NitroModules } from 'react-native-nitro-modules'

const plugin = NitroModules.createHybridObject<MyNativePlugin>('MyNativePlugin')
const frameOutput = useFrameOutput({
  onFrame(frame) {
    'worklet'
    plugin.call(frame)
    frame.dispose()
  }
})

Thanks to Nitro's powerful architecture, your Native Frame Processor Plugin can accept any kinds of arguments, return any kind of values (even a Frame!), and even be asynchronous.

C++ Plugins

Because Nitro supports C++, you can also implement your Frame Processor Plugin in pure C++.

#include <VisionCamera/HybridFrameSpec.hpp>

namespace margelo::nitro::myplugin {
  class HybridMyNativePlugin: public HybridMyNativePluginSpec {
  public:
    HybridMyNativePlugin(): HybridObject(TAG) {}
    void call(const std::shared_ptr<HybridFrameSpec>& frame) {
      // TODO: Implementation
    }
  };
}

Since the Frame is a platform type, you cannot reliably downcast it cross-platform directly. Instead, you can use the public JS APIs (like Frame.getNativeBuffer()) from C++, and downcast per platform if needed:

void call(const std::shared_ptr<HybridFrameSpec>& frame) {
  auto nativeBuffer = frame->getNativeBuffer();
  auto orientation = frame->getOrientation();
#ifdef ANDROID
  AHardwareBuffer* hardwareBuffer = reinterpret_cast<AHardwareBuffer*>(nativeBuffer.pointer);
#else // iOS
  CVPixelBufferRef pixelBuffer = reinterpret_cast<CVPixelBufferRef>(nativeBuffer.pointer);
#endif
}

Alternatively, you can add platform-specific C++/Objective-C++ code to your existing Swift/Kotlin codebases, and use the respective CVPixelBuffer or AHardwareBuffer* APIs.

Depth

While this guide covered the Frame type, the same principle applies to Depth as well. Simply use the Depth type instead of Frame, and cast to NativeDepth.