Constraints API

Understanding the VisionCamera Constraints API to enable Camera features like FPS, HDR and more

A CameraDevice can capture in different resolutions, frame-rates (FPS), dynamic ranges (HDR), output formats (JPEG vs RAW), and more. Often, such features have certain limitations - for example, while 4k and 60 FPS might be supported on the CameraDevice individually, they may not be supported together due to bandwidth limitations.

To avoid exposing a huge compatibility matrix to the user, VisionCamera internally negotiates given constraints automatically - allowing you to express intent - and VisionCamera ensures the closest compatible solution is found:

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device="back"
      outputs={[videoOutput]}
      constraints={[
        { resolutionBias: videoOutput },
        { fps: 60 }
      ]}
    />
  )
}

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  const device = useCameraDevice('back')
  const camera = useCamera({
    isActive: true,
    device: device,
    outputs: [videoOutput],
    constraints: [
      { resolutionBias: videoOutput },
      { fps: 60 }
    ]
  })

  return ...
}

const session = await VisionCamera.createCameraSession(false)
const videoOutput = createVideoOutput({
  targetResolution: CommonResolutions.UHD_16_9
})
const device = await getDefaultCameraDevice('back')

await session.configure([
  {
    input: device,
    outputs: [videoOutput],
    constraints: [
      { resolutionBias: videoOutput },
      { fps: 60 }
    ]
  }
], {})
await session.start()

Array order = Constraint Priority

The order of the array passed to constraints={...} specifies the individual Constraint's priority - a Constraint listed "higher up" (start of array) has higher priority than a Constraint listed "further down" (end of the array).

For example, if you prefer a closer frame rate match over a closer resolution match, list an { fps: ... } constraint above the { resolutionBias: ... } constraint:

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device="back"
      outputs={[videoOutput]}
      constraints={[
        { fps: 60 },
        { resolutionBias: videoOutput }
      ]}
    />
  )
}

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  const device = useCameraDevice('back')
  const camera = useCamera({
    isActive: true,
    device: device,
    outputs: [videoOutput],
    constraints: [
      { fps: 60 },
      { resolutionBias: videoOutput }
    ]
  })

  return ...
}

const session = await VisionCamera.createCameraSession(false)
const videoOutput = createVideoOutput({
  targetResolution: CommonResolutions.UHD_16_9
})
const device = await getDefaultCameraDevice('back')

await session.configure([
  {
    input: device,
    outputs: [videoOutput],
    constraints: [
      { fps: 60 },
      { resolutionBias: videoOutput }
    ]
  }
], {})
await session.start()

For example, the Constraints API may internally negotiate these given device capabilities:

- 4k    @  30FPS  <-- resolution matches, but FPS doesn't - and FPS is more important
- 1080p @  60FPS  <-- will be selected; matches 60 FPS and has best resolution match
- 720p  @  60FPS  <-- FPS matches, but we can get better resolution
- 480p  @ 120FPS  <-- FPS matches, but we can get better resolution

Constraints are safe

In contrast to the old Formats API (or other Camera libraries) constraints make VisionCamera safe, meaning a working Camera configuration is always found. For example, if you pass a { fps: 99999 } constraint, the Camera always starts - the Constraints API internally finds the closest matching FPS range, which is effectively just the highest available FPS range on the CameraDevice:

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device="back"
      outputs={[videoOutput]}
      constraints={[
        { fps: 99999 },
      ]}
    />
  )
}

function App() {
  const videoOutput = useVideoOutput({
    targetResolution: CommonResolutions.UHD_16_9
  })
  const device = useCameraDevice('back')
  const camera = useCamera({
    isActive: true,
    device: device,
    outputs: [videoOutput],
    constraints: [
      { fps: 99999 }
    ]
  })

  return ...
}

const session = await VisionCamera.createCameraSession(false)
const videoOutput = createVideoOutput({
  targetResolution: CommonResolutions.UHD_16_9
})
const device = await getDefaultCameraDevice('back')

await session.configure([
  {
    input: device,
    outputs: [videoOutput],
    constraints: [
      { fps: 99999 }
    ]
  }
], {})
await session.start()

Get enabled Constraints

The Constraints passed to constraints={...} are negotiated (based on the CameraDevice's capabilities to find a matching combination most closely matching your intent) when configuring the CameraSession.

Once a compatible Camera configuration has been found, the onSessionConfigSelected={...} callback will be called with a populated CameraSessionConfig:

function App() {
  return (
    <Camera
      style={StyleSheet.absoluteFill}
      isActive={true}
      device="back"
      onSessionConfigSelected={(config) => {
        console.log(`Config selected: ${config.toString()}`)
      }}
    />
  )
}

function App() {
  const camera = useCamera({
    isActive: true,
    device: ...,
    outputs: [...],
    constraints: [...],
    onSessionConfigSelected: (config) => {
      console.log(`Config selected: ${config.toString()}`)
    }
  })
  return ...
}

const device = ...
await session.configure([
  {
    input: device,
    outputs: [...],
    constraints: [...],
    onSessionConfigSelected: (config) => {
      console.log(`Config selected: ${config.toString()}`)
    }
  }
], {})
await session.start()

Manually resolve Constraints

To manually resolve given Constraints to a populated CameraSessionConfig without starting a Camera session, use CameraFactory.resolveConstraints(...):

const device = ...
const videoOutput = ...
const config = await resolveConstraints(
  device,
  [
    { output: videoOutput, mirrorMode: 'auto' }
  ],
  [
    { resolutionBias: videoOutput },
    { fps: 60 }
  ]
)
console.log(`Config resolved: ${config.toString()}`)

Common Examples

Photo > Video

If Photo capture is more important than Video capture, list your { resolutionBias: ... } and potentially also a { photoHDR: ... } above your video constraints:

const contraints = [
  { resolutionBias: photoOutput },
  { photoHDR: true },
  { resolutionBias: videoOutput }
] satisfies Constraint[]

Low resolution Frame output

Constraints find the closest match to your described intent, which goes both ways - to stream Frames in a low resolution, configure your CameraFrameOutput's targetResolution to a low resolution, like CommonResolutions.VGA_16_9, and use a { resolutionBias: ... } for it:

const frameOutput = useFrameOutput({
  targetResolution: CommonResolutions.VGA_16_9
})
const contraints = [
  { resolutionBias: frameOutput },
] satisfies Constraint[]

Video HDR

Video HDR has to be negotiated across output resolutions, frame rates, and device specific capabilities - you can configure the Camera to stream in a High Dynamic Range (HDR) using a { videoDynamicRange: ... }:

const contraints = [
  {
    videoDynamicRange: {
      bitDepth: 'hdr-10-bit',
      colorSpace: 'hlg-bt2020',
      colorRange: 'full'
    }
  },
] satisfies Constraint[]

Stabilization Mode

To prefer a specific Video Stabilization Mode, use a { videoStabilizationMode: ... }:

const contraints = [
  { videoStabilizationMode: 'cinematic-extended' },
] satisfies Constraint[]

Binned preference

Binned formats combine multiple neighboring sensor pixels into one larger effective pixel. This usually improves low-light sensitivity and reduces noise, but can trade away fine detail compared to a full-resolution non-binned readout. Additionally, binned formats are more performant as they use significantly less bandwidth.

To prefer a binned format over a non-binned format, use a { binned: true } constraint, and to prefer a non-binned format over a binned format use a { binned: false } constraint. As with other formats, not specifying the constraint means "no preference", so binned and non-binned formats are ranked equally.

const contraints = [
  { binned: true },
] satisfies Constraint[]

On this page