Custom sources

important

If you only wish to send camera, microphone or screen share output through Fishjam, then you most likely should refer to the documentation in Streaming media and Managing devices instead of this page.

This section demonstrates how to stream non-standard video or audio to other peers in your web app. The utilities in this section allow you to integrate Fishjam with powerful browser APIs such as WebGL and WebGPU, or higher level libraries, which leverage these APIs, such as Three.js, Smelter or PixiJS

Creating a custom source - useCustomSource()

To create a custom source, you only need to do two things:

1. Call the useCustomSource hook.
2. Call the setStream callback provided by the useCustomSource hook with a MediaStream object.

Usage Example

import React, { useEffect } from "react";
import { useCustomSource } from "@fishjam-cloud/react-client";

const stream: MediaStream | null = null; // Replace with your MediaStream object
const { setStream } = useCustomSource("my-custom-source");

useEffect(() => {
  setStream(stream);
}, [stream, setStream]);

Using a created custom source

Once you have called setStream for a given source ID (in the above example, "my-custom-source"), any subsequent calls to useCustomSource with the same ID will return the same state. This enables multiple components to control and read a shared custom source.

Deleting a custom source

If you wish to remove a custom source, then you should call the setStream callback with null as its argument.

Usage Example

const { setStream } = useCustomSource("my-custom-source");
// ...
await setStream(null);

How to get a MediaStream object?

Depending on your use-case, the way you can obtain a MediaStream will vary. Below are some common examples of how to obtain one from an existing video or audio source.

Canvas

If you have a <canvas> HTML element, then you will need to call the captureStream method of the given canvas. This is particularly useful if you are using Three.js or PixiJS to render complex animations and more.

Usage Example

import React, { useCallback, useState } from "react";

export function CanvasExample() {
  const [canvasStream, setCanvasStream] = useState<MediaStream>();

  const streamFromCanvas = useCallback((canvas: HTMLCanvasElement | null) => {
    if (!canvas) return;
    setCanvasStream(canvas.captureStream());
  }, []);

  // use canvasStream as desired

  return <canvas ref={streamFromCanvas} />;
}

Smelter

If you are using Smelter, then all you need to do is register an output via registerOutput.

Usage Example

tip

If you want to see a full example React app which uses Fishjam with Smelter, then you can check out our full example on GitHub.

import React from "react";
import Smelter, { setWasmBundleUrl } from "@swmansion/smelter-web-wasm";
import { View } from "@swmansion/smelter";

setWasmBundleUrl("/assets/smelter.wasm");

async function run() {
  const smelter = new Smelter({ framerate: 30 });
  const { stream } = await smelter.registerOutput("example-output", <View />, {
    type: "stream",
    video: {
      resolution: { width: 1920, height: 1080 },
    },
  });
  await smelter.init();
}

Video/Audio HTML tag

If you have a <video> or <audio> element, then you need to call their captureStream method, analogously as you would with a canvas.

Usage Example

Video

import React, { useCallback, useState } from "react";

const hasCaptureStream = (video: HTMLVideoElement): video is HTMLVideoElement & { captureStream: () => MediaStream } => "captureStream" in video && typeof video.captureStream === "function";

export function CanvasExample() {
  const [stream, setStream] = useState<MediaStream>();

  const streamFromVideo = useCallback((video: HTMLVideoElement | null) => {
    if (!video) return;
    if(!hasCaptureStream(video)) {
      console.warn("captureStream is not supported on this browser");
      return;
    }

    setStream(video.captureStream());
  }, []);

  // use stream as desired

  return <video ref={streamFromVideo} /* ... */ />;
}

Audio

import React, { useCallback, useState } from "react";

export function CanvasExample() {
  const [stream, setStream] = useState<MediaStream>();

  const streamFromAudio = useCallback((audio: HTMLAudioElement | null) => {
    if (!audio) return;
    setStream((audio as HTMLAudioElement & { captureStream(): MediaStream }).captureStream());
  }, []);

  // use stream as desired

  return <audio ref={streamFromAudio} /* ... */ />;
}