Version: 0.21.0

Fishjam Agent Introduction

tip

We recommend going through the steps in the Backend Quick Start before trying Fishjam agents, as you will need a working backend server to use them.

This page gives an introduction to Fishjam agents and how to use them. You can learn more about how Agents work on the Agent Internals page.

What is an Agent?

An agent is a piece of software that allows your backend server to participate in a Fishjam room, similar to how the Fishjam client SDKs allow your client-side application to participate in a Fishjam room. They can be used to implement features such as real-time audio transcription, audio recording, real-time content moderation and more.

You can simply think of an agent as a peer running within your backend application.

Writing an Agent

In this section we show how to implement an agent using the Fishjam server SDKs. If you are not using the SDKs, then you can check out the Agent Internals, to learn how to integrate with Fishjam Agents.

Prerequisites

Room

To create an agent, you first need a room, as agents are scoped to rooms. This Backend Quick Start section explains how to create a room, but the TL;DR is:

TypeScript
Python

import { FishjamClient } from '@fishjam-cloud/js-server-sdk';

const fishjamClient = new FishjamClient({ fishjamId, managementToken });

const room = await fishjamClient.createRoom();

from fishjam import FishjamClient

fishjam_client = FishjamClient(fishjam_id, management_token)

room = fishjam_client.create_room()

Peer subscriptions

By default, agents won't receive peers' audio streams. This is by design and aims to prevent unnecessary resource usage by the agents.

For an agent to start receiving a peer's audio, the peer must be created with the subscribe option set.

TypeScript
Python

import { FishjamClient } from '@fishjam-cloud/js-server-sdk';

const fishjamClient = new FishjamClient({ fishjamId, managementToken });

const peer = await fishjamClient.createPeer(
  roomId,
  { subscribe: { audioFormat: 'pcm16', audioSampleRate: 16000 } },
);
// Agents will receive peer's audio streams as raw PCM audio with 16kHz sample rate

from fishjam import FishjamClient, PeerOptions
from fishjam.agent import AgentResponseTrackData
from fishjam.peer import (
  SubscribeOptions,
  SubscribeOptionsAudioFormat,
  SubscribeOptionsAudioSampleRate
)

fishjam_client = FishjamClient(fishjam_id, management_token)

peer = fishjam_client.create_peer(
  room_id,
  options=PeerOptions(
    subscribe=SubscribeOptions(
      audio_format=SubscribeOptionsAudioFormat.PCM_16,
      audio_sample_rate=SubscribeOptionsAudioSampleRate.VALUE_16000,
    )
  )
)
# Agents will receive peer's audio streams as raw PCM audio with 16kHz sample rate

Creating an Agent

If you are using the server SDKs, then creating an agent and defining its behavior is very simple.

TypeScript
Python

import { FishjamClient, IncomingTrackData } from '@fishjam-cloud/js-server-sdk';

const fishjamClient = new FishjamClient({ fishjamId, managementToken });

const agent = await fishjamClient.createAgent(
  roomId,
  {},
  console.error,
  (code: number, reason: string) => console.log(`Agent disconnected with code: ${code} and reason: ${reason}`)
);
// Register callback for incoming audio data
agent.on('trackData', ({ track, peerId, data }: IncomingTrackData) => {
  // process the incoming data
})

from fishjam import FishjamClient
from fishjam.agent import AgentResponseTrackData

fishjam_client = FishjamClient(fishjam_id, management_token)

agent = fishjam_client.create_agent(room_id)
# Register callback for incoming audio data
@agent.on_track_data
def _(data: AgentResponseTrackData):
  pass

# Option 1: connect/disconnect manually
await agent.connect()
...
await agent.disconnect()

# Option 2: context manager
async with agent:
  ...

If you are using Fishjam's REST API directly, then check out this Agent Internals section.

Pricing

Agents are billed as if they were normal peers.

For example, a room with 2 peers and 1 agent connected will be billed as if there were 3 peers connected to the room. Exact pricing values can be found on our pricing page.

What is an Agent?​

Writing an Agent​

Prerequisites​

Room​

Peer subscriptions​

Creating an Agent​

Pricing​

See also​