> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jay.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Session

# `configure_session`

An asynchronous function that runs at the beginning of every session. It configures the session
by returning a [`SessionConfig`](https://docs.jay.so/references/configure-session#param-session-config) object, which contains parameters like VAD (voice
activity detection), STT (speech-to-text), TTS (text-to-speech), the initial messages of the session, etc.

## Example usage

```py theme={null}
from jay_ai import ConfigureSessionInput, SessionConfig

async def configure_session(input: ConfigureSessionInput):
    user_timezone = input["custom_data"]["my_user_timezone"]
    return SessionConfig(
        initial_messages=[
          {"role": "system", "content": "You are a helpful assistant."}
        ],
        vad=VAD.Silero(),
        stt=STT.Deepgram(api_key=os.environ["DEEPGRAM_API_KEY"]),
        tts=TTS.OpenAI(
            api_key=os.environ["OPENAI_API_KEY"]
        ),
        session_data={
            "my_user_id": "test-12345",
            "my_user_timezone": user_timezone
        }
    )
```

## Parameters

<ResponseField name="input" type="object" required>
  <Expandable title="properties" defaultOpen="true">
    <ResponseField name="custom_data" type="object" required>
      Arbitrary fields that you can specify when you call the `startSession` API endpoint. Makes it possible to include fields that are specific to the session or to your users. Learn how to set these fields in the [Starting Sessions guide](https://docs.jay.so/guides/starting-sessions).
    </ResponseField>
  </Expandable>
</ResponseField>

Example `input` parameter:

```python theme={null}
{
  "custom_data": {
    "my_user_id": "abc123"
  }
}
```

## Returns

<ResponseField name="SessionConfig" type="object" required>
  <Expandable title="properties" defaultOpen="true">
    <ResponseField name="initial_messages" type="object" required>
      A list of messages containing the conversation so far. Make this an empty array if you want the conversation to start from scratch.

      <Expandable title="properties">
        <ResponseField name="content" type="string" required>
          The contents of the message.
        </ResponseField>

        <ResponseField name="role" type="string" required>
          The role of the speaker. Either 'system', 'user', 'assistant', or 'tool'.
        </ResponseField>

        <ResponseField name="name" type="string (Optional)">
          An optional name for the speaker. Some LLMs, such as OpenAI's, can use this field to differentiate between participants of the same role.
        </ResponseField>

        <ResponseField name="tool_call_id" type="string (Optional)">
          Tool call that this message is responding to. Only present if the `role` is `"tool"`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="vad" type="VAD.Silero" required>
      The voice activity detection (VAD) provider and its settings. Currently, only Silero is supported.

      <Expandable title="properties">
        <ResponseField name="min_speech_duration" type="float" default="0.05">
          The minimum duration (in seconds) of speech needed before the VAD decides speech has started.
        </ResponseField>

        <ResponseField name="min_silence_duration" type="float" default="0.55">
          The duration (in seconds) of silence required before the VAD concludes that speech has ended.
        </ResponseField>

        <ResponseField name="prefix_padding_duration" type="float" default="0.5">
          The duration (in seconds) of audio to include before detected speech begins. This helps retain audio leading up to the first speech frames.
        </ResponseField>

        <ResponseField name="max_buffered_speech" type="float" default="60.0">
          The maximum duration (in seconds) of speech that will be buffered. Once this limit is reached, additional incoming speech data for the current segment will be ignored.
        </ResponseField>

        <ResponseField name="activation_threshold" type="float" default="0.5">
          The threshold for deciding if audio is speech. A value closer to 1.0 requires stronger confidence of speech, while a value closer to 0.0 is more permissive.
        </ResponseField>

        <ResponseField name="sample_rate" type="int" default="16000">
          The audio sample rate (in Hz) used by Silero. Must be either 8,000 Hz or 16,000 Hz.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="stt" type="STT.OpenAI | STT.Azure | STT.Deepgram" required>
      The speech-to-text (STT) provider and its settings.

      <Expandable title="possible types">
        <ResponseField name="OpenAI" type="STT.OpenAI">
          OpenAI's speech-to-text provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your OpenAI API key.
            </ResponseField>

            <ResponseField name="language" type="string" default="en">
              The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes) format will improve accuracy and latency. Defaults to `"en"` (English).
            </ResponseField>

            <ResponseField name="model" type="string" default="whisper-1">
              The OpenAI model that will perform the speech-to-text.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Azure" type="STT.Azure">
          Microsoft Azure's speech-to-text provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your Azure Speech API key.
            </ResponseField>

            <ResponseField name="region" type="string" required>
              The Azure region that hosts your resource (e.g. `"eastus"`).
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="16000">
              The sample rate for the stream.
            </ResponseField>

            <ResponseField name="num_channels" type="int" default="1">
              The number of audio channels.
            </ResponseField>

            <ResponseField name="languages" type="array" default="[&#x22;en-US&#x22;]">
              An array of strings containing the potential source languages. The language is specified in [BCP-47](https://www.rfc-editor.org/info/bcp47) format.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Deepgram" type="STT.Deepgram">
          Deepgram's speech-to-text provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your Deepgram API key.
            </ResponseField>

            <ResponseField name="model" type="string" default="nova-2-general">
              The AI model used to process submitted audio. [Learn More](https://developers.deepgram.com/docs/model)
            </ResponseField>

            <ResponseField name="language" type="string" default="en-US">
              The [BCP-47](https://www.rfc-editor.org/info/bcp47) language tag that hints at the primary spoken language. [Learn More](https://developers.deepgram.com/docs/language).
            </ResponseField>

            <ResponseField name="interim_results" type="bool" default="true">
              Specifies whether the streaming endpoint should provide ongoing transcription updates as more audio is received. When set to `true`, the endpoint sends continuous updates, meaning transcription results may evolve over time. [Learn More](https://developers.deepgram.com/docs/interim-results).
            </ResponseField>

            <ResponseField name="punctuate" type="bool" default="true">
              Indicates whether to add punctuation and capitalization to the transcript. [Learn More](https://developers.deepgram.com/docs/punctuation).
            </ResponseField>

            <ResponseField name="smart_format" type="bool" default="true">
              Indicates whether to apply formatting to transcript output. When set to `true`, additional formatting will be applied to transcripts to improve readability. [Learn More](https://developers.deepgram.com/docs/smart-format).
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="16000">
              Sample rate of submitted streaming audio. Required (and only read) when a value is provided for encoding. [Learn More](https://developers.deepgram.com/docs/sample-rate).
            </ResponseField>

            <ResponseField name="no_delay" type="bool" default="true">
              Only relevant when `smart_format` is `true`. If a speaker begins saying a number while `no_delay` is `false` and Smart Format is enabled, Deepgram will wait to return a transcription until the speaker has finished and continues on to non-numerical speech. This behavior ensures numbers have the best possible formatting and are not broken up over multiple chunks.
            </ResponseField>

            <ResponseField name="endpointing_ms" type="int" default="25">
              Indicates how long Deepgram will wait to detect whether a speaker has finished speaking or pauses for a significant period of time. When set to `true`, the streaming endpoint immediately finalizes the transcription for the processed time range and returns the transcript with a `speech_final` parameter set to `true`. [Learn More](https://developers.deepgram.com/docs/endpointing).
            </ResponseField>

            <ResponseField name="filler_words" type="bool" default="true">
              Indicates whether to include filler words like "uh" and "um" in transcript output. When set to `true`, these words will be included. [Learn More](https://developers.deepgram.com/docs/filler-words).
            </ResponseField>

            <ResponseField name="keywords" type="array of (string, float)" default="[]">
              Unique proper nouns or specialized terms you want the model to include in its predictions, which aren't part of the model's default vocabulary. [Learn More](https://developers.deepgram.com/docs/keywords).
            </ResponseField>

            <ResponseField name="profanity_filter" type="bool" default="false">
              Indicates whether to remove profanity from the transcript. [Learn More](https://developers.deepgram.com/docs/profanity-filter).
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="tts" type="TTS.OpenAI | TTS.ElevenLabs | TTS.Google | TTS.Azure | TTS.Deepgram | TTS.Cartesia | TTS.FishAudio" required>
      The text-to-speech (TTS) provider and its settings.

      <Expandable title="possible types">
        <ResponseField name="ElevenLabs" type="TTS.ElevenLabs">
          ElevenLabs' text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your ElevenLabs API key.
            </ResponseField>

            <ResponseField name="voice" type="Voice" default="DEFAULT_VOICE">
              The voice to use. Defaults to:

              ```
              DEFAULT_VOICE = Voice(
                  id="EXAVITQu4vr4xnSDxMaL",
                  name="Bella",
                  category="premade",
                  settings=VoiceSettings(
                      stability=0.71, similarity_boost=0.5, style=0.0, use_speaker_boost=True
                  ),
              )
              ```

              <Expandable title="Voice">
                <ResponseField name="id" type="string" required>
                  Voice ID to use. You can use [`https://api.elevenlabs.io/v1/voices`](https://api.elevenlabs.io/v1/voices) to list all the available voices.
                </ResponseField>

                <ResponseField name="name" type="string" required>
                  The name of the ElevenLabs voice (e.g. "Sarah").
                </ResponseField>

                <ResponseField name="category" type="string" required>
                  The category of ElevenLabs voice (e.g. "professional"). See the [full list of valid ElevenLabs `category` fields](https://elevenlabs.io/docs/api-reference/voices/get-all#response.body.voices.category)
                </ResponseField>

                <ResponseField name="settings" type="VoiceSettings or null" default="null">
                  <Expandable title="VoiceSettings">
                    <ResponseField name="stability" type="float" required>
                      Defines the stability for voice settings.
                    </ResponseField>

                    <ResponseField name="similarity_boost" type="float" required>
                      Defines the similarity boost for voice settings.
                    </ResponseField>

                    <ResponseField name="style" type="float or null" default="null">
                      Defines the style for voice settings. This parameter is available on V2+ models.
                    </ResponseField>

                    <ResponseField name="use_speaker_boost" type="bool or null" default="false">
                      Defines the use speaker boost for voice settings. This parameter is available on V2+ models.
                    </ResponseField>
                  </Expandable>
                </ResponseField>
              </Expandable>
            </ResponseField>

            <ResponseField name="model" type="string" default="eleven_turbo_v2_5">
              Identifier of the ElevenLabs model that will be used.
            </ResponseField>

            <ResponseField name="encoding" type="string" default="mp3_22050_32">
              The output format of the generated audio.
            </ResponseField>

            <ResponseField name="enable_ssml_parsing" type="bool" default="false">
              Whether to enable/disable parsing of SSML tags within the provided text. For best results, we recommend enabling SSML tags so that fully contained messages are sent to the websockets endpoint, otherwise this may result in additional latency.
            </ResponseField>

            <ResponseField name="chunk_length_schedule" type="array of int" default="[80, 120, 200, 260]">
              Schedule for chunk lengths, ranging from 50 to 500.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="OpenAI" type="TTS.OpenAI">
          OpenAI's text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your OpenAI API key.
            </ResponseField>

            <ResponseField name="model" type="string" default="tts-1">
              One of the available OpenAI TTS models.
            </ResponseField>

            <ResponseField name="voice" type="string" default="alloy">
              The voice to use when generating the audio.
            </ResponseField>

            <ResponseField name="speed" type="float" default="1.0">
              The speed of the generated audio. Select a value from 0.25 to 4.0.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Google" type="TTS.Google">
          Google's text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="credentials" type="object" required>
              Your Google credentials as a dictionary.
            </ResponseField>

            <ResponseField name="language" type="string" default="en-US">
              The language code of the voice to use when generating the audio.
            </ResponseField>

            <ResponseField name="gender" type="string" default="neutral">
              The gender of the speaker. Valid values are: "male", "female", "neutral", or the empty string "". Use an empty string if it doesn't matter which gender the selected voice will have.
            </ResponseField>

            <ResponseField name="voice_name" type="string" default="">
              The name of the voice. If both the name and the gender are not set, the service will choose a voice based on the other parameters such as `language_code`.
            </ResponseField>

            <ResponseField name="encoding" type="string" default="linear16">
              Configuration to set up audio encoder. The encoding determines the output audio format that we'd like. Valid values: "linear16", "wav", or "mp3".
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="24000">
              Audio sample rate (in hertz) for this audio.
            </ResponseField>

            <ResponseField name="pitch" type="int" default="0">
              Speaking pitch, in the range \[-20.0, 20.0]. 20 means increase 20 semitones from the original pitch. -20 means decrease 20 semitones from the original pitch.
            </ResponseField>

            <ResponseField name="effects_profile_id" type="string" default="">
              An identifier which selects 'audio effects' profiles that are applied on (post synthesized) text to speech. Effects are applied on top of each other in the order they are given. See [audio profiles](https://cloud.google.com/text-to-speech/docs/audio-profiles) in Google's documentation for current supported profile ids.
            </ResponseField>

            <ResponseField name="speaking_rate" type="float" default="1.0">
              Speaking rate/speed, in the range \[0.25, 4.0]. 1.0 is the normal native speed supported by the specific voice. 2.0 is twice as fast, and 0.5 is half as fast.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Azure" type="TTS.Azure">
          Microsoft Azure's text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your Azure Speech API key.
            </ResponseField>

            <ResponseField name="region" type="string" required>
              The Azure region that hosts your resource (e.g. `"eastus"`).
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="24000">
              Audio sample rate (in hertz) for this audio. Valid values are: 8000, 16000, 22050, 24000, 44100, 48000.
            </ResponseField>

            <ResponseField name="voice" type="string or null" default="null">
              The name of the voice to use. [Explore Azure voices](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts).
            </ResponseField>

            <ResponseField name="language" type="string or null" default="null">
              The [BCP-47](https://www.rfc-editor.org/info/bcp47) language tag of the voice.
            </ResponseField>

            <ResponseField name="prosody" type="ProsodyConfig or null" default="null">
              Specify changes to pitch, contour, range, rate, and volume for the text to speech output. [Learn more](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup-voice#adjust-prosody)

              <Expandable title="ProsodyConfig">
                <ResponseField name="rate" type="str | float | null" default="null">
                  Indicates the speaking rate of the text. The rate changes should be within 0.5 to 2 times the original audio. You can express rate as:

                  * A number that acts as a multiplier of the default. For example, a value of 1 results in no change in the original rate. A value of 0.5 results in a halving of the original rate. A value of 2 results in twice the original rate.
                  * A string:
                    * `"x-slow"` (equivalently 0.5)
                    * `"slow"` (equivalently 0.64)
                    * `"medium"` (equivalently 1, default value)
                    * `"fast"` (equivalently 1.55)
                    * `"x-fast"` (equivalently 2)
                </ResponseField>

                <ResponseField name="volume" type="str | float | null" default="null">
                  Indicates the volume level of the speaking voice. You can express the volume as:

                  * A number in the range of 0.0 to 100.0, from quietest to loudest, such as 75. The default value is 100.
                  * A string:
                    * "`silent`" (equivalently 0)
                    * "`x-soft`" (equivalently 20)
                    * "`soft`" (equivalently 40)
                    * "`medium`" (equivalently 60)
                    * "`loud`" (equivalently 80)
                    * "`x-loud`" (equivalently 100, default value)
                </ResponseField>

                <ResponseField name="pitch" type="str | null" default="null">
                  Indicates the baseline pitch for the text. The pitch changes should be within 0.5 to 1.5 times the original audio. You can express the pitch as:

                  * `"x-low"`
                  * `"low"`
                  * `"medium"`
                  * `"high"`
                  * `"x-high"`
                </ResponseField>
              </Expandable>
            </ResponseField>

            <ResponseField name="endpoint_id" type="string or null" default="null">
              The ID of a custom endpoint. [Learn more](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-speech-synthesis?tabs=browserjs%2Cterminal\&pivots=programming-language-python#use-a-custom-endpoint)
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Deepgram" type="TTS.Deepgram">
          Deepgram's text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your Deepgram API key.
            </ResponseField>

            <ResponseField name="model" type="string" default="aura-asteria-en">
              The AI model used to process submitted audio. [Learn more](https://developers.deepgram.com/docs/model).
            </ResponseField>

            <ResponseField name="encoding" type="string" default="linear16">
              Expected encoding of the submitted streaming audio. [Learn more](https://developers.deepgram.com/docs/encoding).
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="24000">
              Sample rate of submitted streaming audio. [Learn more](https://developers.deepgram.com/docs/sample-rate).
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="Cartesia" type="TTS.Cartesia">
          Cartesia's text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your Cartesia API key.
            </ResponseField>

            <ResponseField name="model" type="string" default="sonic-english">
              The ID of the model to use for the generation. See [Cartesia's available models](https://docs.cartesia.ai/build-with-sonic/models).
            </ResponseField>

            <ResponseField name="language" type="string" default="en">
              The language that the given voice should speak the transcript in. See [Cartesia's available languages](https://docs.cartesia.ai/build-with-sonic/models).
            </ResponseField>

            <ResponseField name="encoding" type="string" default="pcm_s16le">
              The audio encoding format. Currently, only `"pcm_s16le"` is supported.
            </ResponseField>

            <ResponseField name="voice" type="string or array of float" default="c2ac25f9-ecc4-4f56-9095-651354df60c0">
              Either the string ID of the voice or a 192-dimensional vector (i.e. a list of 192 numbers) that represents the voice.
            </ResponseField>

            <ResponseField name="speed" type="float or null" default="null">
              Either a number between -1.0 and 1.0 or a natural language description of speed ("fastest", "fast", "normal", "slow", "slowest"). If you specify a number, 0.0 is the default speed, -1.0 is the slowest speed, and 1.0 is the fastest speed.
            </ResponseField>

            <ResponseField name="emotion" type="array of string or null" default="null">
              The emotion of the speaker (e.g. "positivity:high"). See [Cartesia's emotion guide](https://docs.cartesia.ai/build-with-sonic/capability-guides/control-speed-and-emotion#emotion-options).
            </ResponseField>

            <ResponseField name="sample_rate" type="int" default="24000">
              Sample rate (in hertz) for this audio.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="FishAudio" type="TTS.FishAudio">
          FishAudio text-to-speech provider.

          <Expandable title="properties">
            <ResponseField name="api_key" type="string" required>
              Your FishAudio API key.
            </ResponseField>

            <ResponseField name="model_id" type="string" default="802e3bc2b27e49c2995d23ef70e6ac89">
              Identifier of the FishAudio voice model to use. Defaults to energetic male voice: [https://fish.audio/m/802e3bc2b27e49c2995d23ef70e6ac89/](https://fish.audio/m/802e3bc2b27e49c2995d23ef70e6ac89/)
            </ResponseField>

            <ResponseField name="latency" type="string" default="balanced">
              Allows you to control the tradeoff between stability and latency. Setting this to `"balanced"` reduces latency, but may be less stable.
            </ResponseField>

            <ResponseField name="normalize" type="bool" default="true">
              Enables normalization of the input text, which improves stability for numbers, dates, and URLs.
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="session_data" type="object" required>
      Arbitrary fields that will be available throughout the session (e.g. in the `llm_response_handler`). Allows you to define custom data related to the user or session. Must be JSON serializable.
    </ResponseField>

    <ResponseField name="first_message" type="string (Optional)">
      An optional string representing a system or agent message to pre-send to the session.
    </ResponseField>

    <ResponseField name="allow_interruptions" type="bool">
      Whether user speech can interrupt the agent mid-speech. Defaults to `true`.
    </ResponseField>

    <ResponseField name="interrupt_time_threshold" type="float">
      Minimum amount of time (in seconds) of user speech that must be detected before the agent's speech is interrupted. Defaults to `0.5`.
    </ResponseField>

    <ResponseField name="interrupt_word_threshold" type="int">
      Minimum number of words spoken by the user that are required to interrupt agent speech. Defaults to `0`.
    </ResponseField>

    <ResponseField name="min_endpointing_delay" type="float">
      Specifies the minimum endpointing delay for STT. Defaults to `0.5`.
    </ResponseField>

    <ResponseField name="max_nested_function_calls" type="int">
      Maximum number of nested function calls allowed. Defaults to `1`.
    </ResponseField>
  </Expandable>
</ResponseField>