> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jay.so/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Response Handler

# `llm_response_handler`

An asynchronous function that's responsible for returning the LLM's response. This function is
called every time a response is expected from the agent during a session. Inside this function, you
can truncate the chat history, call a RAG pipeline, use any LLM provider, or add any other logic
that controls the LLM's response.

## Example usage

```py theme={null}
from jay_ai import LLMResponseHandlerInput

async def llm_response_handler(
    input: LLMResponseHandlerInput
):
    user_timezone = input["session_data"]["my_user_timezone"]

    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    messages = input["messages"] + [{"role": "system", "content": f"User timezone: {user_timezone}"}]
    completion = await client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        stream=True,
    )
    return completion
```

## Parameters

<ResponseField name="input" type="object" required>
  <Expandable title="properties" defaultOpen="true">
    <ResponseField name="messages" type="array of objects" required>
      A list of the entire chat history that has occurred so far in the conversation.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="content" type="string" required>
          The text of the message
        </ResponseField>

        <ResponseField name="role" type="string" required>
          The role of the speaker
        </ResponseField>

        <ResponseField name="name" type="string | None">
          An optional name for the speaker. Some LLMs, such as OpenAI's, can use this field to differentiate between participants of the same role.
        </ResponseField>

        <ResponseField name="tool_call_id" type="string | None">
          If the `role` is `"tool"`, this is the tool call ID that this message is responding to. Otherwise, it's `None`.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="session_data" type="object" required>
      Custom data that you specified in the [`SessionConfig`](https://docs.jay.so/references/configure-session#param-session-config) object
    </ResponseField>
  </Expandable>
</ResponseField>

Example `input` parameter:

```json theme={null}
{
  "messages": [
    {
      "content": "Hello!",
      "role": "user"
    }
  ],
  "session_data": {
    "my_user_id": "abc123"
  }
}
```

## Returns

A stream of chat completion chunks. Each of these chunks must be in the format
below, which conforms to OpenAI's streamed chat completion chunk specification. This means that if
you use an LLM client that conforms to OpenAI's API, such as the official `openai` package, the
generated responses will be in the correct format automatically.

Example returned chunks:

```json theme={null}
// Chunk 1
{
  "id": "chatcmpl-123",
  "object": "chat.completion.chunk",
  "created": 1694268190,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "delta": { "role": "assistant", "content": "The " },
      "finish_reason": null
    }
  ]
}

// Chunk 2
{
  "id": "chatcmpl-123",
  "object": "chat.completion.chunk",
  "created": 1694268190,
  "model": "gpt-4o",
  "choices": [
    { "index": 0, "delta": {}, "logprobs": null, "finish_reason": "stop" }
  ]
}
```

<ResponseField name="ChatCompletionChunk" type="object" required>
  <Expandable title="properties" defaultOpen="true">
    <ResponseField name="id" type="string" required>
      A unique identifier for the chat completion. Each chunk has the same ID.
    </ResponseField>

    <ResponseField name="choices" type="array" required>
      A list of chat completion choices. Must be an array containing either one element or zero elements.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="delta" type="object" required>
          A chat completion delta generated by streamed model responses.

          <Expandable title="properties" defaultOpen="true">
            <ResponseField name="content" type="string or null">
              The contents of the chunk message.
            </ResponseField>

            <ResponseField name="tool_calls" type="array or null">
              A list of tool calls generated by the model.

              <Expandable title="properties" defaultOpen="true">
                <ResponseField name="index" type="integer" required>
                  The sequential index for this tool call.
                </ResponseField>

                <ResponseField name="id" type="string">
                  The ID of the tool call.
                </ResponseField>

                <ResponseField name="type" type="string or null">
                  The type of the tool. Currently, only `"function"` is supported. Must be a string in the very first chunk and null in subsequent chunks.
                </ResponseField>

                <ResponseField name="function" type="object">
                  <Expandable title="properties">
                    <ResponseField name="name" type="string or null">
                      The name of the function to call. Must be a string in the very first chunk and null in subsequent chunks.
                    </ResponseField>

                    <ResponseField name="arguments" type="string">
                      The arguments to call the function with, as generated by the model
                      in JSON format.
                    </ResponseField>
                  </Expandable>
                </ResponseField>
              </Expandable>
            </ResponseField>

            <ResponseField name="role" type="string or null">
              The role of the author of this message. Must be a string in the very first chunk and null in subsequent chunks.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="finish_reason" type="string or null" required>
          The reason the model stopped generating tokens. This must be
          `stop` if the model hit a natural stop point or a provided
          stop sequence, `length` if the maximum number of tokens specified in the
          request was reached, `content_filter` if content was omitted due to a flag from a
          content filter, or `tool_calls` if the model called a tool.
        </ResponseField>

        <ResponseField name="index" type="integer" required>
          The index of the choice in the list of choices.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="created" type="integer" required>
      The Unix timestamp (in seconds) of when the chat completion was created. Each chunk must have the same timestamp.
    </ResponseField>

    <ResponseField name="model" type="string" required>
      The model that generated the completion.
    </ResponseField>

    <ResponseField name="object" type="string" required>
      The object type, which is always `"chat.completion.chunk"`.
    </ResponseField>

    <ResponseField name="usage" type="object or null">
      Must be `null` for every chunk except for the last chunk, which can optionally contain the token usage statistics for the entire request.

      <Expandable title="properties" defaultOpen="true">
        <ResponseField name="prompt_tokens" type="integer" required>
          Number of tokens in the prompt.
        </ResponseField>

        <ResponseField name="completion_tokens" type="integer" required>
          Number of tokens in the generated completion.
        </ResponseField>

        <ResponseField name="total_tokens" type="integer" required>
          Total number of tokens used in the request (prompt + completion).
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>