Bitzu

AI-901 ยท Text and speech solutions

Spoken prompts with an audio-capable model

Sending audio input to a deployed multimodal model and receiving text or spoken output, and how this differs from a separate speech-to-text then chat pipeline.

Key points

  • Deploy an audio-capable chat model (for example gpt-4o-audio-preview or gpt-4o-mini-audio-preview).
  • Send audio in the user message as a content part {type: input_audio, input_audio: {data: <base64>, format: wav}}.
  • For spoken replies add modalities [text, audio] and audio {voice, format}; the reply audio arrives base64 in message.audio.data.
  • One call replaces the pipeline of speech-to-text, chat, then text-to-speech.

Source: https://learn.microsoft.com/en-us/azure/foundry/openai/audio-completions-quickstart

Practice questions

Single choice

What is the content part type used to place recorded audio in a chat completions user message?

  1. input_audio
  2. image_url
  3. audio_url
  4. speech_blob
Show answer

Answer: A. input_audio

The audio goes in a part of type input_audio carrying base64 data and a format. input_text and image_url are for other kinds of input.

Select all that apply

Compared with transcribing speech first and then calling a text chat model, what does a single audio-capable model call offer?

  1. Audio goes straight into one chat request
  2. The answer can be text only or text plus spoken audio
  3. No separate transcription step is needed before the chat call
  4. An Azure Speech resource must transcribe the audio first
  5. Text prompts are no longer accepted
Show answer

Answer: A. Audio goes straight into one chat request, B. The answer can be text only or text plus spoken audio, C. No separate transcription step is needed before the chat call

The model accepts audio directly, can answer in text, and can also return speech in the same call. It does not require a prior Azure Speech transcription step, and text input is still allowed.

True or false

To send a spoken question to an audio-capable chat model, you must first convert it to text with Azure Speech.

Show answer

Answer: False

False. The model accepts audio input directly as an input_audio part. A speech-to-text step is only needed when you choose the separate pipeline design.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More AI-901 topics in this module