AI-901 ยท Text and speech solutions
Spoken prompts with an audio-capable model
Sending audio input to a deployed multimodal model and receiving text or spoken output, and how this differs from a separate speech-to-text then chat pipeline.
Key points
- Deploy an audio-capable chat model (for example gpt-4o-audio-preview or gpt-4o-mini-audio-preview).
- Send audio in the user message as a content part {type: input_audio, input_audio: {data: <base64>, format: wav}}.
- For spoken replies add modalities [text, audio] and audio {voice, format}; the reply audio arrives base64 in message.audio.data.
- One call replaces the pipeline of speech-to-text, chat, then text-to-speech.
Source: https://learn.microsoft.com/en-us/azure/foundry/openai/audio-completions-quickstart
Practice questions
What is the content part type used to place recorded audio in a chat completions user message?
- input_audio
- image_url
- audio_url
- speech_blob
Show answer
Answer: A. input_audio
The audio goes in a part of type input_audio carrying base64 data and a format. input_text and image_url are for other kinds of input.
Compared with transcribing speech first and then calling a text chat model, what does a single audio-capable model call offer?
- Audio goes straight into one chat request
- The answer can be text only or text plus spoken audio
- No separate transcription step is needed before the chat call
- An Azure Speech resource must transcribe the audio first
- Text prompts are no longer accepted
Show answer
Answer: A. Audio goes straight into one chat request, B. The answer can be text only or text plus spoken audio, C. No separate transcription step is needed before the chat call
The model accepts audio directly, can answer in text, and can also return speech in the same call. It does not require a prior Azure Speech transcription step, and text input is still allowed.
To send a spoken question to an audio-capable chat model, you must first convert it to text with Azure Speech.
Show answer
Answer: False
False. The model accepts audio input directly as an input_audio part. A speech-to-text step is only needed when you choose the separate pipeline design.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
