Qwen-Omni-Turbo-Realtime

Copy success!

Add to Compare

Overview

The real-time version of Qwen's new large multimodal understanding and generation model. This model is a dynamically updated version.

Input

TextImageVideoAudio

Output

TextAudio

Features

Prefix Completion

Enable Partial Mode when calling the Qwen API to make the model continue strictly from your provided prefix text.View docs

Function Calling

Use function calling to connect large language models with external tools and systems.View docs

Cache

Context Cache stores shared prefixes for long-context requests to reduce repeated computation, improve latency, and lower cost.View docs

Structured Outputs

Structured Outputs help ensure the model returns a JSON string in the expected format.View docs

Batches

Asynchronously process requests in batches to reduce costs.View docs

Web Search

Enable web search so the model can answer with real-time retrieved data.View docs

Fine-tuning

Train models on sample data to better adapt them to specific tasks.View docs

Pricing

Input: Text
$0.27Per 1M tokens
Input: Audio
$4.44Per 1M tokens
Input: Vision
$0.84Per 1M tokens
Output: Text (When input contains only text)
$1.07Per 1M tokens
Output: Text (When input contains images/audio/video)
$2.52Per 1M tokens
Output: Text&Audio (Output text is not charged)
$8.89Per 1M tokens

Rate Limits & Context

Max Input
30K
Max Output
2K
Context
32K
TPMTokens Per Minute
10K
RPMRequests Per Minute
60

API Reference

Call API

Copy success!

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768

# Dependencies: dashscope >= 1.23.9, pyaudio
import os
import base64
import time

import pyaudio
from dashscope.audio.qwen_omni import MultiModality, AudioFormat, OmniRealtimeCallback, OmniRealtimeConversation
import dashscope


url = f'wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime'
# API key: if DASHSCOPE_API_KEY is not set, use: dashscope.api_key = "sk-xxx"
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
# Voice
voice = 'Ethan'
# Model
model = 'qwen-omni-turbo-realtime-latest'
# Assistant instructions
instructions = (
    "You are Xiaoyun, a personal assistant. Answer the user's questions in a humorous and witty way."
)
class SimpleCallback(OmniRealtimeCallback):
    def __init__(self, pya):
        self.pya = pya
        self.out = None
    def on_open(self):
        # Initialize audio output stream
        self.out = self.pya.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=24000,
            output=True
        )
    def on_event(self, response):
        if response['type'] == 'response.audio.delta':
            # Play audio
            self.out.write(base64.b64decode(response['delta']))
        elif response['type'] == 'conversation.item.input_audio_transcription.completed':
            # Print user transcript
            print(f"[User] {response['transcript']}")
        elif response['type'] == 'response.audio_transcript.done':
            # Print assistant transcript
            print(f"[LLM] {response['transcript']}")

# 1. Initialize audio device
pya = pyaudio.PyAudio()
# 2. Create callback and conversation
callback = SimpleCallback(pya)
conv = OmniRealtimeConversation(model=model, callback=callback, url=url)
# 3. Connect and configure session
conv.connect()
conv.update_session(output_modalities=[MultiModality.AUDIO, MultiModality.TEXT], voice=voice, instructions=instructions)
# 4. Initialize microphone input stream
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
# 5. Main loop: stream microphone audio
print("Conversation started. Speak into the microphone (Ctrl+C to exit)...")
try:
    while True:
        audio_data = mic.read(3200, exception_on_overflow=False)
        conv.append_audio(base64.b64encode(audio_data).decode())
        time.sleep(0.01)
except KeyboardInterrupt:
    # Clean up
    conv.close()
    mic.close()
    callback.out.close()
    pya.terminate()
    print("\nConversation ended.")