> ## Documentation Index
> Fetch the complete documentation index at: https://openclaw.ai2me.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Plugin runtime media helpers

Speech, media understanding, generation, web search, and the low-level media utilities. Part of the [Plugin runtime helpers](/plugins/sdk-runtime) reference.

## FFmpeg command discovery

Official plugin sources use `resolveFfmpegBin` from
the private `openclaw/plugin-sdk/media-ffmpeg` runtime. It resolves the same
trusted system paths as the media runtime and throws an installation hint when
FFmpeg is unavailable. This narrow entry point keeps media generation and agent
runtimes out of source audio worker startup. It has a JavaScript-only host export;
its declarations are excluded from the package. Third-party plugins retain the
existing `resolveFfmpegBin` export from `openclaw/plugin-sdk/media-runtime`.

The official plugin publication builder emits `media-runtime` for this private
import, including worker entries, so published plugins keep working on supported
hosts that predate `media-ffmpeg`. Both paths use the host's FFmpeg resolver.

## Realtime voice playback

Official audio workers use `createRealtimeVoiceOutputActivityTracker` and
`isRealtimeVoiceAudioAudible` from the private
`openclaw/plugin-sdk/realtime-voice-playback` runtime to avoid loading voice
session and agent runtimes from source. The publication builder emits the
existing `openclaw/plugin-sdk/realtime-voice` host import for these helpers,
including worker entries, to preserve supported hosts. The source facade has a
JavaScript-only host export; its declarations are excluded from the package.

## Media and generation namespaces

<AccordionGroup>
  <Accordion title="api.runtime.tts">
    Text-to-speech synthesis.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    // Standard TTS
    const clip = await api.runtime.tts.textToSpeech({
      text: "Hello from OpenClaw",
      cfg: api.config,
    });

    // Telephony-optimized TTS
    const telephonyClip = await api.runtime.tts.textToSpeechTelephony({
      text: "Hello from OpenClaw",
      cfg: api.config,
    });

    // List available voices
    const voices = await api.runtime.tts.listVoices({
      provider: "elevenlabs",
      cfg: api.config,
    });
    ```

    Uses core `tts` configuration and provider selection. Returns PCM audio buffer + sample rate. `textToSpeechStream` is also available for streaming synthesis.
  </Accordion>

  <Accordion title="api.runtime.mediaUnderstanding">
    Image, audio, and video analysis.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    // Describe an image
    const image = await api.runtime.mediaUnderstanding.describeImageFile({
      filePath: "/tmp/inbound-photo.jpg",
      cfg: api.config,
      agentDir: "/tmp/agent",
    });

    // Prepare a capture limit before installing audio receive listeners.
    const budget = await api.runtime.mediaUnderstanding.resolveAudioInputBudget({
      cfg: api.config,
    });
    // budget.enabled is false when audio understanding is disabled. Otherwise,
    // budget.maxBytes includes the container header and covers the largest fallback.

    // Transcribe audio
    const { text } = await api.runtime.mediaUnderstanding.transcribeAudioFile({
      filePath: "/tmp/inbound-audio.ogg",
      cfg: api.config,
      mime: "audio/ogg", // optional, for when MIME cannot be inferred
    });

    // Describe a video
    const video = await api.runtime.mediaUnderstanding.describeVideoFile({
      filePath: "/tmp/inbound-video.mp4",
      cfg: api.config,
    });

    // Generic file analysis
    const result = await api.runtime.mediaUnderstanding.runFile({
      filePath: "/tmp/inbound-file.pdf",
      cfg: api.config,
    });

    // Structured image extraction through a specific provider/model.
    // Include at least one image; text inputs are supplemental context.
    // receiptImageBuffer is your own image bytes, not an SDK-provided value.
    const evidence = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
      provider: "codex",
      model: "gpt-6-astra",
      input: [
        {
          type: "image",
          buffer: receiptImageBuffer,
          fileName: "receipt.png",
          mime: "image/png",
        },
        { type: "text", text: "Prefer the printed total over handwritten notes." },
      ],
      instructions: "Extract vendor, total, and searchable tags.",
      schemaName: "receipt.evidence",
      jsonSchema: {
        type: "object",
        properties: {
          vendor: { type: "string" },
          total: { type: "number" },
          tags: { type: "array", items: { type: "string" } },
        },
        required: ["vendor", "total"],
      },
      cfg: api.config,
    });
    ```

    Returns `{ text: undefined }` when no output is produced (e.g. skipped input).

    `describeImageFileWithModel(...)` describes an already-known image through a specific provider/model, bypassing the default active-model resolution that `describeImageFile(...)` uses.
  </Accordion>

  <Accordion title="api.runtime.imageGeneration">
    Image generation.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    const result = await api.runtime.imageGeneration.generate({
      prompt: "A robot painting a sunset",
      cfg: api.config,
    });

    const providers = api.runtime.imageGeneration.listProviders({ cfg: api.config });
    ```
  </Accordion>

  <Accordion title="api.runtime.videoGeneration">
    Video generation, mirroring the image generation shape.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    const result = await api.runtime.videoGeneration.generate({
      prompt: "A drone shot flying over a coastline at sunrise",
      cfg: api.config,
    });

    const providers = api.runtime.videoGeneration.listProviders({ cfg: api.config });
    ```
  </Accordion>

  <Accordion title="api.runtime.musicGeneration">
    Music generation, mirroring the image generation shape.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    const result = await api.runtime.musicGeneration.generate({
      prompt: "An upbeat lo-fi track for a coding session",
      cfg: api.config,
    });

    const providers = api.runtime.musicGeneration.listProviders({ cfg: api.config });
    ```
  </Accordion>

  <Accordion title="api.runtime.webSearch">
    Web search.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    const providers = api.runtime.webSearch.listProviders({ config: api.config });

    const result = await api.runtime.webSearch.search({
      config: api.config,
      args: { query: "OpenClaw plugin SDK", count: 5 },
    });
    ```

    Search callers may supply a synchronous `assertCurrent` callback with `signal`
    to retain their authority through provider preparation. Guarded HTTP requests
    check it after transport preparation and before each request or redirect.
    A registered search provider using another transport must call the execution
    context's `assertCurrent` before each external side effect, after awaited
    preparation. The callback belongs to the caller and expires when the search
    finishes; providers must not replace it or treat its absence as permission.
  </Accordion>

  <Accordion title="api.runtime.media">
    Low-level media utilities.

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    const webMedia = await api.runtime.media.loadWebMedia(url);
    const mime = await api.runtime.media.detectMime({ buffer });
    const kind = api.runtime.media.mediaKindFromMime("image/jpeg"); // "image"
    const isVoice = api.runtime.media.isVoiceCompatibleAudio({ fileName: filePath });
    const metadata = await api.runtime.media.getImageMetadata(buffer);
    const resized = await api.runtime.media.resizeToJpeg({ buffer, maxSide: 800, quality: 85 });
    ```

    QR helpers are exported by `openclaw/plugin-sdk/media-runtime`:

    ```typescript theme={"theme":{"light":"min-light","dark":"min-dark"}}
    import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";

    const qr = await import("openclaw/plugin-sdk/media-runtime");
    const terminalQr = await qr.renderQrTerminal("https://openclaw.ai");
    const pngQr = await qr.renderQrPngBase64("https://openclaw.ai", {
      scale: 6, // 1-12
      marginModules: 4, // 0-16
    });
    const pngQrDataUrl = await qr.renderQrPngDataUrl("https://openclaw.ai");
    const tmpRoot = resolvePreferredOpenClawTmpDir();
    const pngQrFile = await qr.writeQrPngTempFile("https://openclaw.ai", {
      tmpRoot,
      dirPrefix: "my-plugin-qr-",
      fileName: "qr.png",
    });
    ```
  </Accordion>
</AccordionGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.