OpenAI’s Multimodal Toolformer: LLMs Now Call and Compose with APIs Across Images, Video, and More

OpenAI just published a preprint about Multimodal Toolformer, and it’s a big deal: core GPT models can now orchestrate API calls to external tools not just with text, but across images, video, and even audio pipelines. If you care about LLMs as software infrastructure, this is the next frontier.

Why does this matter for engineers?

Toolformer was already cool for letting LLMs ‘invoke’ APIs as part of a chain-of-thought. But the new twist is: models can now decide, mid-generation, “I need to call an OCR API on this image, then summarize the result, then call a video transcription API, then format output as JSON.” And it all happens in a single context window—no brittle controller code, no manual function routing.

This dramatically lowers the floor for building agentic LLMs that actually do real work, instead of just chatting. You can wire up vision APIs, database lookups, code execution engines, and let the model orchestrate complex tasks—like extracting key data from a scanned invoice and updating a spreadsheet, with full multimodal IO.

The catch: debugging tool-chaining at this level is nontrivial. There’s a lot of clever prompt engineering, careful model fine-tuning to avoid runaway API calls, and new risks around hallucinated tool usage. But if you want LLMs to move beyond toy demos and become the backbone of real agent systems, this is a giant step. Expect open-source variants to follow fast.