WWDC 2026: Apple Integrates Third-Party AI Models Into Foundation Models Framework
In Focus
- The multimodal image input allows developers to send images and text together
- The Python SDK allows developers to build AI-powered apps using Python
- The model abstraction layer allows developers work with third-party AI systems
Apple announced updates to Foundation Models at the WWDC 2026. The updates largely focus on integrating AI capabilities across Apple’s operating systems and supporting third-party AI systems. The tech giant is expanding model support and bringing in new primitives to enable developers to build agentic experiences.
What is Apple’s Foundation Models Framework?
Apple introduced the Foundation Models framework last year. At the time, the framework came with features such as guided generation, snapshot streaming, and a powerful tool protocol. This year, Apple has announced multiple updates that expand its capabilities.
The Foundation Models framework is a Swift tool that allows developers to access the AI models built into Apple devices and used by Apple Intelligence. The framework also allows apps to work with other AI models, including cloud-based options like Claude and Gemini, as long as they conform to Apple’s Language Model standard.
During the WWDC 2026 keynote, Apple announced two major updates to its Foundation Models framework.
1. The Multimodal Image Input Update
This update allows developers to send images alongside text prompts to Apple’s on-device AI models. This enables developer apps to analyze and reason about the visual content. For instance, the apps can identify objects in images, extract text, and even understand screenshots. Apple’s multimodal image input update comes with integrated Vision framework tools like barcode and text recognition that AI models can use directly on the device.
2. Apple’s Python SDK Update
With the Python SDK update, Apple is making AI tools more flexible by giving developers access to the same on-device Foundation Models used by Apple Intelligence. Using the Python SDK, developers can create sessions, send prompts, and build AI-powered apps using Python instead of Swift. Apple has also extended support beyond Apple platforms. The company will also support Linux, which makes its Foundation Model framework accessible to AI researchers and developers.
How Does the Apple Model Abstraction Layer Work?
Beyond the multimodal image input and Python SDK updates, Apple is opening up its model abstraction layer. This move makes it possible for nearly any large language model (LLM) to be used with the Foundation Model framework.
The abstraction layer is designed around a new LanguageModel protocol that allows local and server models to back a LanguageModelSession. The LanguageModel protocol gives developers a single way to work with different AI models, including Apple’s on-device model, Anthropic’s Claude, and Google’s Gemini.
This means developers can switch between models with minimal code changes instead of rebuilding their apps. Additionally, Apple has added a layer that makes it easy to swap AI providers without changing the entire app code. This means the Foundation Models framework is no longer limited to Apple’s own AI models or ecosystem.
What the Models Update Means for Developers
Apple’s latest Foundation Models updates give developers more flexibility when building AI-powered apps. Instead of being limited to Apple’s on-device models, developers can now switch between Apple, Claude, Gemini, and other compatible AI models with minimal code changes.
The addition of image input support allows apps to understand both text and visual content, while the new Python SDK makes Apple’s AI tools accessible to a wider developer audience. These changes are expected to significantly reduce development complexity, making it easier for developers to build, test, and scale AI applications across different models and platforms.
