Apple announces AI for your iPhone — and OpenAI will be involved

During Apple’s messy and fast-paced WWDC 2024 keynote, the company announced its big move into AI — short for Apple Intelligence. Props to the clever marketers at Apple for hijacking an already well-established abbreviation. And, as rumored, Apple is officially partnering with OpenAI to integrate ChatGPT into the iPhone and iPad, but the vast majority of Apple’s new AI smarts are from Apple itself.

Unfortunately, Apple did a poor job of communicating this fact during the keynote, which it clarified during its State of the Union. While the keynote is aimed at a general audience, the State of the Union is when Apple addresses developers directly. The vast majority of Apple’s AI tasks will be handled by its own 3B parameter SLM that processes as much as possible on the device itself for speed and privacy. If it can’t be handled on-device, it’s farmed out to Apple’s Private Cloud Compute, and if that can’t handle the job, it will (with your permission), hand it off to ChatGPT or, at some point in the future, a model of your choice.

Not everyone appreciates Apple’s foray into AI or its partnership with OpenAI. Elon Musk was quick to post on X: “If Apple integrates OpenAI at the OS level, then Apple devices will be banned at my companies. That is an unacceptable security violation.” Of course, there is bad blood between Musk and OpenAI.

Before you get too spooked about AI-powered iPhones, realize that this is going to be a slow rollout. When launched (not necessarily at the same time as iOS 18, iPadOS 18, and macOS Sequoia), it will still be in beta, and it’s limited to the following devices:

• iPhone 15 Pro
• iPhone 15 Pro Max
• iPads with an M1 or later processor
• Macs with an M1 or later processor

So many of us will be waiting a while to experience Apple Intelligence. But what exactly is Apple Intelligence?

Like many of Apple’s big initiatives, such as Continuity or iCloud, Apple Intelligence is an umbrella branding term that encompasses many smaller but related features. Let’s explore them individually to explain what Apple Intelligence really is.

Supercharged Siri

Apple had an early lead on voice assistants with Siri, but Siri has fallen woefully behind in the last decade, and that’s apparent if you’ve spent any time with ChatGPT’s Voice Mode. Siri can barely understand simple commands, while ChatGPT can hold a full conversation with you.

Apple’s new operating systems will introduce a new, supercharged Siri. Apple was unclear about how much of it is built on Apple’s tech and how much is being borrowed from ChatGPT, so it’s unclear just how many of these features will be available to all users:

• Type to Siri: With iOS 18 and iPadOS 18, you can double-tap the bottom of the screen to bring up a text prompt where you can type requests to Siri instead of speaking them out loud.
• Product knowledge: In an attempt to put tech writers out of business, you’ll be able to directly ask Siri questions about your device and how to use it.
• Better handling of ambiguity: Voice assistants can be especially frustrating when we fumble in our speech, and the new Siri will be able to better handle that. An example Apple provides is, “Siri, set an alarm for — oh wait no, set a timer for 10 minutes. Actually, make that 5.”
• Context awareness: Siri will remember previous points in the conversation. For instance, if you ask for directions to a location, you can then follow up with “What is the weather there?”
• Screen awareness: Siri will be able to read what’s on your screen and act upon it. For instance, if a friend texts you his address, you can ask Siri to add it to his contact card.
• Personal context: The new Siri will collect data on your device and use it to better understand your intent. For instance, you can ask Siri to “Play that podcast my wife sent me the other day,” or you could ask it for your passport number, assuming it’s stored on your device.
• Action across apps: You’ll be able to give Siri commands like “Make this photo pop” and tell it to put that photo into a specific note in the Notes app.

Of course, with Siri knowing so much of your personal information from your devices, there are some serious privacy concerns that we’ll discuss below.

AI writing tools

Writing Tools are a set of new features coming to macOS, iOS, and iPadOS that let you select a block of text and take certain AI actions, such as:

• Proofread the text;
• Rewrite text to be more concise, friendly, or professional;
• Create a summary of the text;
• Break the text down into bullet points;
• Create a table or list from the text.

You can also enter a custom prompt to direct how you want the text to be rewritten.

Images Courtesy of Apple Newsroom

Image generation

Apple also introduced several new features to generate images. The one that seems most poised to be a hit is Genmoji, which lets you create a new, custom emoji based on a text description. You can also create a Genmoji that resembles a person from your photo library.

Maybe the creepiest of the new features is the built-in image generator, called Image Playground, which will be available for use in apps and will have an app of its own. It’s very much akin to MidJourney or DALLE-3.

One of the most unique features for the iPad is called Image Wand, where you draw a circle in the Notes app and it will generate an image based on the content of the note. I’ll be curious to find out how well it works.

Finally, AI is coming to the Photos app to help create Memories based on a text description, remove unwanted objects from photos, and improve photo search.

Ephemeral AI features

Apple is integrating AI throughout its operating systems in all sorts of little ways that will change your user experience. Apple’s AI will be able to:

• Prioritize emails and notifications it thinks are the most important;
• Summarize long emails;
• Help you automatically reply to emails;
• Transcribe voice notes;
• Use AI insights for a new Focus mode that eliminates distractions while elevating notifications that require your attention.

How the ChatGPT integration works

The built-in ChatGPT integration will be free for everyone, will not require an account, and will be completely opt-in: You’ll be prompted before Apple phones ChatGPT for help. If you have a paid subscription to ChatGPT Plus, you can log into your account and tap into those additional features.

Images Courtesy of Apple Newsroom

Unfortunately, we don’t yet know much more than that, and many think Apple is being gullible. “It’s patently absurd that Apple isn’t smart enough to make their own AI, yet is somehow capable of ensuring that OpenAI will protect your security & privacy! Apple has no clue what’s actually going on once they hand your data over to OpenAI. They’re selling you down the river,” Musk posted. Musk also pointed out that OpenAI had asked Scarlett Johansson to license her voice for ChatGPT, and did it anyway even after she refused.

There are many devils yet in the details, and more is sure to be revealed as this White Boy Summer progresses.