APPLE's Vision Pro is 'spatial computing.' Nobody knows what it means...
APPLE’S VR HEADSET WILL SCREW YOUR LIFE UP
Story by Geoffrey A. Fowler
Imagine you’re in a waiting room, and someone sits next to you with four iPhones strapped to their forehead. You might swiftly relocate.
Yet that’s exactly what’s happening when someone straps on Apple’s new Vision Pro headset. Each of these goggles contains the rough equivalent to a head full of iPhones: 2 depth sensors, 6 microphones and 12 cameras. It uses them to continuously track people and rooms in three dimensions — every hand gesture, eyeball flick and couch cushion.
Apple touts the $3,499 Vision Pro, arriving on Friday, as the next big thing after the smartphone. When you wear one, you see the world around you with computer-generated images and information superimposed on top. You might be intrigued or think the idea of a face computer is dumb. Regardless, you might want to know this device collects more data than any other personal device I’ve ever seen.
If this is our potential future, then I’ve got lots of questions. At launch, Apple has taken steps to restrict some of the data collected by the Vision Pro, including what people’s eyes are looking at. That’s a very good thing. But there are also new kinds of risks Apple doesn’t appear to have addressed, or might not be able to given how the tech works.
I see a privacy mess waiting to happen. Among the new dilemmas flagged to me by privacy researchers: Who gets to access the maps these devices build of our homes and data about how we move our bodies? A Vision Pro could reveal much more than you realize.
The last time a gadget raised these sorts of societal questions was in 2013 with Google Glass. It contained a small screen and just one camera that people worried might be used to covertly record them. Glass was so reviled, the nickname for people wearing them was Glassholes. Now we have to brace for, perhaps, the Vision Bros.
Most of my Vision Pro concerns are, at this point, speculative. But it matters to all of us if the technology Apple and others are inventing to replace smartphones could end up supercharging online problems like location tracking, the loss of anonymity and data brokers gathering the intimate details of our lives.
“Should we as a society really be going headfirst into virtual reality and augmented reality in our lives before we have strong privacy legislation?” says Cooper Quintin, senior public interest technologist at the Electronic Frontier Foundation. “Data brokers already have way too much intimate knowledge about everything I do. I don’t want them to have this level of knowledge.”
Adding to my concern is that Apple, which has staked its reputation on privacy, wouldn’t answer most of my questions about how the Vision Pro will tackle these problems. Nor has it, to date, allowed The Washington Post to independently test the hardware.
But from Apple’s limited statements, as well as conversations with developers making apps for the Vision Pro, I’ve been able to piece together a picture of its initial privacy strategy — and what it’s not talking about.
Homes and bodies, up for grabs?
I’m pretty sure Apple does not want to be known for creating the ultimate surveillance machine. But to make magical things happen inside its goggles, apps need loads of information about what’s happening to the user and around them. Apple has done more than rivals like Meta to limit access to some of this data, but developers are going to keep pressing for more.
“There’s a tension between having these types of experiences and your privacy,” says Jarrett Webb, technology director at design firm Argo, who has been exploring developing for the Vision Pro. “It has to get this data to get an understanding of the world to invoke these experiences.”
And once developers have data, it’s hard to ensure they don’t also use it for purposes that might feel like a violation.
On some issues, Apple has drawn a line in the sand — at least initially. To combat people being surreptitiously filmed with the Vision Pro, there’s an indicator on the device’s front screen when it’s shooting a photo or video. Apple also isn’t allowing third-party Vision Pro apps to access the camera to capture photos and videos. That would, in theory, also prevent third-party apps from doing creepy things like running facial recognition algorithms on people while you’re looking at them.
But privacy researchers tell me photographs alone aren’t the biggest concern here. We have, since the days of Google Glass, come to terms with the idea a smartphone could be filming us at any time.
The new problem is what else the device is gathering: a map of the spaces around you. The device needs to know the contours of the world around you so it can know where to insert digital things into your line of sight.
Understanding what’s in the room around you can be even more invasive than having a photograph of it, says Joseph Jerome, a visiting professor at the University of Tampa and the former policy lead on sensor data at Meta’s Reality Labs.
Vision Pro apps have the ability to access this data, if a user grants permission — like how an iPhone app asks for your location. These worldview maps might just look like a wireframe mesh to a human, but to a computer it reveals a lot.
On a basic level, the Vision Pro might know it’s in a room with four walls and a 12-foot ceiling and window — so far, so good, Jerome says. But then add in that you’ve got a 75-inch television, suggesting you might have more money to spend than someone with a 42-inch set. Since the device can understand objects, it could also detect if you’ve got a crib or a wheelchair or even drug paraphernalia, he says.
Advertisers and data brokers who build profiles of consumers would salivate at the chance to get this data. Governments, too.
Think of it as an extension of the kinds of issues we know can come from someone tracking your location. A phone alone, Jerome says, might be able to report that you’re generally near a hospital or a strip club. “These devices know where you are down to the centimeter, and then they’re combining it with a bunch of other sensors to know exactly what you’re looking at at the same time,” he says.
APPLE AND TIM COOK STEAL YOUR IDEAS AND PATENTS
SEE HOW THEY DO IT AT: HTTP://WWW.USINVENTOR.ORG
When Apple Comes Calling, ‘It’s the Kiss of Death’
– Aspiring partners accuse tech giant of copying their ideas
– Apple steals what-ever they want and then aims billions of dollars of in-house lawyers at the hapless actual inventors that Apple steals from
By Aaron Tilley
It sounded like a dream partnership when Apple Inc. AAPL -0.16% reached out to Joe Kiani, the founder of a company that makes blood-oxygen measurement devices. He figured his technology was a perfect fit for the Apple Watch.
Soon after meeting him, Apple began hiring employees from his company, Masimo Corp., MASI -0.70%
including engineers and its chief medical officer. Apple offered to double their salaries, Mr. Kiani said. In 2019, Apple published patents under the name of a former Masimo engineer for sensors similar to Masimo’s, documents show. The following year, Apple launched a watch that could measure blood oxygen levels.
“When Apple takes an interest in a company, it’s the kiss of death,” said Mr. Kiani. “First, you get all excited. Then you realize that the long-term plan is to do it themselves and take it all.”
Hell in the metaverse: How virtual reality worlds are a breeding ground for gang rapes, child grooming and sexual harassment… with sick perverts hiding behind their avatars
DailyMail.com has witnessed such acts firsthand after spending time in Meta’s Horizon Worlds, finding it’s becoming a haven for people to carry out sexual assaults.
Mr. Kiani is one of more than two dozen executives, inventors, investors and lawyers who described similar encounters with Apple. First, they said, came discussions about potential partnerships or integration of their technology into Apple products. Then, they said, talks stopped and Apple launched its own similar features.
Apple said that it doesn’t steal technology and that it respects the intellectual property of other companies. It said Masimo and other companies cited in this article are copying Apple, and that it would fight the claims in court.
Apple has tried to invalidate hundreds of patents owned by companies that have accused Apple of violating their patents. According to lawyers and executives at some smaller companies, Apple sometimes files multiple petitions on a single patent claim and attempts to invalidate patents unrelated to the initial dispute.
Many large companies, particularly in tech, have been known to scoop up employees and technology from smaller potential rivals. Software developers have given a name to what they describe as Apple’s behavior in such cases: sherlocking. The term refers to an episode about two decades ago, when Apple released a software product called “Sherlock” that helped users find files on its Mac computers and perform internet searches.
Joe Kiani, founder and CEO of biotechnology company Masimo, in Irvine, Calif. He said that after meeting with him about his company’s blood-oxygen measurement technology, Apple began hiring Masimo employees. Apple later launched a watch that could measure blood oxygen levels. Photo: Philip Cheung For The Wall Street Journal
After an outside company built a tool that had a few more capabilities, which it called “Watson,” Apple released an updated version of Sherlock with many of the same features. According to the engineer who built Watson, which he subsequently sold, Apple co-founder Steve Jobs personally called him to defend the move.
Companies that allege Apple copied them fight back in two ways: complaining publicly to get attention from regulators interested in Apple’s market power, or filing lawsuits against Apple.
App developer Blix Inc. has alleged that Apple stole its technique for anonymizing email addresses during online service sign-ups when the company launched its “Sign in with Apple” feature in 2019. Tile Inc., the maker of object-tracking devices that once integrated seamlessly with the iPhone, has faced off against Apple after the company launched a similar product called the AirTag in 2021.
Currently, the Justice Department is investigating whether Apple favors its own products over those of third-party developers such as Tile, according to people familiar with the matter.
Many of the patent battles focus on technology that smaller companies said they developed and that Apple appropriated for its watch. Apple said many of the patent violation claims it faces are based on rivals’ patents that are overly broad.
“The truth, is these companies are blatantly copying our products or stifling competition by using invalid patents,” an Apple spokeswoman said. “We will continue to fight these baseless claims in court and to advance technologies on behalf of our customers and public health.”
Since its founding, Apple has had a reputation for innovation, and it spends enormous amounts of money developing its own technology. In its 2022 fiscal year ended in September, its research and development budget was $26 billion, up nearly 20% from the prior year.
Under Chief Executive Tim Cook, Apple has tried to boost profit margins and differentiate its products by designing more components of its products in-house. It sometimes uses acquisitions to gain access to technology, and typically avoids licensing agreements with smaller operators, according to executives and patent lawyers who have attempted to strike deals with Apple.
Apple said it pays licensing fees to many companies of different sizes. The spokeswoman said it has licensed more than 25,000 patents from smaller companies over the past three years.
A model of a pulse oximetry sensor in Masimo’s lab. Photo: Philip Cheung For The Wall Street Journal
In 2016, a company called AliveCor Inc. announced a watchband accessory for conducting electrocardiograms that could pair with the newly released Apple Watch. Before the product launched, AliveCor founder David Albert was invited to Apple’s Cupertino, Calif., headquarters, where he met for 45 minutes with Apple Chief Operating Officer Jeff Williams, the leader of Apple’s healthcare initiatives.
Mr. Albert said he placed a prototype of the device onto Mr. Williams’s wrist and checked his heart rhythm. Mr. Albert recalled Mr. Williams telling him: “We’d like to find a way to work with you, but we might compete with you.” Mr. Williams didn’t respond to requests for comment.
In 2017, AliveCor became the first medical-device accessory for the Apple Watch approved by the Food and Drug Administration.
In 2018, Apple released its Series 4 watch, which could perform an electrocardiogram without the AliveCor accessory. Around that time, Apple changed its operating system in a way that AliveCor’s hardware and software integration no longer worked with the watch. A year later, AliveCor stopped selling the Apple Watch accessory.
The Apple spokeswoman said the company had been developing its own electrocardiogram for the Apple Watch since 2012, three years before it launched the watch.
In 2021, AliveCor filed a patent-infringement complaint before the International Trade Commission, a federal agency that investigates unfair trade practices, alleging that Apple violated three of its patents. In December, the commission ruled in favor of AliveCor, barring imports into the U.S. of all Apple Watches with the heart-sensing capabilities.
Separately, Apple took the dispute to the Patent Trial and Appeal Board system, which was set up to invalidate bad patents and help companies defend themselves against so-called patent trolls, litigants who file broad patents on numerous technologies without producing any real products. That board invalidated the AliveCor patents under dispute, thereby nullifying the import ban. AliveCor has appealed that ruling.
Apple also has sought to invalidate seven other AliveCor patents, AliveCor said.
Apple said that when it files multiple patent board petitions against companies, all stem from the original dispute.
AliveCor board chairman and investor Vinod Khosla, a venture capitalist, said he now steers companies in which he has invested away from having any talks with Apple. “Apple will talk to everybody and then try to steal the best people who are developing the technology,” he said.
Apple said that it doesn’t steal technology and that it respects the intellectual property of other companies. Above, Apple employees at an event last fall. Photo: CARLOS BARRIA/REUTERS
Since 2012, Apple has attempted to invalidate more patent claims before the Patent Trial and Appeal Board than any other petitioner, according to intellectual-property research firm Patexia.
Executives and lawyers involved in such cases said it can cost about a half-million dollars to defend against each petition, a high cost for small tech companies.
Apple said it is selective in how it uses the patent system, and that it isn’t its intent to drown any company in legal filings. The Apple spokeswoman said its use of the petition system is consistent with other similarly situated companies.
Andrei Iancu, who stepped down as director of the U.S. Patent and Trademark Office in 2021, said the patent system “is tilted in favor of the largest established firms. This is not a coincidence. It’s the result of decades of policy push to implement a variety of policies in the patent system that make it increasingly more difficult to enforce patents.”
A spokesman for the Patent and Trademark Office said it is committed to a patent system that serves all U.S. innovators, and that the appeal board is a “neutral judicial body.”
In 2013, an Apple manager contacted Valencell Inc. about a possible partnership, according to Valencell. The Raleigh, N.C.-based company had developed sensing technology that enables heart-rate monitoring when a user is moving, such as when running. Such technology is essential for wearable devices that track exercise and health data.
In the ensuing discussions, Apple repeatedly sought information from Valencell about its technology and discussed the possibility of licensing it, testing a prototype for several months, according to Valencell. Soon before the Apple Watch launched in 2015 with its own heart-monitoring feature, Apple ended discussions with Valencell.
The following year, Valencell sued Apple in federal district court in North Carolina, alleging that Apple had violated four of its patents. Apple filed petitions to the patent appeal board seeking to invalidate those four patents. Apple also filed another seven petitions against Valencell patents in areas unrelated to the initial case, according to Valencell.
Valencell President Steven LeBoeuf said the company, weary of fighting Apple, settled out of court with it in 2019. He declined to disclose the terms of the settlement.
An employee works on sensor technology at the Masimo offices. Photo: Philip Cheung For The Wall Street Journal
Masimo, the developer of blood-oxygen measurement devices, unveiled the mobile pulse oximeter designed to work as an accessory with Apple devices at a trade show in 2013.
Adrian Perica, then head of mergers and acquisitions at Apple, told Masimo executives in an email, seen by The Wall Street Journal, that Apple wanted to “dig deep” into Masimo’s technology and what the company had coming next. “Let’s discuss any ideas you have about how Apple could or should integrate some [of] these technologies in our products,” Mr. Perica wrote.
A few months later, Mr. Kiani said, he got a call from his chief medical officer, Michael O’Reilly, informing him he was joining Apple, which he said had agreed to double his salary and pay him millions in Apple shares. Mr. O’Reilly didn’t respond to requests for comment.
Mr. Kiani said Apple urged him not to worry about the hiring of Mr. O’Reilly, and the two companies continued to talk about potential plans. Apple went on to hire 30 of Masimo’s employees, he said.
In 2014, Apple hired Marcelo Lamego, a former Masimo employee who was chief technical officer at Masimo spinoff company Cercacor Laboratories Inc., which licenses Masimo’s technology. In an email before he was hired, Mr. Lamego told Mr. Cook that he could “add a significant value” to Apple “without conflicting with the large IP I have developed for Masimo and Cercacor during the same period.” Mr. Lamego didn’t respond to requests for comment.
In 2020, Masimo sued Apple in federal district court in Southern California, accusing the company of gaining access to proprietary information by hiring Masimo employees. The trial is under way now.
In 2021, Masimo also filed a patent-infringement complaint against Apple before the International Trade Commission. In January, a commission judge issued an initial finding that some models of the Apple Watch had infringed on one of Masimo’s patents. The investigation is expected to be completed by next month.
According to Mr. Kiani, Apple continues to try to hire Masimo employees. Apple said it doesn’t specifically target Masimo employees.
Mr. Kiani said that Masimo has so far spent $55 million on its lawsuits against Apple and defending its patents before the appeal board. He estimates it is likely to cost his company more than $100 million in the end.
Apple didn’t answer my questions about what visibility it has into what apps do with this data, or how it plans to vet them. On a website for Vision Pro developers, Apple warns, “It’s your responsibility to protect any data your app collects, and to use it in responsible and privacy-preserving ways.” So users just have to trust them?
Other privacy researchers say the risks are even higher that devices like the Vision Pro expose a stream of data about the one thing we can’t change: our bodies.
Information about how you’re moving and what you’re looking at “can give significant insights not only to the person’s unique identification, but also their emotions, their characteristics, their behaviors and their desires in a way that we have not been able to before,” says Jameson Spivak, a senior policy analyst at the Future of Privacy Forum.
Apple has addressed the privacy around one extra-sensitive organ: your eyeballs. The Vision Pro tracks your eyes so you can select things with your gaze like you might move a mouse on a computer. But Apple says it doesn’t share where users look with apps, websites or even itself. Instead, the device only reports what you’ve selected with your gaze after you tap your fingers together, the Vision Pro equivalent of a mouse click.
This is a solid place to start. But what about the rest of the body? Developers tell me apps can get access to a stream of data about users’ movement, right down to the wiggle of a finger.
Researchers at the University of California at Berkeley blew my mind when they explained just how revealing data about how your body moves while dancing could be.
Last year, they discovered they could uniquely and consistently identify about 55,000 different VR users based solely on data about the movement of their head and hands. It’s as useful as a fingerprint, maybe more.
And in another study, they used head and hand motion from a game to guess some 40 different personal attributes of people, ranging from age and gender to substance use and disability status.
What’s to stop Vision Pro apps from doing the same? “In cases where that motion data is being streamed to the cloud … even Apple has very little visibility into what is happening to it after it leaves the device,” said one of the researchers, Vivek Nair. “Because this data can’t be entirely eliminated from most applications, our suggestion would be to develop a privacy-preserving tool for VR motion data.”
I asked Apple what it was doing to protect this kind of data. Its response: Crickets.
Mixed-reality devices are “very exciting with huge potential,” says Berkeley computer science Professor James O’Brien. “But I also think that privacy considerations need to be primary design criteria, not afterthought.”
