A series of reports in The Information paint a detailed picture of the progression, policies and issues that Apple’s plan to develop a virtual, augmented or mixed reality headset has faced since the initiative took off back in 2015.
Referring to several people familiar with the product, including some who worked directly on it, the reports describe a testament contest about the direction of the device. The standoff was between Apple’s mixed reality product team (called the “Technology Development Group”) and famed Apple designer Jony Ive and his industrial design team. The report sheds light on Apple’s direction for the device, which Bloomberg recently reported is approaching launch.
They also claim that Apple CEO Tim Cook has been relatively dismissive of the product compared to others like the iPhone, and that the location of the technology development team in a separate office from Apple’s headquarters has been a source of trouble and frustration.
Sources in the information say that Apple’s mixed reality efforts began almost by accident when the company bought a German AR startup called Metaio to use some of its technology on Project Titan, its self-driving car project. Another important moment was when Apple hired the AR / VR project team leader, Mike Rockwell, away from Dolby Laboratories. From 2015, Rockwell built a team that included Metaio co-founder Peter Meier and Apple Watch manager Fletcher Rothkopf.
In 2016, several AR demos were shown to members of Apple’s board. In one, a small life-size triceratops grew in front of the eyes of board members. In another, a space was transformed into an immersive, green environment. But the board was not Rockwell and the company’s main barrier. According to The Information, it was Ive who oversaw both industrial design and human interface teams at Apple.
Ive and his crew argued against a VR headset because they believe that VR separates users from people and the world around them and that VR headsets look outdated. But the Technology Development Group got the backing of the industrial design team by presenting a concept: an outward-facing screen on the front of the headset that showed images of the wearer’s facial expressions and eyes to people around them. The wearer could see the people around them through an external camera feed.
Rockwell and his colleagues developed and released ARKit in 2017, an application development package that enabled developers to create AR apps for the iPhone and iPad using technologies and techniques that could later be customized into a headset.
First, Rockwell and the rest of the mixed reality team wanted the headset to be tied to a base station to provide maximum impressive graphics and performance, and some on the team imagined that it was primarily a tool for professionals and creatives to use by their desks. . But I did not like any of these ideas and wanted it to be a lifestyle product on the mass market that consumers could take with them on the go. Apple’s top management backed Ive’s plan, and Ive still played an active role in the headset’s development, although he now works with Apple as a consultant.
The decision to make the headset a standalone device reportedly caused significant headaches. For example, some felt that the most optimal way to make it work well on its own would have been to put multiple features on a single chip. But since the silicon work was already done, they had to find ways to combat the inherent latency of having multiple chips in the device to communicate with each other. They had also made software, provided the base station plan would continue.
Nevertheless, the device is moving into the final stages of development. Bloomberg claimed last week that an advanced version of the product was recently shown to Apple’s board, and that Apple has “accelerated” the development of the headset’s software, which is an iOS offshoot called rOS. (The R stands for “reality”).
The information reporting reveals many details about the upcoming headset. It would have a resolution of at least 4K for each eye, which the team believed was the absolute minimum for users not to perceive the image as pixelated, unlike most current consumer VR headsets. Its built-in processor will be closely related to the M2 processor expected to reach Macs and iPads in the coming months.
The headset would also have 14 cameras, some facing outwards and some facing inwards. It would allow users to see the outside world and allow people nearby to see a video representation of the user’s eyes. It would live track the user’s facial and body movements to be mapped to a 3D avatar (probably similar to the iPhone Memoji) that could be used to hold remote meetings and social gatherings with other headset carriers that are far away.
Due to the limited processing options of the M2 chip in a headset without a bound base station (the canceled base reportedly had an ultra-high-performance M1 Ultra), the avatars would be cartoonish. The sources of the information also say that more photorealistic avatars were tried when the base station was part of the plan, but the eerie valley was a problem.
Apple originally planned to launch the headset in 2019, but it now looks like it could be announced instead either later this year or in 2023. In addition, Apple plans to introduce more natural-looking AR glasses as a follow-up product, but that device may still be years away from shipping.