The Ultimate XR Device - A Dream? Part 2
In Part 1 of this series I looked at Form Factor and three aspects of User Experience (User Interface, Visual Interface and Audio Interface). In the second part of this series I want to look at more aspects of my dream, these include:
Participation in a Multiple Modality Ecosystem
The Role of Security using AI and Cloud Services
Sharing XR Devices
Participation in a Multiple Modality Ecosystem
My dream XR Device must be part of a Multiple Modality Ecosystem. To integrate XR devices into a multi-modal ecosystem alongside laptops, desktops, and mobile devices, it is essential to leverage their distinctive capabilities to augment and harmonise with the functions of these existing devices. Achieving this requires a strategic approach to software development that is hardware-agnostic, ensuring that as XR hardware evolves, the software can adapt to utilise the new capabilities. This necessitates an Endpoint Agnostic approach in developing immersive experiences accessible across all devices, thus bridging the gap between XR devices and traditional computing platforms.
When I talk about XR and Metaverse technologies, I often refer to the Movie “Minority Report”. Most people remember the scenes with Tom Cruise using a gesture based interface to facilitate preventing a crime. That Movie, predicted a lot more! It has been well over 20 years since that Movie was released and in 2017, 15 years after the Movie, CNN looked at the technology in the story retrospectively. Take a look at the video below that shows not only how much of the forecasted technologies are available today in some form or another; but also the key of a devices participation in a Multiple Modality Ecosystem:
The video emphasises our need for the flexibility to begin a movie at home, then seamlessly transition to a mobile device while en route to the airport. Aboard the plane, an XR device allows us to enjoy the film on a virtual large screen, and upon reaching the hotel, we can either finish watching on a laptop or cast it to the hotel room's TV. Now, let's consider this experience from an enterprise perspective:
Cross-Device Collaboration/Integration/Synchronisation: XR devices should be designed to operate in tandem with laptops, desktops, and mobile devices, facilitating a unified workflow. For instance, you might initiate a project on your laptop, proceed to examine it in 3D through an XR headset, and then refine it on your mobile device. Such inter-device connectivity bolsters both productivity and creative processes. I must be able to access all my data from whichever device I use, and be able not only to read it, but create data and update it too.
Spatial Collaboration and Meetings: XR devices must enable virtual meetings with participants connecting through various devices. Some may join using laptops or mobile devices, while others might opt for XR headsets, offering a more immersive experience. Such a multimodal setup can enhance remote collaboration, making it more interactive and productive. The system should facilitate content sharing and collaborative creation. Additionally, it must allow me to share my data with collaborators while providing the necessary security controls. These controls should be granular yet simple to manage.
Enhanced Visualisation: XR devices offer advanced visualisation capabilities beyond what traditional screens can achieve. For example, architects and designers must be able to utilise XR headsets to navigate virtual models of their projects while continuing to use their desktops for intricate design tasks.
Data Integration and Interaction: XR devices should be able to gather and synchronise data from various sources, such as desktops and mobile devices, to create an immersive and interactive environment. For example, a healthcare professional could use an XR headset to visualise patient data from a desktop application in a three-dimensional space, making it easier to understand complex information.
Content Creation and Consumption: XR devices must be able to enable the creation and consumption of content in ways that traditional devices cannot. For instance, a user might create a 3D model using a desktop application and subsequently use an XR headset to interact with and enhance the model within a virtual environment. For example OnShape from PTC facilitates this using the Apple Vision Pro, and includes remote immersive collaboration capabilities on such 3D models. See the video below:
In my dream world such capabilities should be ubiquitous across applications, allowing content interaction and co-creation in almost any application. This will revolutionise remote collaboration allowing teams to work together much more effectively. This needs to go beyond “screen sharing” it must enable “experience sharing” and true co-creation.
Spatial Computing: XR devices must facilitate spatial computing by overlaying digital information onto the physical world. This is discussed further in my previous use case posts, especially Situational Awareness and XR and the (Industrial) Metaverse, Now or Later? however the users of desktops and mobile devices should still be able to access the data and collaboration sessions all be it with a different experience.
By integrating these various modalities, XR devices can create a more dynamic and interactive ecosystem that leverages the strengths of each device type.
Achieving a natural, reliable, and effective user experience in a Multiple Modality Ecosystem requires data availability, synchronisation, connectivity, and advanced AI. Security is a crucial component of my dream; my ideal XR device will access valuable data and integrate deeply into my life, necessitating robust security measures. Furthermore, to achieve the dream form factor outlined in Part 1 of my article, it must utilise cloud services securely.
The Role of Security using AI and Cloud Services
Security is crucial when considering XR devices and their use of AI for several reasons:
Data Privacy
Preventing Unauthorised Access
Integrity of AI Models
User Trust
Compliancy and Regulations
Data Privacy: XR devices and AI collect and process a vast array of personal data, including biometric information, location data, and user interactions. It is imperative to handle and store this data securely to protect user privacy.
Preventing Unauthorised Access: XR devices can act as gateways to sensitive environments, both virtual and real. Unauthorised access could lead to data breaches or even physical security risks, such as when an AR device is used to control actual machinery for example.
Integrity of AI Models: AI models used in XR applications require protection against tampering. If an AI model is compromised, it can lead to incorrect or harmful outputs, adversely affecting the user experience and potentially causing harm.
User Trust: The successful widespread adoption of XR and AI technologies is heavily dependent on users' trust in the security of their data and interactions. Significant security breaches have the potential to greatly reduce this trust, thereby hindering the integration and usage of these technologies.
Compliance with Regulations: A multitude of regulations and standards have been established to control data security and privacy. It is essential for XR and AI applications to comply with these regulations to avoid legal complications and maintain user trust.
If we look at how Apple have tacked security with respect to Apple Intelligence they have taken several measures to ensure security and privacy when using using it. That being said, Apple have not as yet launched Apple Intelligence for the Vision Pro but I am assuming the feature will reach that platform and will share the same architectural design.
On-Device Processing: Apple emphasises processing data on your device whenever feasible, ensuring that your information doesn't leave your device and minimising the risk of exposure.
Private Cloud Compute (PCC): For tasks necessitating cloud processing, Apple employs a system known as Private Cloud Compute (PCC). This infrastructure guarantees that your data is encrypted and processed with security. Additionally, it removes data post-request completion and conceals IP addresses to safeguard user information. Additionally:
PCC enhances the security and privacy capabilities of Apple devices by extending them to the cloud. Consequently, personal data transmitted to PCC remains inaccessible to anyone but the user, including Apple itself.
Apple has developed the complete PCC stack in-house, featuring custom hardware and a robust operating system tailored for privacy.
When a user submits a request to AI, the PCC processes the data in the cloud. This data is encrypted during transmission and is neither stored nor retained once the request has been completed.
End-to-End Encryption: The connection between your device and Apple's servers is secured with end-to-end encryption, guaranteeing that your data remains protected during transit.
Transparency and Verification: Apple permits independent security specialists to examine the code operating on its servers. Such transparency aids in confirming that privacy and security protocols are correctly executed.
User Consent: For functionalities that incorporate third-party services, such as integration with OpenAI's ChatGPT, Apple mandates user consent and upholds stringent privacy protocols.
Security is also critical when considering how XR devices utilise local and cloud data, as well as local and cloud-based processing. Some of the reasons for this are outlined below:
Data Privacy and Protection
Integrity of Data and Processing
Access Control and Identity Management
Compliance with Regulations
Resilience Against Cyber Attacks
User Trust and Adoption
Data Privacy and Protection: XR devices collect sensitive information, including personal, biometric, and interaction data. This data must be protected against unauthorised access and breaches, whether it is stored on the device or in the cloud, to ensure the privacy of the user.
Integrity of Data and Processing: Ensuring data integrity during storage and processing is crucial. Data that is compromised or altered can lead to incorrect outputs, negatively affecting the user experience and potentially causing harm.
Access Control and Identity Management: Both local and cloud-based systems necessitate robust access control mechanisms to prevent unauthorised access and to customise user permissions. This is especially critical in cloud environments, where data is accessible over the internet.
Compliance with Regulations: Various regions have distinct regulations regarding data storage and processing. Adhering to these regulations is essential to avoid legal issues and maintain user trust.
Resilience Against Cyber Attacks: XR devices and applications, especially those connected to the cloud, are susceptible to cyber-attacks. Implementing strong security protocols and architectures is essential to protect against these threats, ensuring the system's reliability and security.
User Trust and Adoption: The successful widespread adoption of XR technologies is heavily dependent on user trust in data security. Any security breaches can rapidly erode this trust, thus hindering the assimilation of these technologies into mainstream use.
A primary factor in the failure of Google Glass was the potential of privacy and security issues. To avoid repeating this, the industry needs to tackle these concerns by establishing a robust security framework. Utilising Zero Trust models can enable developers to create more reliable and secure XR experiences. Moreover, combining AI with a mix of local and cloud computing architectures can significantly enhance the user experience.
To support XR devices in a way that users and enterprises can trust, a robust security and identity management architecture should include the following key features:
Strong Authentication and Authorisation
Data Encryption
Privacy Controls
Device and Network Security
Monitoring and Incident Response
Compliance and Governance
Strong Authentication and Authorisation:
Multi-Factor Authentication (MFA): Ensure that access to XR devices and data necessitates multiple forms of verification.
Zero Trust Security: A cybersecurity framework is based on the principle of "never trust, always verify." It means that by default, no user or device, whether inside or outside the organisation's network, is considered trustworthy. Each access request must be authenticated, authorised, and continuously validated before granting access to applications and data.
Data Encryption:
End-to-End Encryption: Protects data in transit and at rest to prevent unauthorised access.
Secure Storage Solutions: Ensures that sensitive data is stored securely on devices and in the cloud.
Privacy Controls:
User Consent Management: Ensures that users are aware of and consent to data collection and usage.
Anonymisation and Pseudonymisation: Protects user identities by anonymising or pseudonymising data.
Device and Network Security:
Secure Boot and Firmware/Software Updates: Ensures that devices boot securely and receive regular security updates.
Network Segmentation: Isolates XR devices from other network segments to limit the impact of potential breach.
Monitoring and Incident Response:
Real-Time Monitoring: Continuously monitors for suspicious activities and potential threat.
Incident Response Plans: Prepares for quick and effective responses to security incidents.
Compliance and Governance:
Adherence to Standards and Regulations: Ensures compliance with relevant laws and industry standard.
Regular Audits and Assessments: Conducts regular security audits and assessments to identify and mitigate risks.
When looking at Apple’s implementation of iCloud, they employ several robust measures to ensure data security :
Encryption: iCloud data is encrypted during transit and while at rest, ensuring that your data is secure as it is transmitted to and from iCloud servers, as well as when it resides on the servers.
End-to-End Encryption: Apple employs end-to-end encryption for specific categories of sensitive data, including Health data, iCloud Keychain passwords, and Home data. This encryption guarantees that only you have access to this information, and even Apple cannot decrypt it.
Two-Factor Authentication (2FA): Apple mandates two-factor authentication (2FA) for all new Apple IDs and for accessing various features within its ecosystem. This provides an additional security measure by necessitating a second form of verification beyond your password.
Advanced Data Protection: This optional setting provides the highest level of cloud data security. Once activated, it ensures that the bulk of your iCloud data, such as iCloud Backup, Photos, and Notes, is secured with end-to-end encryption.
Security Keys: Users have the option to enhance their Apple ID security by utilising physical security keys, providing an additional safeguard against unauthorised access.
iMessage Contact Key Verification: This feature enables users to confirm that they are communicating exclusively with their intended contacts, adding an extra layer of security for sensitive communications.
Apple examples are given above, however other vendors such as Microsoft, Amazon etc have similar security features.
These measures collectively ensure that your data remains secure and private, even in the event of a data breach.
Implementing these features can help build a secure and trustworthy environment for XR devices, protecting both users and enterprises from potential threats.
Sharing XR Devices
In my previous article My Apple Vision Pro Journey, Part 4 : Where do we go from here? I discussed that one of my biggest criticisms of the Apple Vision Pro, was the fact that it has been designed to be a highly personal device, one that cannot easily be shared. Apple haver stated many reasons for this, however in my opinion there is no unsolvable roadblock to sharing the device, all the reasons Apple has given for making a personal device has a solution, either in the hardware design, or software feature design. MacOS is a multi-user OS, however iPadOS and iOS Where OSX was the mother of all these OS’s have had this feature removed. In my dream XR device, this must change.
Unless the price of my dream XR device is so low, that everyone having one is not an issue, the devices need to be shareable. If this means having a stock of accessories like lenses, light seals and headbands that are personal, then so be it; however in my opinion, this is a problem some vendors do not want to solve. Android has multi-user capabilities, Apple OS’s should have them too; their desktop OS’s do!
Summary and Conclusions (so far)
Participation in a Multiple Modality Ecosystem
It's essential that I have the flexibility to choose my device based not just on the task at hand, but also on my current situation. For instance, I mentioned watching a movie across different devices, but this flexibility should apply to all my activities and their respective applications. To achieve this with ease and fluidity, every software application I use needs to be endpoint agnostic. This means that each application should operate on any device I choose, leveraging the distinct features of each, while also accommodating devices with limited capabilities. Moreover, the data managed by these applications should be stored in the cloud, with synchronisation as necessary across my devices, wherever my identity will be authenticated.
My dream XR device would enable me to run all the necessary apps and maintain my data accessible and synchronised across my devices. When collaborating with others, the others must be able to utilise as many application features as their device permits, ensuring that, at a minimum, they can partake in my XR experience to the fullest extent possible, or at least view what I'm seeing.
The Role of Security using AI and Cloud Services
Security by design is essential for both XR hardware and the associated software and services. For a device to gain traction, it's crucial that organisations and consumers trust that their interactions, and most importantly, their data, are safeguarded against cyber threats. My ideal XR device would embody this principle, integrating into an ecosystem that allows users to manage their security settings with ease and precision.
Sharing XR Devices
The ideal scenario is for my dream XR device to be shareable, or priced in such a way that sharing is no longer required. Currently, there are no technical barriers preventing vendors from enabling device sharing, yet there are numerous obstacles to creating a device affordable enough to make sharing a non-issue.