The Key : The Potential of AI in XR

The advent of AI is of course a massive shift in human cognition, with many pros and cons. The issue around how we get results is indicating a reduction in how we, and especially the younger generations, explore knowledge*. There is less curiosity, more acceptance of AI presented results. 

XR however, is structurally about connections and relationships. In order for AI not to only deliver linear results, XR can really help the user think about the results, ‘grab’ onto it and see relationships, patterns, lacuna, contradictions and more. The demo video above is an example of that, where I have asked Claude to give me information about some of the people involved in the development of XR and then imported that into the Author environment. 

The spatial Map is designed to augment how people see, interact with, and create relationships, not simply consume narratives. This allows users to experience spatial presentations to allow the user to interact with the knowledge in a significantly more contextual form than following chatbot lines of dialogue.

The headsets today are not generally up to delivering this. The Apple Vision Pro used as proof of concept is both expensive and heavy, but I think it shows the future, where this power will be in smart glasses, opening up knowledge spaces to everyone. This is the time I am working to be ready for. There is a lot of research to do on usability and open data flow.

 

Deeper Issues

 

Open Format


The format goes back to what Vint Cerf wrote about in the editorial in ACM Communication, noting that Visual-Meta adds “an exploitable self-contained self-awareness within some of the objects in this universe and increases their enduring referenceability.”†

This is now being extended to include XR elements. The spatial data is part of a regular word processor document, as well as a published document, in any format, including PDF. This format is completely open and contains no new inventions to increase complexity, it uses academia BibTeX and basic notations for defined concepts, brought together with a plain layout X, Y, Z description to keep it open and accessible. Furthermore, I feel it’s important to point out that the spatialized knowledge layout is retained when the user goes back to a traditional device, to be integrated with an a flat way, and then expand into depth again in XR.


The XR Experience


The primary principle behind how I develop the interactions is not overwhelming the user. This is why no connective lines are shown until they are initiated by the user through the act of selecting a node (no other special action, just a part of selection). This goes further with the Focus mode where everything not selected or connected temporarily hides, allowing the user a clear view of what they are interested in. Beyond this, in a fully visually immersive environment a large amount of information can be presented easily so that requires that the user can interact with that data easily, to view further information and to lay out their space in whatever way is most useful for them. This entails a mass of interaction design choices, as you are so very well aware, and this is where I spend my time; polishing the interactions between the user and their knowledge.


The Near Future


What is possible today is so different from what was possible only a few years ago and, what I can see going forward is continued advances in weight, cost and performance of headsets to the point where they are only glasses. What will likely remain quite similar will be our ability to read and interact with textual knowledge in textual form as nodes in ‘knowledge sculpture’. Therefore I believe that much of what we can learn through experiments to really experience knowledge in XR today, will have lasting value. In many ways, I feel we are with spatial knowledge where the pioneers were with ‘framed’ text (text on screens) in the 1960s. What we ended up with from that era is the Graphical User Interface of today and I can see that the experimental interfaces of today will also stick around for a long time. For that reason, I think this is the right time to really experiment with such interactions, both with the aim of inspiring something effective and lasting, and also to open the window to as wide a range of new interactions as possible, so that we do not only end up with a single paradigm, as the GUI would come to be.


The Research


The research going forward is based on two years of Alfred P. Sloan Foundation supported work where different ways to be in XR were experimented with and two symposia and two books were produced. Going forward the focus will continue to be to learn what the real opportunities for spatial thinking can be, developing for different systems and producing open formats for sharing with an increasing community of developers and other participants.
This is why I approached you, to see if the new generation of XR, with actual shipping headsets (which will likely soon be lightweight glasses) can be different from the early days of VR where the hardware was confined to development labs.