This week I saw two friends who are not in my industry and whom I don’t get an opportunity to talk about the work we are doing very often and they both asked about the benefit of putting on a headset. The questions were earnest and they simply could not see why anyone would want to put a ‘bucket’ in their heads. These two friends are intelligent and thoughtful so I owe it to them to answer as clearly as I can. This is for you Elisabeth and Asle.
Caveats & Context
I first pointed to the Apple Watch we all wore and–as an illustration of how technology moves fast–told them that the Watch has roughly the same processing capacity as a high end PC 5 years ago. I followed this up with the expectation that in maybe 10 years an AR headset could be as light as a regular pair of glasses today. Imagine a double tap on regular glasses and you have a full AR view of your work I said.
I then made it clear that I don’t expect anyone to work in full VR/AR any time soon, though of course when the devices are lightweight enough, like the glasses mentioned above, or when they have shrunk to contact lenses, then maybe, but that is still a ways off. At this point I think of them as ‘thinking caps’.
Similarly I pointed out that they already have made part of the point themselves, by not doing their main work on their iPhones, they use computers with large screens so that they can see more. This is of course a basic benefit of XR: More space.
Headset On
Once these caveats and long term perspectives were out of the way, I could start with the most basic use of VR which is to put yourself in a different environment from the one you are physically in. If you work in a crowded office or dorm, you can put on a headset and visually ‘be’ somewhere else. This can be as basic as reading a book in traditional form while sitting in a nice country cabin in front of a fireplace. Or in space, orbiting the Earth.
I find it useful to continue with the “reading of a document” as an example, particularly in a book called The Future of Text. I suggested to my friends that they may be reading an academic paper in AR–so they would remain in their environment–and instead of a single or two page view, they could view the document as an accordion folding out to show all the pages in a sequence, or perhaps as a large mosaic so that they could quickly move around the virtual ‘pages’. They could perhaps gesture to have all the images in the document move across to a physical wall in their room and the reference section could remain in view somewhere else. Citations could be instantly checked with a click, bringing forth the source document.
Interlude : Open & Rich Data & Metadata
These interactions are of course something which can be done in a traditional digital document (some of which we have done here with Reader augmented with Visual-Meta) and this is something which highlights the importance of the system having access to the data and metadata to enable. It’s as simple as this: A system can only let you interact with something to the degree of which it understands what it is. If it understands something with semantic meaning that provides much richer opportunity or interaction than if something is understood simply as a rectangle.
Dimensions of Thought
Beyond the flat document in XR it starts to get more interesting.
We can continue with a flat, traditional document as the base for a little longer however, and we can do some dreaming together. Suppose you are the user and you have been reading a document which you feel is a bit dense so you’d like a more expansive view so you put on your headset. Now, if the system is well built and open, you should be able to expect to have your document come with you into the XR view, but this will not be as possible if different companies own proprietary environments and the interconnections are badly thought out. Refer to the ‘Interlude’ above.
Let’s start instead with a scenario where you choose not to take the document you had on your laptop into XR. Let’s look at how you find/see/locate open a document instead of looking at how you read the document.
What might this process be like and what might it look like? This is where we can see questions opening up.
Your ‘Library’, to call it that, in XR, can be a 3D representation of a traditional, physical library, with extra features. It can be a list, just like you have on your desktop computer. It can be a graph with nodes for documents, themes, authors or institutions. It can be a landscape, it can be a timeline. The options are vast at this point in time, because we have not built and therefore not decided what it ‘truly’ is.
‘Real’ Library+
Imagine also looking at your physical shelf and since your headset has scanned all the book’s spines, you can ask the headset to show you where a specific book is, which will then be highlighted, or you can ask it to search inside the books and show you relationships. The ‘pass-through’ video which allows you to view the world can also analyse your world.
A Second Interlude : Ownership
If our documents (primarily for the sake of the scenario presented here, containing traditional academic information, with or without augmentations, and other media) are truly ‘data owned’ by us, the end user, we will be able to use any Library view we want, from any vendor, for any aspect of how we want to experience the data. If data formats become proprietary, we will have to use whatever the initial vendors choose to give us, and then make a real effort to try to export to use another.
If the formats are free and open, the user owns the data and can choose what software system to interact with it.
Library of the Mind
Let’s return to the Library and assume that economic and political battles have been won and data and metadata is open, robust, rich and owned by you, the end user.
What can your ‘Library of the Mind’ be like?
One thing you can expect is to be able to instantly change what it is, with a gesture, voice command or other. You can expect to be able to have the most traditional form of document representation and you can expect AI powered analysis and views. You can expect wild and crazy things like holding up two documents and telling your AI system to let them ‘fight it out’ in the XR environment. You can imagine relevant quotes from both documents being compared and references being analysed to see if they truly back up the assertions made.
Further beyond this we can imagine learning new ways to visually represent data and having the ability to change representations at will.
& so much more.
Walls as Views & Connectors
At some point we will be comfortable enough in XR to have headsets on for longer sessions and this is where putting information pieces around your space can become useful. Meta calls this ‘Augments’, which is an interesting term.
For example imagine your calendar on your physical wall being instantly able to toggle from a basic list of events to showing dynamically the weather in days ahead, including weather in other locations where you are planing to travel to. Imagine further that days gone past becomes easy access to pictures you have taken as well as transcripts from meetings. All of this clear and explicit to you, but hidden from visitors to your physical space and only showing what you would like it to show to visitors in your virtual space.
Payback in 2D
If you are thinking that much of this can be done in 2D on traditional computer screens, you are right of course, but innovation has slowed to a crawl for traditional computer systems partly because users no longer see them as spaces for innovation so therefore there is little demand.
One of the real benefits of investing in ‘dreaming in XR’ can be bring is exactly this; similar innovations–when they can be implemented–in traditional systems.
XR brings the hype we need to dream again.
3D expansions
Our Stone Age ancestors did not simply paint animals on cave walls. They drew symbols, most of which we are still trying to decipher. They also use the medium of rough cave walls to bring our movement when living flame from torches illuminated them. Humanity has been experimenting with eternal representations of thought for longer than we consider ourselves to be Homo Sapiens. And we have a long way to go.
I predict that text will remain a central medium for communication for a very, very long time, but it will become augmented with not only moving and still images and interactions as we are familiar with in games and on the Web, but also in entirely new ways.
Brandel Zachernuk created a ‘magic carpet’ of records from conversations. There is no text there when experiencing the system at scale, but text can be brought forth to present meaning and records. It’s a thoughtful exploration of how text can be presented at different scales.
Text, in the form of alphabetic script, has a tremendous range in expressiveness of thought and grammar connected in powerfully augmentive ways.
Text is also surprisingly non-visual, despite being a predominantly visual medium, giving other means of representation an opportunity to shine. Large amounts of text can be summarised into smaller amounts of text but they still need to be read, which takes effort and is therefore no glanceable.
Will the 3D nature of XR with near-infinite potential for re-presentation be able to augment text, without dumbing down, without resorting to only a cartoon presentation of knowledge? I hope so, I think so, but we will only find out if we dream about work in extended reality. Together.