We call it by many names; VR, AR and XR, but I think it will soon be referred to by the general public simply as putting on a headset. This is similar to how we used to work ‘with hypertext systems’ but now people just ‘go online’ and ‘click on links’.
I am a firm believer that in the coming years our work style will include most of using headsets for at least part of the day, similar to how we might work on a smartphone, laptop and desktop, and even with our watches, as part of our workday. I don’t think the headset will take over, but it will definitely become a useful part of our work.
Since this way of working offers much greater opportunities for information presentation, my own thinking is that this will be the ‘native’ information environment for many people and all the traditional media will be thought of as limited access points.
We MUST Dream NOW
Doug Engelbart was once referred to as a ‘dreamer’. He replied that “dreaming is hard work”. With 2D displays, what digital text can be, seems set in stone–most people think they know what a word processor is and what a web browser is and so on. This caused much of the work of the pioneers, including Doug, to be ignored since we “now know” what text is, so why look deeper? This has caused a huge loss of capability for how we can interact with our knowledge.
With the possibilities VR, AR and XR offers, we have the chance to stir our imaginations and dream again.
This is not an opportunity we can afford to waste.
Text on traditional two dimensional displays became ossified once ‘we’ thought we knew what it was and innovation slowed to crawl. We cannot afford to let this point in time–the final time before working in XR becomes normal in history–to go by without investing in Doug Engelbart level dreaming to really explore the possibilities offered.
We MUST Experiment & Experience NOW
We have experimented aspects of text in VR in the Lab, with VR experiences available for you to try for Basic Reading in VR and Text in VR in general. I feel it’s important to experiment in order to experience new interactions rather than keeping them abstract.
Work in XR
Below is my perspective of how I see XR coming into our lives initially (virtual displays!) and how we can go beyond the frame to really take advantage of this new environment (metadata!), as well as a few observations from early use.
Virtual Displays, the initial ‘killer app’
I further believe that the initial ‘killer app’ for headset use, whether in VR or AR mode, will be for virtual displays, and I expect Apple will provide a solid solution for this.
At the time of writing, early 2023, Meta’s Quest Pro features both third party virtual screen apps, such as ‘Immersed’, and Meta’s own virtual display (‘Remote Display’ Beta). Immersed provides better quality display I feel but it is quite finicky, where you really need hand tracking to set it up but then if you have hand tracking enabled when working, you have laser beams coming out of your hands. Meta’s own virtual desktop is quick and easy to set up but not very reliable (for me, at home and in cafe’s at least) or flexible. For example adding additional displays is not clear how to do and resizing does not scale to the screen so it’s easy to get odd black lines on the sides of the display.
Once Apple releases their (reputedly named) ‘realityOS’ headset, I expect the solid default will be that when you put it on, your ‘Apple World’ will just be there, whether you primarily use macOS or iOS.
If you sit with your laptop open in front of you, picture the screen dimming to black and the virtual display taking over. You can now use your hand to ‘virtually’ grab the top of the screen to pull the screen into a tall portrait mode if you prefer, to work on a long document, or pull the sides to have a large monitor for many documents or one giant movie screen. The screen will be instantly malleable to your preference.
Spaces beyond ‘displays’
Once you are in this environment and go beyond the confines of your (very useful) virtual screens and start taking advantage of the additional digital workspace you have, you need to define ‘what’s what’ and ‘where’s what’.
Picture opening a document and taking it our of the screen, so that it is free in the space in front of you, not tied to the flat world of the screen…
Virtual Architecture & Metadata
Virtuality architecture refers to architecture of the virtual realm, as opposed to using CAD for architecture to be built.
- In physical space the ‘affordances’ of what something is, is quite clear; a desk is something to put something on, a wall is something you cannot walk through, a switch is something you can flip, and so on.
- In a digital flat space, such as the interface we use on a computer monitors, everything must be defined and understood by the system to have an interactive function. You can draw a button in Photoshop but it won’t do anything when you click on it until you move it into Xcode and assigned programatic functions to it. Interactions in such an environment may not be immediately visible. This is because something which can be interacted with and something which cannot be interacted with may appear to be exactly the same, such as text. A PDF with Visual-Meta in Adobe’s Acrobat Reader, in Apple’s Preview our in my Reader will have very different interaction possibilities.
- In a digital inhabited space (such as in VR/AR or even games viewed on a traditional screen), the environment becomes further layered and everything needs to be able to tell everything else what it is if there is to be a chance of integration. Otherwise you do not know if you can walk through a wall or climb a hill. Or anything else with any certainty. This is why Apple had ‘evangelists’ whose job it was to promote unity of the user experience of the application developers. When anything can do anything, we need to decide on standards, though not hold the standards too tight, expand them when it is useful to do so.
This issue of ‘what’s what’ is significant and such context is not only nice or useful, it is core and fundamental to what these space can be for us and what affordances then can offer.
The context of what something is, is data about the data, hence we use the term ‘metadata’. In inhabited spaces, as opposed to spaces rendered purely for non-interactive entertainment such as a movie, there is two forms of metadata:
- metadata about the environment
- metadata about what you bring to the environment
Let us return to the document you pulled out of the monitor. You are now holding it as a 3D object in space and you can place it anywhere, as though you are in a zero gravity environment and read. This is nice.
Assigned Spaces
Imagine further that you perform an action, such as putting your finger on a picture and flipping the picture out of the document and into space. You then put it on a wall you have assigned as a ‘picture wall’. The second part of this is crucial-if you assign a wall, in your VR or AR environment as being a ‘picture wall’ you can then close the document, go to another physical, or virtual, location and when you open the document you can expect the picture to be on your ‘picture wall’ in that location as well, even though in this other location you have the picture wall in a different part of the space.
The question is: When you take something into a virtual environment, where is it? And furthermore where is it in terms of-what is it placed relative to?
Is it in a location relative to the document you opened? If so, what if you move the document around in the space, should all the elements you have pulled out (pictures, graphs, outlines etc.) follow? Or are the elements now moved from the document to the space? And of so, to the space ‘as a whole’ or to a defined region? If the elements are in the space as a whole, then what happens when you open the document in a different environment, will you need to pull out the elements again or will they appear according to the coordinates of the original space?
This also matters in a 2D digital environment. If you open a document on one computer and choose a certain place for it on the screen, then maybe take a few screenshots and so on and place them somewhere useful, when you go to another computer and open the document none of this placement or extra information follows you around. In 2D this is a known limitation and we don’t tend to think of it as a great big issue. In inhabited 3D however, you will want to use the extended space you have available and although you could work in the same VR environment wherever you are, the real-world environment where you use AR will necessarily change. A simple analogy can also be folders on your desktop. If you don’t put something into a folder (aka an assigned space), you won’t take it with you if you move the folder.
Assigned Spaces in Practice
This is why it will be powerfully useful to assign different functions to different spaces and to have a way through which you can use them across your environments.
Imagine having that picture wall and also having a large concept map, such as we are building in Author, using a very large wall. Imagine further assigning areas for people to appear when they visit you, as flat video or 3D avatars.
Every time you open a document and flick the images, they will appear in a space you find useful, reliably.
You can of course choose to have spaces which are not tied to the document, so that you can drag an element from a document onto it, let’s say it’s a digital ‘cork board’ and the connection to the document snaps and it is now connected to the cork board and you can view the cork board no matter what documents you have open.
3D CSS
Hand-Crafted Layouts
There are fantastic, immersive, experiences already now being experimented with, including in our own Lab. However, hand-building experiences for each document is labour intensive and would likely defeat the purpose for most work. If the author of the document could easily and robustly assign the metadata for what the document contains, then the assigned spaces could be automatically filled to provide truly rich interactive experiences.
Automatic Layouts
If the document has something as simple as chapter markers then the environment can use this, and embedded images, in a virtuality layout where deeper information, such as the documents citations, are also known and visibly connected to sources, should you wish to see them.
User’s choice of Hand-Crafted or Automatic Layouts
A language authoring software can use to describe the elements in the document should also be able to contain suggestions for display, much like a choice of font or layout in CSS but for 3D, where the user can open the document and it will appear exactly as the author preferred. If the user prefers a different layout, this should also be easily available.
This is something Visual-Meta is being designed to accommodate.
In other words, think that flexibility comes from carefully considered constraints. This something we have experimented with in the Lab and this guides the direction of Author for XR.
Metadata matters enough make it robust and open, and to get the most value; able to contain rich information.
Further XR Observations
Open World, but seated
Further to the crucial aspect of XR to make rich metadata open and freely available (to developers, while the data stays secure for the user), I believe that, just as now, most knowledge work will be conducted seated or standing near a desk. For this reason I think that room scale VR will be interesting, but not for the general knowledge work, not for general work.
This means that motion sickness won’t be much of a problem.
Freedom of Movement, but Static
Interactions, such as we have experimented with in the Lab, where instead of using a joystick to walk, you pinch to get things closer, such as a massive wall mural, might look the same, but our brains do not interpret the visual scale change as motion, but as the mural being moved, since we intentionally pinch and move with our fingers. This was quite an interesting surprise.
For further perspectives on working in XR, please have a look at vol 3 of The Future of Text or join us in the Lab.
The ‘real’ world
Here I am looking a bit silly with the current ‘state of the art’, a Meta Quest Pro, at the Lavazza Cafe in London, February 2023. I don’t often work with a headset in public but it’s interesting that nobody–so far–has seen it as a novelty. Anyone I talk to know what it is and either use one for gaming themselves or know someone who does, though not so much for work. Yet.
