Skip to content

‘The Future of Text in XR’

This initiative is based on the premise that we don’t know exactly what will happen when our knowledge environment is not constrained by the rectangle on our laptop but takes advantage of our entire field of view and is richly interactive. However, one thing is sure, it will be fundamental because when we see different we become different.

It is easy–and important–to criticise news technology and this has been done since the beginning of invention and is being done with XR now, with concerns one one hand about the headsets being uncomfortable and on the other hand that people will spend too much time in the virtual worlds, and more. There are many forums for this aspect of the future but this is not one. This forum is to engage in dialog and experiments to try to discover how powerful working in XR can be.

Imagine working on your laptop and you need more mental ‘space’ so you put your ‘Thinking Cap’ on for a while (think of the XR/VR/AR Headset as more like magic sunglasses, not ‘helmets’ as they are today) and your workspace unfolds before you…

This is knowledge unleashed:

  • Text documents spread out on your wall. You can use a wall in your office like a TV ‘murder wall’ where all the elements of what you are working on is connected and you can instantly reconfigure the views to suit whatever sparks your curiosity. Imagine further being able to tuck it away and work on another huge graph and then switch as you see fit.
  • Edit a document with all the elements around you for easy overviews, not limited to a small rectangular view.
  • Your notes are stored in dynamic memory palaces, connected and available.
  • A timeline is available at a gesture, to see your future calendar and your diary of past events, including recordings of meetings and photographs.
  • Real things–maps, engineering parts, buildings–will be possible to experience in entirely new ways.
  • Abstract information–citation trees in academia, graphs and charts–will be seen in a new light. Something as simple as a diagram takes on a new and more useful character when it can be instantly scaled and traversed.
  • A docuverse of millions of documents is available for you to fly through and see connections between.
  • A comfortable reading space of a single document can be called forth, to let you focus.
  • A billion data points clear as a sculpture and as interactive as clay where you can ‘literally’ unfold subspaces to access different sets of dimensions.
  • You no longer need to look down at your small laptop screen and develop neck and spine injuries which can affect your health over long term, you can look ahead and you can look up, at your world of information.

  

The Future of XR is Now

“We Shape Our Tools, and
Thereafter Our Tools Shape Us*”

The above scenarios are not far off in terms of functionality, though it will take a bit longer for the headsets to be more like sunglasses than what they are today. However, this is the time to deeply consider what working in XR can be like, since what we decide today will have repercussions for countless years.

   
When posted to ‘X’, the video above attracted over 10,000 views, indicating a strong community interest in the subject and our approach.

The shaping of what the personal computer could be was completed in the early 1980s with the graphical user interface, keyboard and mouse, icons to click and segmentation of capabilities into word processing, email, spreadsheets, web and not much more. Inventing new ways of using the personal computer today is difficult not only technically but because we think we know what the personal computer is, and “truth kills creativity”*.

With the urgent complex problems we are facing we need to invest in the means through which we think and communicate. Text is a component of this.

XR (VR/AR) is coming of age with Apple’s Vision Pro and this is an opportunity to think anew of how we can work with knowledge in primarily textual form, inspired by the new dimensions of an infinite canvas with unlimited interactions.

We are still at the crucial stage between when dreaming is still possible–we have not been constrained by implementations of working in XR–and we have access to experiment to experience what working in XR can be, both for our immediate needs and to spark curiosity in others and hopefully keep innovation going, without being prematurely stuck in a single XR paradigm, as we are with the personal computer.

We ask: How can we truly unleash text in an immersive environment, to augment how we think & communicate?

  

The Project

This is a project to work to realise the potential of work in XR (VR/AR) where the opportunity is to greatly increase our visual and mental space for work.

The primary challenges to working in XR as we understand them today are:

  • More opportunities also come with more mess when working in a larger space and therefore interactions will need to be developed to–literally–get to grips with the much larger amounts of visual information available.
  • Lack of metadata to support rich interactions in XR and connected with traditional environments.
  • Lack of imagination for the continued development of new interactions and modes of work.

The Actions needed to realise deal with the challenges and to realise the potential of working in XR are to:

  • Build systems to enable us to experiment to experience what actual work can be in XR.
  • Host and foster wide dialog to better understand the potential of work in XR.
  • Improve metadata infrastructures to support richer interactions with information in an open ad robust manner, connected with current workflows and practices

The Results Desired. Better understanding of text in space and how this can expand how we think:

  • Software built for visionOS integrated with software already built for macOS to explore text in space.
  • Hosted and recorded weekly Future Text Lab discussions on the topic.
  • Hosted Symposium on The Future of Text in XR and actively widening the community to include more perspectives.
  • Published volume of The Future of Text in XR.
  • And the ultimate result will be to fire the imagination of developers and users for what working with textual knowledge in XR can be, for generations to come, rather than settling for what the corporations which develop current XR headsets are focused on, which is almost exclusively entertainment and social.

Why now?

  • Apple is about to release the Vision Pro, which is leaps and bounds more powerful, comfortable and integrated into the user’s work environment than anything which has come before, though it comes at a high cost, it is reasonable to expect costs to reduce over time.
  • We are the last people in the history of our species which do not work at least part of the time in fully immersive environments and how we enter this environment–what mind sets and priorities we bring with us–will shape the environment for decades.
  • This is similar to how the awesome creativity of the computer pioneers of Doug Engelbart, Ted Nelson, Andy Van Dam coalesced into the desktop computer metaphor we all use today: Word processing, email, web and spreadsheets controlled primarily by mouse clicks in a graphical user interface.
  • If we do not take active effort we will be left with working in what the corporate decisions of Meta, Apple, Google and Microsoft deem most useful for them, not what research, development, community engagement and testing shows is most powerfully useful for the knowledge worker of today, tomorrow and generations to come.

Why us?

  • We have focused on the future of text for over a decade.
  • We have hosted over a decade of The Future of Text Symposium while also growing the range of people involved.
  • We have published three volumes of The Future of Text.
  • We have produced the macOS word processor Author and the PDF viewer Reader.
  • We host a weekly Future Text Lab which is open to anyone.

What are we already doing and will continue to do?

  • We host the community (Symposium, weekly Lab meetings and series of books)
  • We develop software for textual knowledge work in the form of Author and Reader for macOS.
  • We experiment in XR with the resources available to us.

What will change with funding?

  • We will be able to invest in coding for XR, initially building for the Apple Vision Pro which is the most advanced system available and hence the best sneak peek into the future. We have conducted basic XR Experiments already.
  • We will be able to promote the community to more diverse audiences.

How will we make this work known?

  • The system will be available on the Apple visionOS App Store.
  • The associated systems (Author and Reader) are already available.
  • The books and symposium is available for anyone and will be presented to university audiences.

The bottom Line:

  • If we truly value knowledge, we must also value how knowledge is created, stored, shared, accessed & interacted with.
  • If we truly value the future of knowledge, we must invest in it.

    

    Frode Hegland
    London September 2023

   

the future of text, the future of thought, is richly multidimensional

      

 

  

  Endnotes

*= “truth kills creativity” as a professor at Syracuse University once told me. I am afraid do not remember his name, nor can I find him online.