This aspect of our VR work includes, building, testing and delivering the means through which users can employ AI to augment their thinking in a visually and interactively rich way using VR, rather than simply outsource their thinking to AI. The system will be free, open and accessible for anyone to use as an individual user as well as to integrate into other systems.
Since AI is not a single ‘super brain’, we will need interactive processes for how to deal with the capabilities AI offers. This interaction will need to develop far beyond the typed text of chat interfaces and speech interfaces we have today. We will need to immerse ourselves in order to realise the full potential of AI. Simple AI interactions through voice and text will remain useful for some work but to deeply augment our thinking we need to employ more of what evolution has given us in terms of capabilities; not just looking at the small rectangles of current computer screens but using the full environment VR (incl. AR/XR) has to offer, looking and interacting with full rooms of interactive knowledge.
Problem to address
There are many questions posed by working in this new dimension, only some of which commercial operations will see the need to address, so the extended research community needs to come together from the perspective of the new knowledge worker.
A Presentation on this will be made at the ACM Hypertext ’23 Conference. A public preview video is available.
The substrate almost disappears
- What happens when the substrate almost disappears, will interactions get easier, or will it be more like how interactions in, for example, sport is today? Or will we require training wheels and if so, will they retard dreaming of what VR can be?
- The promise of working with knowledge in textual form in a VR space can seem like simply an opportunity since there will be considerably more space available to work with than is even possible on a large desktop display. A larger space also provides the opportunities for more mess. How might this be resolved with interactions?
New types of tools
- When we can reinvent tools in this entirely new environment: Which tools will be useful and how will we define ‘tool’?
- Will we be able to separate environment from tool and information from interaction?
- Where is something in VR and in relation to what?… Selection control & view control: Where will the equivalent of a ctrl-click be in VR? Gestures? Menus? Will they remain wild for long, or become standardised like ⌘ keys on the Mac? This is interesting, fun and important.
Symbol manipulation at scale
- What might symbol manipulation at scale be?
- What might symbol manipulation in detail be?
What environment will we be working in, what is where?
- What is a ‘wall’, is it something defined as a rectangle or semantically–can I call a wall my ‘graphs’ wall and another my ‘pictures’ wall?
- What software owns it?
- What is it positioned in relation to; my head, my desk, the room?
- What is a space, how do we move between spaces?
- How do we transfer information?
- What can be seen, interacted with and ‘taken’ in shared spaces?
Goals
- To build prototypes of environments of such information rich environments where someone will be able to explore the space of knowledge in new ways, far beyond what a visualisation can muster on a laptop sized screen, including fostering deep and wide dialog on the potential of this approach, in the Future of Text community, where the result of such dialog will be published.
- To provide infrastructures for anyone to take advantage of this capability in their own VR systems.
The aim is to augment not just simple following of logical trails but also to augment our intuition and how we see and make patterns, to go beyond how computer systems can inform us with current paradigms. It is easy to vaguely imagine such a system in a sci-fi setting with the idea of the user being inside a ‘sea’ of interactive knowledge but the reality is that there needs to be infrastructures developed to make this possible and to build it to experiment to experience what is truly useful.
Flow
1) Specify Corpus. In a Web based interface the user will be able to specify what corpus of text should be used. The input text will be from a collection of documents in HTML, .doc or PDF, as well as live sources in JSON.
The initial corpus to work on will include what we have produced as The Future of Text, including almost half a million words in our Journal (part of which are transcripts from presentations and Q&A’s) and two volumes of the book series The Future of Text.
2) Initial AI processing. The system will plug in to available AI services, initially OpenAI for pre-processing the data into formats suitable for further manipulations. This can include summaries, extractions of keywords, creation of timelines and listing text by specific authors or speakers (from transcripts), if known. This is key research, to process the raw input into useful forms for further analysis by the user, and to let the system to enable this be open to anyone to use, not only large companies who will focus on their own priorities. The future of text, the future of knowledge interaction, the future of AI is too important to leave to the tech companies alone.
3) Further User Prompts. Once done, the system will allow the user to perform further prompt based interactions on the text. This will likely be presented in a node based interface (such as what is used for colour grading in DaVinci), but a key research question is how to make this interaction as useful as possible. The user should not need to be a programmer to use this interface.
Any result from any stage of AI manipulation will remain tagged with what the source material was and what manipulation was used.
The product data will be presented via API using JSON for use in any visualisation environment, including what this research project aims to build.
4) VR interactions. When in VR the user should then be able to draw on this data for instant views to see information in powerful ways and to perform further AI queries on the data.
This will enable previously only dreamt of interactions such as ‘show me what else this person has said on this topic’ or ‘show how this persons view has changed over the last year’ as well as seeing timelines with multiple layers of knowledge, sentiment analysis from different perspectives and more.
This will also benefit a user outside of VR when using a large display or a projection, the amount of data which can be shown and interacted with will greatly increase.
Timeline & Funding
This is intended to be an ongoing project with an initial two year pilot phase.
Outcomes
Full reports as part of The Future of Text publications, including recorded dialog with the wider community as the project progresses.
Working infrastructures for ingesting, processing and presenting data, as outlined above.
Experience
We have built the well regarded software macOS Author and Reader and host weekly Lab dialog on VR.
Principal Investigator
The Principal Investigator is Frode Alexander Hegland, former teacher of the Year, mentee of Doug Engelbart and developer of Author & Reader.
Advisors
Chief advisors are Vint Cerf, co-inventor of the Internet, and Ismail Serageldin, former VP of the World Bank and Founder of the New Library of Alexandria in Egypt. They are augmented by a strong advisory team and Lab team, as outlined in About Us.
University Partner
AI for VR is a partnership between The Future of Text initiative and WAIS (Web And Internet Science) at Southampton with co-investigators professors Les Carr and David Millard, Frode Hegland’s PhD supervisors.
Summary
In summary, the inherent multidimensionality of text can finally be truly unleashed in a VR environment, augmenting us like never before, staving of a future where AI might only be used for outsourcing thought, giving us instead what we may one day think of as real ‘super powers’ of observation, reasoning and thought.
Edgar Kazu Ballard Hegland, my six year old son, whose future I am trying to augment. I can imagine a workspace much bigger than this, as big as his curiosity and as interactive as his thoughts.
Frode Alexander Hegland
Wimbledon
26 August 2023