This metadata will at the same time be stored in the same way as highlighted text in the document is, allowing the user to perform Find commands in the Library for this metadata.
Parsing
When parsing for this metadata it is important to note that the regular Visual-Meta starts (when parsing from the last page of the document) with @{visual-meta-end} so when looking for temporary Visual-Meta it is important to look for @{ai- since that will be the last section of Visual-Meta AI appendices.
Of note, @{xr- will be used for XR data, which will not be developed now.
In use: macOS
The Reader software will make this metadata available for Find operations in the Library.
In use: WebXR
This metadata will be packaged, along with the regular Visual-Meta, into JSON for transmitting to a headset for use in WebXR.
Further Metadata
User can perform further Ask AI and Save Metadata functions and each time a new Visual-Meta will be added.
User is using their Library in macOS Reader (this is our test environment, any others should work soon too)
User puts on a headset to continue working in their Library.
The user accesses ‘Reader XR’ (through web page or app) and their Document Metadata and Library Metadata is downloaded.
The user can interact with their Library and choose to open any document for reading, changing layout of and annotating.
When done they can put down their headset and return to their traditional computer.
Data Flow:
The data in Reader is in the form of Visual-Meta at the end of all documents and when writing any new Visual-Meta to a document it is also saved in a Document Metadata database.
This Document Metadata database will be stored in the form of JSON.
All the user’s metadata will be syncing with a server.
When done, the Library Metadata is transmitted back to Reader macOS (or equivalent), as well as any changes to Document Metadata (such as annotations being added or removed).
Definitions:
‘Computer’ means desktop or laptop computer, in other words computer in the traditional sense.
‘Headset’ means VR/AR/XR headset running webXR Reader software.
‘Server’ means software which provides continual access to data, via HTTP.
’Storage’ means software which stored document metadata (in List format) documents and library context metadata on physical storage.
‘Document Metadata’ includes core visual-meta, highlighted text and AI and XR generated temporary Visual-Meta.
‘Documents’ means initially PDF but will support any format.
‘Context Metadata’ means data which describes a use and/or display of the collection of documents and their associated metadata. This can be a single headset in a single real or virtual environment where documents are assigned positions etc., and it can mean the same headset in a different virtual environment or Augmented Reality environment etc.
‘Reader XR’ is the software to be developed for XR headsets.
Bjorn’s notes
If we think of the data store as the authoritative source of user’s data, what we are faced with is essentially a replication problem. That is, whenever an application that uses data from the datastore starts up, it wants to know “what has happened since the last time I was connected to this store? What’s new, what’s gone, and possibly what has changed. Of course this can be refined to “what has happened with the subset of data that this application on this device is interested in”.
Change is difficult to deal with, which is why immutability is a great approach: rather than allow data records to change, you just add new versions of them. A given version of something is never allowed to change. This will ensure you keep any references to previous versions intact, while providing a mechanism for discovering that the version you have has been superseded by one or more newer versions. It also means replication is simple – you either have the document with id abc123 version 1 or you don’t. There is no need to compare it to anything to determine which version it is.
It makes sense to make the version part of the ID you use whenever referring to a piece of data. So for instance abc123@1, abc123@2 etc. (This is essentially how some software systems manage dependencies: you use both the unique name of the dependency and its version). You can also create shorthands whenever you are interacting with the user. For instance abc123@latest to instruct the system to “figure out what the greatest version number of abc123 is and then use that when we store the reference”.
When the Headset connects for the first time it will of course want to download all the metadata. On successive connects you want to know “what has changed since the last time” and then only download the changes. You could do this by timestamp, for instance. The response can either be all the metadata or just the IDs of the versioned metadata items (allowing the Headset to determine what it actually needs to download).
Also: Reader can announce itself on local network using Zero-configuration networking.
User is using their Library in macOS Reader (this is our test environment, any others should work soon too)
User puts on a headset to continue working in their Library.
The user accesses ‘Reader XR’ (through web page or app) and their Document Metadata and Library Metadata is downloaded.
The user can interact with their Library and choose to open any document for reading, changing layout of and annotating.
When done they can put down their headset and return to their traditional computer.
Data Flow:
The data in Reader is in the form of Visual-Meta at the end of all documents and when writing any new Visual-Meta to a document it is also saved in a Document Metadata database.
This Document Metadata database will be stored in the form of JSON.
All the user’s metadata will be syncing with a server.
When done, the Library Metadata is transmitted back to Reader macOS (or equivalent), as well as any changes to Document Metadata (such as annotations being added or removed).
Definitions:
‘Computer’ means desktop or laptop computer, in other words computer in the traditional sense.
‘Headset’ means VR/AR/XR headset running webXR Reader software.
‘Server’ means software which provides continual access to data, via HTTP.
’Storage’ means software which stored document metadata (in List format) documents and library context metadata on physical storage.
‘Document Metadata’ includes core visual-meta, highlighted text and AI and XR generated temporary Visual-Meta.
‘Documents’ means initially PDF but will support any format.
‘Context Metadata’ means data which describes a use and/or display of the collection of documents and their associated metadata. This can be a single headset in a single real or virtual environment where documents are assigned positions etc., and it can mean the same headset in a different virtual environment or Augmented Reality environment etc.
‘Reader XR’ is the software to be developed for XR headsets.
Bjorn’s notes
If we think of the data store as the authoritative source of user’s data, what we are faced with is essentially a replication problem. That is, whenever an application that uses data from the datastore starts up, it wants to know “what has happened since the last time I was connected to this store? What’s new, what’s gone, and possibly what has changed. Of course this can be refined to “what has happened with the subset of data that this application on this device is interested in”.
Change is difficult to deal with, which is why immutability is a great approach: rather than allow data records to change, you just add new versions of them. A given version of something is never allowed to change. This will ensure you keep any references to previous versions intact, while providing a mechanism for discovering that the version you have has been superseded by one or more newer versions. It also means replication is simple – you either have the document with id abc123 version 1 or you don’t. There is no need to compare it to anything to determine which version it is.
It makes sense to make the version part of the ID you use whenever referring to a piece of data. So for instance abc123@1, abc123@2 etc. (This is essentially how some software systems manage dependencies: you use both the unique name of the dependency and its version). You can also create shorthands whenever you are interacting with the user. For instance abc123@latest to instruct the system to “figure out what the greatest version number of abc123 is and then use that when we store the reference”.
When the Headset connects for the first time it will of course want to download all the metadata. On successive connects you want to know “what has changed since the last time” and then only download the changes. You could do this by timestamp, for instance. The response can either be all the metadata or just the IDs of the versioned metadata items (allowing the Headset to determine what it actually needs to download).
Also: Reader can announce itself on local network using Zero-configuration networking.
I propose that we need a simple way to transfer whatever is in a document’s metadata, as well as the document itself, irrespective of contents. I further propose that this simply be Visual-Meta wrapped in JSON.
• Traditional computer host application reads all Visual-Meta from a document and wraps it in JSON for transmission.
• webXR receives information and parses Visual-Meta, same as the traditional computer application would (code available for this).
This removes the need for transcoding. An additional dummy file (PDF) acts as Library, storing all Library view and layout information, also in the format of Visual-Meta and is transferred in the same way. It will simply be called Library.PDF Information is to be designed as a group. This file will include list data for 2D devices and 3D information for XR.
User’s laptop needs to be online with WebDAV running in the Library software (in the case of our basic implementation that is Reader) to synch with the webXR system. Once done the user can put laptop to sleep. Synch should be continuously attempted in order to automatically reconnect to receive data from the headset to the laptop.
You are sitting at your desk and put on your XR glasses, then you make a fist and move your hand up, as though you are pulling up a sheet, and a map of the earth appears out of the ground, centred on your location. You pull it up further and it shapes to comfortably fit on your desk where you can scale it and move around with intuitive gestures.
This is an idea for gestures as ‘shortcuts’, similarly to how we can use keyboard shortcuts or gestures on our computers to do specific things instantly, instead of going through menus.
Imagine further that if you stand up, the map covers your room, and if you gesture in a circle, your room disappears and the map stretches to the horizon. You use your hands to move anywhere on earth and if you like, you can scale the earth to fit in your hand so you can then look around the solar system, our galaxy and perhaps beyond, eventually holding Laniakea in your hands.
Picture this as a default you get from your preferred vendor of XR glasses but at any scale of looking at the world you can open your hand, palm face towards you, and have access to choose any other model for that scale, time or location. In other words, your virtual desktop is infinite but you can always choose to view a different version of anywhere, any time.
I think that in the same way that if someone wears headphones and talks to someone today and people will understand that they are likely not in Noise Cancellation mode, that they are likely in a Transparency mode and that the headphones do not necessarily fully block out sound, soon it will be the case with AR glasses that they will only augment when required.
As of December 2023 I am writing my thoughts in the Journal in Author (cmd-J in Author to instantly access) which is one single long Author document. I write a new entry with the heading level 1 and when ready to post I copy the heading, select the body of the post/thought and use Liquid to post to WordPress, which is quick and simple, since the body text is automatically posted and I just have to paste or type the title. I then set the Category to ‘Journal’ and off I go.
For those who might wonder, PDF does not enter into this, though I might produce quarterly PDF Journals or maybe keep it included annually in the Book.
I am also considering starting a document similar to the Journal for transcripts of our Monday meetings and posting them in a similar way. We’ll see…
Work for porting the software Author and Reader to visionOS is necessarily explorative. These are my personal notes to guide the research, based on experience and personal preference to be revised as we build experiments to experience what it actually feels like and learn what really works in this new environment. A design constraint is that the user should be able to seamlessly move between working in spatial and traditional environments. The practical opportunities available for coding in visionOS and the resources available are not unconstrained. We have therefore had to decide on which problem to solve first. Since space is the key difference with this medium, we have decided to go large and build a Defined Concepts Map, as already implemented in Author macOS.
Initial Problem to Solve : Mapping Concepts
Author supports the user defining concepts which can be viewed on a Map, something which testing shows can be more powerful in an expanded space where the Map can cover a larger area.
Integration with other views can include the ability for the user to toggle the Map being a background or foreground element, such as the main document view and document outline:
Above, schematic view of basic Author views. Below, same Author views tested in visionOS:
Layout Tests
Layouts have been prototyped and experimented with to experience in currently available headsets, such as the Quest Pro:
Interaction Test
A test implementation by Brandel is available to try on any headset. To use, open the following link in Google Chrome then drag an Author document (.liquid) onto the bar top left. You will then see the Map layout and you can interact with it. You can also choose to view this in your VR headset and then you will need to visit the same location to interact with the Map: https://t.co/nEIoUpiUsW
Currently implemented Map in macOS
The Defined Concepts Map in Author. The potential for porting to visionOS is truly powerful. A key aspect of the Map is that all connections are not shown all the time, which produces a cluttered view, but only when nodes are selected, allowing for more fine grained understanding of how the concepts relate:
Multiple Maps
It is not trivial, but it is relatively easy to create a Map of nodes of a large size. The issue becomes one of control of views.
The integration of different Maps, both in terms of data access and use and with the user interactions, is expected to be the main challenge, to integrate maps such as listed below, with questions such as inn visionOS, should these be accessed as windows, volumes or spaces, or–ideally–anyone the user chooses?
Defined Concepts
Library of References
Journal Research Notes
User’s articles
Online Resources
Interactions & Connections
Below is an illustration of several such Maps. Key design challenge will be how to move between them, how to see connections, how to change layout and views and how they can be nested:
Below is a set of commands for the user to specify what should be seen in a Map, for consideration as to what options should be visible in spatial environments:
Openness & Metadata Interoperability
The defined concepts are subsequently exported as Glossaries currently, and work should be done to explore including the layout of the Map view on export. Such a Map view should be extractable from documents when reading and in the Library, so there needs to be a visual and interaction means for the user to easily understand the different Maps and what scale they related to.
Early Tests
Discussion of further Work Modes
Authoring
Writing a short document should be relatively similar to writing on a laptop or desktop.
Writing a longer document becomes more of an issue of editing, with moving elements around and seeing how they related. This can likely benefit from innovative views in a larger space, going further than Author’s fold views, to maybe being able to move section off into separate displays to work on them, while still being part of the whole.
Reading
Finding what to read in a document can likely benefit from innovative views of the document, going further than what we have done in Reader, including showing all pages on a very large ‘wall’ display.
Reading pages or screens of text in this environment should be optimised for what is most comfortable for the eyes.
Library
A Library which supports innovative views may also benefit from a fully immersive space in order for the user to be able to arrange documents/books in space using layouts with specific themes/topics etc.
In order for a Library to have open books, so to speak, the metadata needs to be readily available. Visual-Meta as well as JSON and other means will be investigated to support this.
Research Journal/Notes
Another challenge is how to augment the user’s ability to take notes and then find their notes later, including notes on documents for citing them.
This is probably one area where spatial environments can really shine, with easily manipulable ‘murder wall’s and other types of views being deployed.
University of Southampton, and I was a former teacher of the year 2014 at London College of Communication.
Frode Hegland June 2023
the future of text, the future of thought, is richly multidimensional