New Years Eve morning 2023 Letter to the Future of Text Community  

Us humans evolved the way we did because of the evolutionary pressures which shaped us throughout time. How might we have been different if our environment and thus evolutionary pressures were different, and how might we evolve now, now that we exist in a very different environment from our ancestors? How The Mind Changed(Jebelli, 2021) by Joseph Jebelli and Lewis Dartnell’s Origins(Dartnell, 2019) (both of whom I plan to invite to contribute to The Future of Text Vol V) are wonderful at outlining how our bodies and minds changed over evolutionary time. Not only have we evolved arms and hands, we have also evolved mental circuits, such as the amygdala which helps us integrate perceptions to inform us of potential danger. We might now ask: how should we evolve now, considering we have the potential to shape the environment we live in and thus shape ourselves? 

I would contend, to no-ones surprise, that text has been one of the most powerful augmentations of the human mind. Text allows for freezing of statements for communicating across time and space. I say ‘statements’ and not ‘thought’ since it is of course not pure thought which is frozen and communicated, but thoughts framed as text. The act of writing is an act of structuring, of shaping thought, from a single sentence to a paragraph and beyond. We might look at thought as being two dimensional with a vector in one direction; always pulling to the future and receding into the past. Writing thought down gives it a constraint which allows for multiple dimensions to occur; it remains in place for reference in a moment or in a thousand years. At the most basic, we can read and re-read a sentence for as long as we like. With speech, if we want to revisit what was said, we will at some point fatigue the speaker and every utterance will carry subtly different weights and tones. With text we can fill an index card, a Post-It, a page, a huge paper roll, a digital screen or projection with text and refer to different parts of our thought, greatly expanding our capacity to express ourselves and see how the different ‘strings’ of our thought connect–or don’t connect, as the case may be. This is why writing of any length beyond the basic social media post coherently takes real mental effort, as does reading anything beyond basic complexity and novelty. 

This is why I would say that vastly improving how text is written/recorded and read/extracted, bears a huge opportunity for how we can augment our minds by upgrading the mental environment we operate in. We definitely did not evolve to live through tiny rectangles. 

Our Lab’s mission over the next year–as we have chosen to accept it–is to first make it practical and frictionless for an academic user (our initial use case) to access their own Library of documents in a headset, as well as the ‘knowledge’ of what the documents is and how they relate–the metadata. We hope to complete this soon. We are then tasked with working with expanding our minds through expanding how we can view and interact with this text. We hope you will join us in cyberspace, or the ‘metaverse’ to test what we build and be part of the process. 

The avenues we choose to go down when looking at textual knowledge work in XR/eXtended Reality, will have repercussions for generations–this is the first–and only time(!) us humans are stepping into a fully visually immersive world for the first time. I expect that just like early PC interactions (copy, paste etc.) got frozen into the culture of how we interact with our knowledge on traditional computers (in ‘word processing software’ ‘spreadsheet’, ‘web browsers’, ‘email’ and only a few other categories of software) so will our early interactions in XR be frozen. More than simply freezing interactions though, ambitions will also be frozen, once we think we know what working in XR will be, there will be little cultural movement to dream up what it might be. This is what happened to traditional computing, in my view. 

This is why I ask you: If you are not already thinking about work in extended reality, please join us, as actively or passively as you would like to. 

The wider the discourse around what working with textual knowledge in fully immersive visual environments can be, the deeper the insights and potential will be. If you have thoughts on this and if you know anyone who might be interested in contributing, please do tell me, whether is someone you know personally or just someone whose work you are familiar with. 

Remember, the future of text is not yet written. 

Here’s to a 2024 where we can learn to extend our minds to better connect with our knowledge and each other. 

Much love and gratitude, 

Frode Hegland 

This is the Journal in PDF form up to the end of 2023:

New Years Eve morning 2023 Letter to the Future of Text Community

Good evening,

A thought on our future, and a request & invitation, below.

Us humans evolved the way we did because of the evolutionary pressures which shaped us throughout time. How might we have been different if our environment and thus evolutionary pressures were different, and how might we evolve now, now that we exist in a very different environment from our ancestors? How The Mind Changed(Jebelli, 2021) by Joseph Jebelli and Lewis Dartnell’s Origins(Dartnell, 2019) (both of whom I plan to invite to contribute to The Future of Text Vol V) are wonderful at outlining how our bodies and minds changed over evolutionary time. Not only have we evolved arms and hands, we have also evolved mental circuits, such as the amygdala which helps us integrate perceptions to inform us of potential danger. We might now ask: how should we evolve now, considering we have the potential to shape the environment we live in and thus shape ourselves?

I would contend, to no-ones surprise, that text has been one of the most powerful augmentations of the human mind. Text allows for freezing of statements for communicating across time and space. I say ‘statements’ and not ‘thought’ since it is of course not pure thought which is frozen and communicated, but thoughts framed as text. The act of writing is an act of structuring, of shaping thought, from a single sentence to a paragraph and beyond. We might look at thought as being two dimensional with a vector in one direction; always pulling to the future and receding into the past. Writing thought down gives it a constraint which allows for multiple dimensions to occur; it remains in place for reference in a moment or in a thousand years. At the most basic, we can read and re-read a sentence for as long as we like. With speech, if we want to revisit what was said, we will at some point fatigue the speaker and every utterance will carry subtly different weights and tones. With text we can fill an index card, a Post-It, a page, a huge paper roll, a digital screen or projection with text and refer to different parts of our thought, greatly expanding our capacity to express ourselves and see how the different ‘strings’ of our thought connect–or don’t connect, as the case may be. This is why writing of any length beyond the basic social media post coherently takes real mental effort, as does reading anything beyond basic complexity and novelty.

This is why I would say that vastly improving how text is written/recorded and read/extracted, bears a huge opportunity for how we can augment our minds by upgrading the mental environment we operate in. We definitely did not evolve to live through tiny rectangles.

I believe firmly that we can extend how we interact with text to give us a mental environment where we can grow. I can vaguely imagine (and this is why we need to experiment to see what the realities will be) some way of shaping textual knowledge, using all the visual, tactile and auditory means at our disposal and all the interaction potential from a simple ‘look’ to using our hands as a sculptor might, on to a dancer who truly dances in the information. Imagine reading text where books and academic papers become transparent and display their contents in entirely new and intuitively visible ways. Imagine experimenting with extracts and connections to the point where we only see traditional sentences when we slow down to focus–the web of knowledge is truly spun. How will our minds develop, how far can we take this?

Our Lab’s mission over the next year–as we have chosen to accept it–is to first make it practical and frictionless for an academic user (our initial use case) to access their own Library of documents in a headset, as well as the ‘knowledge’ of what the documents is and how they relate–the metadata. We hope to complete this soon. We are then tasked with working with expanding our minds through expanding how we can view and interact with this text. We hope you will join us in cyberspace, or the ‘metaverse’ to test what we build and be part of the process.

The avenues we choose to go down when looking at textual knowledge work in XR/eXtended Reality, will have repercussions for generations–this is the first–and only time(!) us humans are stepping into a fully visually immersive world for the first time. I expect that just like early PC interactions (copy, paste etc.) got frozen into the culture of how we interact with our knowledge on traditional computers (in ‘word processing software’ ‘spreadsheet’, ‘web browsers’, ‘email’ and only a few other categories of software) so will our early interactions in XR be frozen. More than simply freezing interactions though, ambitions will also be frozen, once we think we know what working in XR will be, there will be little cultural movement to dream up what it might be. This is what happened to traditional computing, in my view.

This is why I ask you:

     If you are not already thinking about work in extended reality,
     please join us, as actively or passively as you would like to.

The wider the discourse around what working with textual knowledge in fully immersive visual environments can be, the deeper the insights and potential will be. If you have thoughts on this and if you know anyone who might be interested in contributing, please do tell me, whether is someone you know personally or just someone whose work you are familiar with. This is our Lab website: https://futuretextlab.info/ and this is the Journal where I post updates, which is a simple WordPress blog: https://thefutureoftext.org/category/uncategorized/

Remember, the future of text is not yet written.

Here’s to a 2024 where we can learn to extend our minds to better connect with our knowledge and each other.

Much love and gratitude,

Frode Alexander Hegland

Learn to Dance (end of year status)

This week I watched my son Edgar wave about his magic Harry Potter wand and dance to the music of Jungle and thought:

“If you are young,
learn to dance.

Interactions in XR
will use much more
of our bodies than
fingers on keyboard
and trackpad.”

That is to say, if we do our job right…

I’m writing this at the end of 2023 when the world is in turmoil with major wars, a climate emergency, and so much more of what Doug Englebart would most likely describe as urgent complex problems. He introduced this notion–and what we should do about it–in his seminal 1962 paper:

“By “augmenting human intellect” we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by “complex situations” we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers- -whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human “feel for a situation” usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids. Man’s population and gross product are increasing at a considerable rate, but the complexity of his problems grows still faster, and the urgency with which solutions must be found becomes steadily greater in response to the increased rate of activity and the increasingly global nature of that activity. Augmenting man’s intellect, in the sense defined above, would warrant full pursuit by an enlightened society if there could be shown a reasonable approach and some plausible benefits.” (Engelbart, 1962)

As I have written in The Future of Text 4, we must nurture our imagination (2023). Doug invented much of what we consider personal computing and then the world felt it knew what’s what and innovation with how we interact with text slowed to a crawl… We now have a once-in-a-species opportunity to dream anew, upon entering XR. We must not squander this.

As we stand on the threshold of 2024 we have a focused mission in the Lab: To develop powerfully useful systems to interact with and view textual knowledge in XR, powered by robust, accessible, low-cost metadata and informed by wide and deep dialog and much testing of prototypes.

We had a meeting this week, during the holidays, where we got much further with how to upload documents and metadata to XR, as outlined at https://futuretextlab.info/headset-computer-transmission/ and next week we will kick off working out exactly how to make it all connect. Then it is up to developing interactions and more interactions and invite a wider range of people to contribute to The Future of Text 5.

That’s all really, all there is to update at this point. Let’s see how it goes.

Happy New Year!

In the book by our friend Tim who is in The Future of Text, he writes about manuscript hand-writing VS. print:

“When I’m giving talks about the Alphabets, especially if I’m speaking to a class, I often call for a volunteer and, when someone puts a hand up, I hold out a marker and ask, ‘Will you come up here and draw a capital E?’

They come up to the whiteboard, a little warily, sensing a trick, and do it.

I say, ‘That’s great, but the three horizontals are not quite parallel. Could you just fix that?’

When they’ve erased the lines and drawn them again much more carefully, I say,

‘Well, actually this one’s a bit longer than the other one…’

So they correct that, and then I say, “And the vertical isn’t quite vertica…’ And maybe I go on to ask for serifs, and get more and more annoying in my polite demands.

The point is, it can’t be done. What they are attempting to “write” is a mechan-ically-reproduced image derived from letters on monuments that were created using stonemason tools – squares, straight-edges, agents of geometry. The human body is not designed to do that.

Then I show the class the E from the Eastern Cham script of Vietnam, and I say,

‘Okay, I want you to write this, with your fingertip, in the air.’

They’re a little self-conscious about being so demonstrative in public, but they do it, and I tell them to keep doing it, writing in the air, until the motion starts to feel natural. Then I say, “Okay, without stopping, just look around at everyone else.

Look at their wrists, their hands, their fingers. That is the hand of a Thai dancer.” I’m making two points here, I hope. One is the difference between a script, something written by hand, and a typeface, something designed to be printed.

The first is an extension of the natural, graceful movements of the human body; the other is not.

The other point has to do with geometry, and with the values that are embea-ded in writing. The Latin alphabet as we use it today descended not from everyday Roman handwriting but from monuments to emperors. The letteriorms themselves represent the virtues of a military empire: stability, balance, longer ity. Almost any Latin uppercase letter stands on its own feet, as if bestriding the known world. What’s more, every Roman emperor was by definition a god, so graphically, these qualities had to be represented not by ordinary, everyda vernacular shapes but by ideal Euclidean forms: symmetry, parallelism, rectan-gularity, perfect circles. Shapes not found in nature.

The Cham script doesn’t care about those values. There’s a grace, and an implied balance but no symmetry. It is still a human script rather than a divine one or a mechanical one”

Tim Brooks(Brookes, 2024)

Writing about how he was photographing brain cells, not drawing freehand: “…I was being more mechanical, more objective. My bet to myself was that if I was to repeat the process I had come up with, using the same set of negative images of the same cell, I could draw an identical picture again and again.

But that is not the way it worked out. Several times I found that I had unintentionally reconstructed the same brain cell twice. The same multiple negatives, the same pool of light, but two drawings that were subtly different.

In the shifting of the paper, the flick of the pen, the guesses that bridged two sharp dots separated by a blur, in the incremental flow of the thickening branches and the finest of spines, something subjective crept into my method.

Imagination was an essential ingredient to the way I was working. Which was the ‘right’ drawing? With practice, my drawings became more compelling, and more true to life. I realised they exhibited not just a modicum of artistic licence but also an artistic style.

What is artistic style when you are essentially tracing an outline? Something seeps into the way your hand moves after you have drawn brain cells again and again. Something about the energy of the way in which they have grown transfers into the drawings to give them life. I can tell immediately if a drawing is second-hand, penned by a professional illustrator copying from another source. There is something lifeless in the execution and the intention of the lines. The intuitive sympathy with the form is gone. To draw is to know. And for this reason, drawing became central to the development of brain cell theory. There is art at the heart of this science.”

Richard Wingate(Wingate, 2023)

Adding

User does ctrl-click without selecting text (or with selecting text if they prefer) and run a command from Ask AI (a prompt).

The results appear in the current dialog style, including the ability to edit this text, so as to remove results which are not interesting.

New: A button is at the bottom of the results screen with ‘Save Metadata’

When actioned, the entire contents of the Ask AI dialog is saved in a new Visual-Meta appendix, as described at https://visual-meta.info/visual-meta-ai/

This metadata will at the same time be stored in the same way as highlighted text in the document is, allowing the user to perform Find commands in the Library for this metadata.

Parsing

When parsing for this metadata it is important to note that the regular Visual-Meta starts (when parsing from the last page of the document) with @{visual-meta-end} so when looking for temporary Visual-Meta it is important to look for @{ai- since that will be the last section of Visual-Meta AI appendices.

Of note, @{xr- will be used for XR data, which will not be developed now.

In use: macOS

The Reader software will make this metadata available for Find operations in the Library.

In use: WebXR

This metadata will be packaged, along with the regular Visual-Meta, into JSON for transmitting to a headset for use in WebXR.

Further Metadata

User can perform further Ask AI and Save Metadata functions and each time a new Visual-Meta will be added.

User Story: 

User is using their Library in macOS Reader (this is our test environment, any others should work soon too) 

User puts on a headset to continue working in their Library. 

The user accesses  ‘Reader XR’ (through web page or app) and their Document Metadata and Library Metadata is downloaded. 

The user can interact with their Library and choose to open any document for reading, changing layout of and annotating. 

When done they can put down their headset and return to their traditional computer. 

Data Flow: 

The data in Reader is in the form of Visual-Meta at the end of all documents and when writing any new Visual-Meta to a document it is also saved in a Document Metadata database.  

This Document Metadata database will be stored in the form of JSON. 

All the user’s metadata will be syncing with a server.  

When done, the Library Metadata is transmitted back to Reader macOS (or equivalent), as well as any changes to Document Metadata (such as annotations being added or removed). 

Definitions:  

‘Computer’ means desktop or laptop computer, in other words computer in the traditional sense. 

‘Headset’ means VR/AR/XR headset running webXR Reader software. 

‘Server’ means software which provides continual access to data, via HTTP. 

’Storage’ means software which stored document metadata (in List format) documents and library context metadata on  physical storage. 

‘Document Metadata’ includes core visual-meta, highlighted text and AI and XR generated temporary Visual-Meta. 

‘Documents’ means initially PDF but will support any format. 

‘Context Metadata’ means data which describes a use and/or display of the collection of documents and their associated metadata. This can be a single headset in a single real or virtual environment where documents are assigned positions etc., and it can mean the same headset in a different virtual environment or Augmented Reality environment etc. 

‘Reader XR’ is the software to be developed for XR headsets. 

Bjorn’s notes 

If we think of the data store as the authoritative source of user’s data, what we are faced with is essentially a replication problem.  That is, whenever an application that uses data from the datastore starts up, it wants to know “what has happened since the last time I was connected to this store? What’s new, what’s gone, and possibly what has changed.  Of course this can be refined to “what has happened with the subset of data that this application on this device is interested in”. 

Change is difficult to deal with, which is why immutability is a great approach: rather than allow data records to change, you just add new versions of them. A given version of something is never allowed to change. This will ensure you keep any references to previous versions intact, while providing a mechanism for discovering that the version you have has been superseded by one or more newer versions.  It also means replication is simple – you either have the document with id abc123 version 1 or you don’t. There is no need to compare it to anything to determine which version it is. 

It makes sense to make the version part of the ID you use whenever referring to a piece of data.  So for instance abc123@1, abc123@2 etc.  (This is essentially how some software systems manage dependencies: you use both the unique name of the dependency and its version).   You can also create shorthands whenever you are interacting with the user.  For instance abc123@latest to instruct the system to “figure out what the greatest version number of abc123 is and then use that when we store the reference”. 

When the Headset connects for the first time it will of course want to download all the metadata.  On successive connects you want to know “what has changed since the last time” and then only download the changes.  You could do this by timestamp, for instance.  The response can either be all the metadata or just the IDs of the versioned metadata items (allowing the Headset to determine what it actually needs to download). 

Also: Reader can announce itself on local network using Zero-configuration networking. 

#https://en.wikipedia.org/wiki/Zero-configuration_networking