Imagine a pair of reading glasses that allow you to read more immersively, more critically, more to your liking. Imagine an augmented reality experience that allows you to keep the best part of reading the newspaper: the smell of newsprint, the flop of the page in your hands, the neat columns and sections that help you focus on one facet of life at a time.

Combine the two together and you have an augmented reality experience that allows you to keep all the best parts of reading the paper while harnessing the power of the internet.

design process 

I took care to notate our design process. It’s not everyday that we develop (hack together!) an un-tethered virtual reality experience and then work backwards to consider the possible applications of the device. While I can’t diagram out a particular iterative process, I can say that three different processes added three different types of insight in the process. Making technical prototypes helped us understand the affordances and limitations of VR, the Leap Motion, and the processing capabilities of Unity running on Android. From these affordances and limitations, we were able to draw out a core mechanic for our interaction designs. Considering real world uses for the tool helped us identify activities with rubrics for use that align with the core mechanics and draw attention to real world problem spaces. Considering imagined worlds, whether through game design or design fiction, helped us understand what design features were successful or unsuccessful and imagine future uses.

After building two technical demos, we sat down and talked about what we learned. We expanded our discussion to consider using the video feed from the phone and building an AR experience. We listed the affordances of limitations of AR and gesture recognition and came to the conclusion that the core mechanic we wanted to pursue was concealing/revealing. We then brainstormed real world scenarios that might include concealing/revealing. I used those brainstorms to quickly and indiscriminately write some design fiction-inspired use-scenarios. You can read the scenarios and more about the brainstorming process here.

The scenarios I wrote were predictable, stereotyped, shallow and sometimes even preachy. But the objective wasn’t to be clever or even imaginative, it was to riff off a brainstorm and mine our mundane visions of the future. By pushing myself to come up with some use scenario for each item that we brainstormed, I was able to look across the scenarios and pull out what was necessary for a successful use-scenario:

Guidelines distilled from design fiction scenarios:

1. The scenario must feature one object.
2. The object and its use must have subtext or a backstory.
3. There must be an established gestural vocabulary and rubric for using the object.
4. You must be able to interact with the object close to your body, in the space immediately in front of your head and chest.
5. The interaction must be constrained to the rules of the physical world.

Based off of these guidelines, we brainstormed a number of interactions from taking a toaster apart, to caring for a pet, to examining a patient. I broke down each interaction according to the features above.

Considering the core mechanic: concealing and revealing, and the above criteria, we landed on reading the newspaper. Investigative journalism is all about uncovering the truth, yet the news is veiled by subjectivism, bias, and sometimes outright deceit. The mechanic and the subtext of the object seemed to be in alignment.

I then began breaking down the gestural vocabulary of both newspaper reading and the Leap Motion:

matching gesture with mechanics:

Then thought about the paper itself, breaking down the interactions–starting with concealing and revealing–capable when interacting with a fully printed paper or when interacting with a completely blank paper:

Landing on three different models for the reading the news:

I then created some design fiction scenarios, including drafting fictional testimonials for a few of the a-news reading scenarios:

Paradigm 1: truth telling

The news represents the journalists’ best attempt at the TRUTH. The problem that this causes are 1) the sheer overwhelming affect of the truth 2) the truth is subjective. Leading us to…

The news comes to you redacted. When you put on the glasses and point to a redacted box, a warning menu pops up…

You could then choose to swipe off the redactions and read the text as it was written. Of course, you could designate your own personal tolerance settings before reading the paper…

Now let’s hear from one of our fine customers…

Full testimonial: “Every morning I sit at my desk with my cup of coffee and read the news. I survey the state of the union, so to speak, the state of the world, actually. There’s something about that that just, puts me in the right frame of mind to start my day. Reminds me of my place in the world. Why I’m doing all this in the first place. But the news today, it’s gotten so…lurid. So, colored…I mean, what are we doing about Syria? Just spell it out. Then there’s all this millennial-speak, the newest do-dad and such. If I hear the word millennial one more time, I think I’ll pull my hair out. Kids always think they’re the first one to do everything, to invent anything, to have a coherent thought. So I just block it out. All of this nonsense about standing in the streets, in the cold, with your little poster boards. You think you invented that? My generation, we perfected that, and what did it lead to? Death, death of all of those young men, the undermining of our military, of our whole society if you ask me. Read a history book. Oh, look at me, this is no way to start the day…no way at all.”

Paradigm 2: pandering

In this model, we think of the newspaper as a commercial product that panders to the views and values of its readership. If we think of the newspaper  this way…

…then we might think that in the future, all that we retain from the paper is its organizational structure. The paper itself becomes a generic associated press feed, organized into columns, by importance, date and theme. All of the interpretive power of the news is customized to the reader. For example, our advanced machine learning algorithms can translate the news into the language, style or vernacular of your pleasing:

If you can imagine translating the language of the newspaper, you might also translate the visual style of the paper. For example, you might translate both the linguistic and visual style of the paper to the era of your choice (Do you really belong in the ’80’s?).

Permutations of paradigms 1 & 2:

While you’re annotating the news, why not share those annotations with others? Start your own subscription channel, perhaps.

Local newspapers are closing down left and right. Maybe their value is not in reporting the news but in interpreting the news in ways that make sense to their local readership. The job of the writer is not to be the authority, to have the authorial voice, but in our (post-)postmodern understanding, the writer stands beside the reader as a guide in the interpretation of global and personal events. It’s up to the writer to convince the reader that he is a reliable guide because of the life experience, political, religious, or national allegiances, beliefs, points of view, and humanity he shares with the reader.

Let’s here a testimonial from one of the subscribers to our annotation subscription service:

Full testimonial here: “And I say, “down with the authorial voice!” You can pretend to be more important than the rest of us. You can pretend to be smarter, more worldly, more worthy. But we all know that you lucked into your position, just like the rest of us fell into ours. You’re just another guy, probably another white guy, probably from Harvard. I meet guys like you all the time.

And you, local newspapers, stop pretending like you’re doing more than wrapping up AP reports and press releases with exclamation points and sweeping generalizations. I know they’ve slashed you to the quick. I know it’s not your fault, but you gotta let go.

I’m an informed citizen. I read the news. In fact I read the news five or six times a day. Sometimes I read the same article five or six ways, just to try to figure out what’s really going on, you know, under the surface. I mean who doesn’t trust Ira Glass’s commentary, and I can just plug in the ear buds and listen to him explain shit to me anytime I want. Then there are articles that I just Russell Brand gonna go ape-shit on with some kooky video, dressed in his pajamas, and I just have to…I mean have to watch at least some of it. I follow serious stuff too. Rebecca Solnit and DeRay’s annotations are always on point. Maggie Atwood, or even Miranda July if I’m feeling a little whimsical. The point is staying informed is work, and I can’t just close my eyes and turn off my brain and assume that you know what the fuck you’re talking about.”

Paradigm 3: investigative reading

All of us are capable of reading and writing, all we lack is access to information. If you start with a blank newspaper, add an article or topic of your interest, you could  fill the pages yourself with your investigation.

Let’s hear from one of our investigative news readers:

Full testimonial: “You’ve noticed it right? A phrase that sticks out. It sounds official, like it’s always been there, but has “gentle loner” always been a category, a subsection of the sect: loners? Was my mortgage always “subprime”? Does Rick Warren “radicalize” his followers? When did the “war on Christmas” begin? And then there’s the idioms, the turns of phrase “the lipstick on the pigs” tucked away in the “lock boxes” of our minds–if you will. And we all know the coded language like “thug,” or “athletic”. I mean is it a “riot” or an “uprising”? Who makes those things up? Where’s the spin machine? I mean it’s not like it’s one news channel or the White House or a particular lobbying group. How can I read the news, I mean actively, critically, read the news, if I don’t know where this shit comes from? Who said it first? Who is the person or agency or machine? I don’t just mean your name. I mean, I want to read your Twitter stream. I want to know who pays your salary. What’s your record on this issue? All that information is out there, if you dig a little. I can fill this whole paper, the whole damn thing tracing one turn of phrase back to it’s origins, and then figuring out the origins of the origin. That’s how you learn, you know. Question what you read.”

In conclusion:

As you read through these, I’m sure that some stuck out more than others. Some of them make more use of AR and/or a gestural vocabulary than others. Some would be more successful that others if used with our system.

But none of them are intended to be built. They are intended only to be provocations. To be an imagined future that allows us to discuss and evaluate the direction of news, of reading, and of mixed reality. If there is one thing that we’ve learned over the course of the project is that the imagined world, the creative content itself, is a sort of discounted prototyping tool for the future. It is only intended to last as long as the experiment, and then the prototype is to be destroyed and forgotten.

I have never been in this situation. I’ve never had a piece of tech that I then tried to find and design a use for. My first thought was to riff off of something AR is already being used for: instruction. So, I bounced around the idea of making an experience that was a “critical how to.” In a previous blog I looked at some material from destructibles.org, some classics (how to pick a lock, The Anarchists Cookbook), some political propaganda, and some activist-oriented pieces.

Next, as a team, we had a brainstorming session. We started with the tech: AR and touch and tried to list the mechanics, affordances and limitations of those things as they stood. Ultimately, we were looking for the overlap between what AR is good for and what touch is good for. We both agreed that “to uncover” might be a good mechanic that might incorporate the hand and using the AR as a window to what lies beneath. This mechanic was potentially inline with the “critical how to” idea.

Top left: mechanics with AR; bottom left: mechanics and limitations of touch; mid: experience; right: connecting experience to life.

However, we wondered, if you are manipulating a virtual object overlaid on video feed of the real world, how necessary is the real world? Why wouldn’t it just be a virtual experience? How can we add interaction with the physical object and physical environment? It lead us down the path of thinking about a two-person experiences where one person uses the lenses to see a layer of reality hidden from the other person and must help the second person navigate the world. To help make the idea concrete we listed ways that this experience–when one world has an understanding of the world that other isn’t privy to–occurs naturally, in real life.

The next class session I expanded on our brainstorming. As an exercise, I tried to flesh out a possible use scenario for each category of experience we brainstormed. I typed quickly, paying little attention to detail or grammar. I also didn’t tie myself down to the limitations of our current technology (or any technology) instead I tried to let the story unfold and just “made up” the technical functions that I needed to keep the scenario moving. I also wasted no time trying to be original. I did not choose the best example from the category. I chose the most obvious, the most readily available in my mind. The point wasn’t to be brilliant, but just to find a direction.

Google Docs – via Iframely

After writing these out, as scattered as they were, it got me thinking about writing a series of short “plays” for VR/AR as design fiction. Not a “play,” per se, but still a script, a scenario, based on current technology but not being hampered by it. It might be an interesting way to get at the question posed earlier in the semester: “where is the subtext in VR?”

Then we talked with Larry. He was interested in the “man on the street” scenario. We watched Bjork explain television. We discussed the theatre involved in polling folks for their understanding of something like “the internet.” We also discussed using physical props to help someone explain the internet.

The most compelling part of the conversation for me was when he asked me if this was a useful methodology for me to pursue. The semester has been a difficult one for me research-wise as I try to define my area of study. I’ve been thinking a lot about methodology. I’ve been thinking about what it means to bring my humanistic background and expertise to system design. I am still an artist. In fact, my greatest strength as a scientist is that I am an artist. As a side project I’ve been developing movement/acting workshops as design probes–getting people up and moving and making together to reveal their values, expectations or mental models related to future technologies. The process is not so dissimilar to what I did on paper–hashing out scenarios not necessarily to design better things, but to understand what expectations and assumptions I have of the experience and/or technology. Time is running out in the semester, but maybe there’s the opportunity to pull some of these threads together.

What if…

We collected folks mental models of “going viral” or “twitter celebrity,” by piloting a movement/acting workshop that asks folks (folks in this case being people in class and those we can convince to come in) to create going viral in 3-d space and time. Then use the results of the workshop to suggest an AR experience that guides you through “going viral.” The workshop appeals to me over asking random people on campus 1) (IRB) 2) Any materials that I provide to explain the “going viral” will plant a metaphor in someone’s mind, 3) I’m, interested generally in using the body to express thoughts and feelings.

YouTube / Bloomberg Business – via Iframely

(This is less advanced, but you can see more of the interface:)

YouTube / Steven Henderson – via Iframely

…then, why not play the idea of an AR how-to that asks us to think about our world differently. Our set-up involves virtual touch. What does it mean to virtually touch something?

I am considering themes:

how to perform…

safety

resistance

respectability

tolerance

freedom (of speech)

In just the last week we’ve seen these performed with

football helmets, cameras, posters and sharpies, hand-holding, black clothing, poetry (Claudia Rankine’s “Citizen,”), not to mention horrible violence.

Thinking about how-to’s:

The Freedom Fighter's Manual (CIA pamphlet dropped over Nicaragua in 1983)

Make your own Newspaper Headlines

uniteyouthdublin.files.wordpress.com – via Iframely

capricorn.org – via Iframely

antenaantena.org – via Iframely

Then there are survival guides:

Pocket Survival Guide

Also thinking about James Pierce’s work on designing resistant interactions (having just seen his thesis presentation last week). And the instruction manual that accompanied the Obscura 1C Digital Camera:

jamesjpierce.com – via Iframely

Here is “Publish: Your Journey!” In the first iteration, we were installed in a farmer’s market. We were invited by Manifest Destiny: Engaging a Changing Landscape, a group from UC Irvine. We set up typewriters for the public to type up their response to “Where will you be?” then they pinned their writing to a location in the Orange County map book.

Another installation of this project was located in Union Station in downtown Los Angeles. In this professional writers were installed in the information booth and worked one-on-one with the public to write a story of “Where did you begin?”

YouTube / Chocolate Films Workshops – via Iframely

“ArtMaps: Interpreting the Spatial Footprints of Artworks”

This appears to be two things (at least). One is mapping art works to their actual location in the real world. The other is tagging photographs and text to specific locations (creating art work). Comments and conversation centers around the tagged art works. I love how the art work anchors the conversation.

Vimeo / Alexander Reederなこそま – via Iframely

N Building is a commercial building near Tachikawa station. The facade is a static QR code. When viewed through an iPhone app, characters appear, store information and live Tweets from people inside the building. From the designers: “Our proposed vision of the future is one where the facade of the building disappears, showing those inside who want to be seen.”

YouTube / pokemon – via Iframely

I couldn’t pass up the opportunity to share Pokémon Go! Pervasive games have been around since the 1970’s, but have never really caught on. More recent pervasive games were basically played by sending the player text notifications. Pokémon Go! promises an augmented reality game that uses location tracking to encourage players to play in real-life locales.

Vimeo / epure – via Iframely

Guests personify the character, who is undergoing a surrealistic transformation. Guests experience the strange body transformations themselves. Their movements affect the movement of the projected character, but changes in the projected character also affect the way the guest moves. Designed by artists in the collective, Lab212, in Paris.

Vimeo / schnellebuntebilder – via Iframely

As the guest moves, his body appears to disperse and combine with it’s surroundings. Using Kinect and Ableton Live. The coordination of movement, visual effects and music is impressive. Schnellebuntebilder is an animation and interaction design studio in Berlin.

Vimeo / The Heavy Projects – via Iframely

A mobile augmented reality app that projects curated works of art over advertisements in the New York subway system. The app reclaims ad space as art space.