This week we continued research and design, focusing on three main aspects: designing UX concepts, playtesting our game concept, and testing the technology we plan on using. We acquired two Holokits as well as two Google Pixels to use for development, and so were able to start playing with those.

Designing UX Concepts

Our designers spent some time researching AR/VR UI design guidelines, and came up with a few guidelines of our own. Chiefly, we want to make sure that UI elements and interactions are anchored in physical space and actions, and avoid 2D menus. To that end, we developed a few concepts for micro-interactions, based on using hands as the primary interaction tool.

Unfortunately, we haven’t heard back from Manomotion about acquiring a license for hand recognition. While we will continue reaching out to them, we also started designing concepts for marker-based interaction–using a marker for painting, as well as a marker in the opposite hand for a palette or prompt card. We incorporated some of the gestures conceived for hand-based interaction into our marker concepts, like attaching a marker to a strap to slip over the flat part of the hand so that a player can flip their hand over like viewing a card.

We tested wearing the Holokit headsets, as well as a modified Google Cardboard headset. The Holokit headsets are top-heavy, because of the placement of the phone, and so we began developing concepts for a multi-piece strap that goes over and around the head to keep the headset on.

Playtesting the game

We also spent some time playtesting our game concept, as much as we can without having developed the game itself. Two of us played Pictionary using Tilt Brush on the Vive, and noted what about the game works well in VR, and what doesn’t. We concluded that the most fun out of playing Pictionary in 3D space involves 1) using the physical space fully, like drawing items of clothing on yourself or drawing a trampoline and then jumping on it; 2) taking advantage of the different brushes and textures available in Tilt Brush, like using the fire texture to draw real fire; and 3) watching your friend draw in the space and the excitement of the reveal of what they’re trying to draw.

Our other concepts–like drawing something that your friend describes–weren’t as fun as straight-up guessing a word that your friend is drawing. As one of our team members put it, what makes it fun is the combination of Pictionary and Charades.

Testing the tech

Since we finally acquired the Holokits and phones for testing on, we were able to start fully testing the capabilities of the different hardware and software we’re considering using.

Tilt Brush sdk

The Tilt Brush SDK supports importing Tilt Brush drawings into AR, not drawing them. Still, it gives us access to the brushes and textures used in Tilt Brush, so we plan on incorporating those into our own painting system.

AR CORE

With the Pixels, we were able to test the capabilities of AR Core, Google’s AR platform for Android devices. While its plane recognition capabilities are far better than Vuforia’s, it doesn’t support markers. Since we need to use markers to draw in 3D space, we have to pass over AR Core for now.

Holokits

We downloaded the Holokit app and tested some of the demos in the headset. As far as we can tell by running the demos, the Holokit app appears to first recognize a plane using the phone camera, place an object on the plane through the phone camera, and then remove the camera feed and switch to a binocular view for the user to finally put their phone in the headset. The result is that the object (or objects) is reflected off the glass in the headset to appear like a hologram projected onto the real world.

Without viewing the world through the camera feed, the projected object is a little offset from the real world, and the in-app plane recognition isn’t that great. While those errors could be fixed by using a different plane recognition and aligning the video reflection with the real world, a more concerning problem with the Holokits is how the edges of objects are clipped by the edge of the phone’s camera. While this wouldn’t be a problem if one was viewing it through the phone directly, when reflected onto the real world the effect is odd.

Regardless, developing the game for the Holokit or a Google Cardboard-like direct view experience will be mostly the same, as will the UI/UX and headstrap design. We plan to keep testing with the different headsets once our prototype is more developed.

Vuforia

Vuforia supports plane recognition as well as marker recognition. While its plane recognition is buggy, it is functional, and its marker recognition works well. Using Vuforia and markers, were able to successfully draw in 3D space!

Granted, the drawing was limited to a small space, as the camera has to be very close to the plane marker for the software to recognize it. On top of that, if the plane marker is obscured in any way, the device loses track of the positions in 3D space and prevents the drawing from being rendered in 3D. Because our paintbrush marker was just a printed-out marker on a sheet of paper, this was problematic–but can definitely be solved by designing a better paintbrush marker.

Summary

This week, we learned that the most fun out of a 3D Pictionary game comes from adding in a Charades-like physical action element to it. We developed some UI design principles and began designing straps to create a hands-free AR headset. We also tested Holokits and were able to do some rudimentary drawing in 3D space using markers.

For next week, we plan to keep pushing on developing this game with markers. Can we make it possible for the fun interactions with physical space that we discovered in playtesting to work using marker recognition? If not, can we make the game fun in other ways? We also plan to continue developing headset and paintbrush+palette marker designs, to make the interactions as straightforward as possible.

Keep checking back here for more updates on Studio Mar’s development!