I am making this post right now, describing a work-in-progress as a means of documenting where I am, since currently my connection with the robot is limited and I am unable to control it real-time through RoboDK (although I think I might get it to work using RDTE or OSC). Additionally, I had very limited access to the Studio as I was out of town for an extended period and there were conflicting events the weekend I came back.

My project is a performance/interaction between a person and UR5, exploring curiosity and mimicry.

Specifically, my goal is to create a robot that looks at and follows a person, by appending a camera to the UR5 and “correcting” the movements to follow a person in real time.

Using Oz Ramos’ method for real-time control of the UR5, through RoboDK, I was able to control the robot pretty well and support interactions through HandsFree.js (which additionally supported some pretty great pose/face tracking capabilities which would allow me to implement my own robot copycat).

Unfortunately, I started having problems with connecting the robot in real time. I attempted to reset my network settings, redownload RoboDK and connect through other methods. But unfortunately it’s getting to the point that trying to connect to the robot through this SDK is proving to be more time consuming that constructing the same program using RTDE or OSC.

I think I easily could have mocked a project either hard coding motions on the UR5 or by using handsfree.js to create an interactive experience. But I am not interested in these two venues, since my goal for this project is to obtain sufficient skill in controlling the UR5 in real time.

Right now I am considering different ways of implementing this project, because I am devoted to making UR5 my baby.

This is me controlling the UR5 with my simulation:

 

 

 

This project explores individuals in time through slit-scan.

Inspiration

Slit Scan–> I am fascinated by the possibilities of distorting the body, time, and space.

Slit Scan Archive: http://www.flong.com/archive/texts/lists/slit_scan/index.html

Alvaro Cassinelli & Masatoshi Ishikawa

“Alvaro & Ishikawa‘s Khronos Projector is a multi-faceted project exploring space-time representations from video. One manifestation is an interactive spatial browser for time-lapse sequences. Time-lapse photographic sequences are formed by taking a snapshot every minute, hour or day from a fixed camera shooting at a natural or artificial landscape. A ‘Time-Punch’ brings the night as a dark eye in the middle of the sky. More simply (and classically), a Spatio-temporal gradient can be formed on the image by selecting a plane temporal filter.”

System

1. 1. Obtaining/ editing the footage,
   2. Generating a time displacement Map,
   3. Applying the map to my footage and creating the slit-scan effect in After Effect

Work 1: ME 

Gradient Map:

Final Product (original footage from past work):

 

Work 2: ME + MOVEMENT in SPACE

Displacement Map:

Final Video:

 

Discussion and Evaluation: Slit-Scan in Videos

• • Explore narratives: how stories can be told through distortion of the body
  • Against censorship: how slit-scan might “protect” certain video contents from internet censorship
  • Challenge: After Effect difficulties
  • More possibilities: photography/ videos with more calculation/design
    Derek Burnett,  “In the Spirit of Things” , Kansas album cover (1988)

This project captures the quiddity of a vibraphonist embedded in improvised music by presenting the music from mallets’ perspectives.

Music improvisation allows musicians to freely express and communicate their feelings and stories at the moment. For a vibraphonist, two pairs of mallets held by their left and right hands are the main vehicle of their feelings and thoughts. Through deliberate controls of the two pairs of mallets, a vibraphonist forms harmony and composes music.

This project delves into the quiddity of a vibraphonist in the process of improvisation by shifting the perspective to these two pairs of mallets. The system tracks the movements of two mallets and fixes the focus on each pair of mallets. This project is inspired by Evan Roth’s White Glove Tracking.

To be more specific, the system works in the following steps. The steps are visualized in the video below.

1. The software built with OpenFrameworks tracks red and yellow blobs separately as shown in the picture below. In the picture, you can see white bounding boxes  with green labels around the yellow mallet heads. The software uses ofxOpenCv, ofxCv, and ofxGui packages for tracking and parameter adjustment.
   
2. Based on the positions of two mallet heads, the software rotates, scales, and translates the video so that the pair of mallets is always centered with the same distance.
3. I used Adobe Premiere Pro to composite the two red mallet-centered and yellow mallet-centered videos.
4. For additional exploration, I tried out different visual effects in an attempt to simulate the improvisation process inside the vibraphonist’s head — for example, splitting the screen into left and right cuts, overlaying the two sides, and flipping the videos upside down as if we’re looking through the vibraphonist’s eyes.

 

This mallet-centered view of improvisational music reveals the quiddity of the vibraphonist more than still photos or normal videos.

 

 

 

 

 

 

 

Point Cloud Visualizer Documentation: https://jakubuhlik.com/docs/pcv/docs.html#installation

 

I then asked my friends and dog friends to walk for 2 minutes and avoid walking in straight lines as much a possible. Here’s what was captured in that process:

https://3dscannerapp.com/

What 3D Can 3D Scanner App Export to?

The home scanner app gives you all of these different export options, all of which I tried to import into C4D and failed.

I chose to capture my body performing. I performed for myself and the camera in front of a large-scale drawing. I decided to film myself as an extension of my previous project. Also, recording my progress has been a way to research parts of my performance practice. This method allows me to analyze and find areas of strength and weakness.

Takeshi Murata, Monster Movie, 2005, single-channel video, color, sound;04:19 minutes( https://www.youtube.com/watch?v=t1f3St51S9I)  

My process included experimentation with a high-speed camera (Edgertronic).

Phase 1: Using the high-speed camera, record clips at 10 -30 second movement intervals.

Phase 2: Break down the footage further for processing.

Phase 3: process the footage through Datamoshing program.

Phase 4: Export as a GIF

I am interested in the glitch and the experience of queering material. The slowness and movement generated by the camera are visually seductive, adding a layer of distortion.

The camera can capture a short amount of movement. It becomes a compact version of the performance. I am thinking about compression and expansion. The data moshing provides an opportunity to pull apart elements into a different realm besides just a slowed version of the vogue. The slow motion offers me the ability to understand my technique. It has the potential to be a coach to break down movement into its specificity (consider lines, facial expression, posture, etc.) However, Datamoshing gives a new level of play. The distortion becomes a tool of experimentation linked to the vogue’s improvisation. All this is formed into a GIF and animated moving g image that acts as a loop or a vignette. It emphasizes a moment of urgency and repetition.

This process exercises my level of play within my performance and the aftermath. I have discovered a new way for my performance to exist in a digital space. I am also dissecting my vogue elements, creating new forms of movement.

EDGERTRONIC (EDITED PROCESS CLIPS)

GIF:

 

 

 

A few months ago, I encountered this 2-min monologue while randomly browsing youtube. I was first convinced it was an interview because of its tone and cinematic expression, as well as its content since I knew the person talking is an actor and she is talking about what made her want to become an actor. However, this monologue is actually taken from the 1995 fiction film “Last Summer in the Hamptons” which revolves around a family of theatre actors. Once I learned it’s a fictional piece, it’s ts deceptive and reflexive qualities became even more compelling to me – an actor acting an actor talking about what is it to be an actor.

For my Person-in-Time experiment, I created a sort of “broken telephone” game of acting: I asked 4 actors (all students at CMU school of drama) to improvise the mentioned short monologue while seeing and hearing it in real-time, without having any prior knowledge about it. Moreover, one by one, each of the actors was watching and hearing the person before them performing the monologue while improvising (so only the first actor in the chain actually watched the original). For this purpose, I arranged a small set containing a teleprompter and wireless earphones. I manually updated the file for the teleprompter between each session, so that the next actor in the chain will watch the performance of the one that acted before them (I’m aware that there’s a way to automate this process, but since I don’t have any prior experience with the needed tools or coding I preferred to stick to a manual process for this time frame).

All actors performed the monologue twice, and I used their second takes since I wanted my broken telephone experiment to center around their acting choices, rather than the disruption or accuracy of the text. In a sense, this experiment is more of a typology machine than a person in time.

The final video result 

After sharing my conceptual framework with a friend, they immediately brought up the Fibonacci Sequence as a reference. I found the Fibonacci spiral relevant to my experiment since it can be correlated as a loose metaphor for how the actors’ performances are built on top of one another. I, therefore, chose to use the Fibonacci spiral as a loose guide for “blocking” the footage within the frame:

Another version of the editing following the crit.

A short behind-the-scenes of the chain

The setup:

For my project I wanted to do an interactive installation

My initial idea was an app on a large monitor

app would take real time footage of passerbys and crop their body part and assemble a single Frankenstein body from them. Ex: left leg from person A, right leg from person B

concept: have fun with friends in front of this exhibit

However, finding a real time frame to frame stable multi person limb segmenter ML model is difficult. There’s no way for me to really train one   these models are produced by like teams at google.

 So I pivoted to a slightly different concept:

Roachifier!

This is a one person interactive exhibit

Here’s me interacting with this, real time:

 

 

 

 

It roachifies the participant by segmenting out the arms and the legs and then sticking it onto the torso of a roach!

It rotates your limbs so that your thigh and upper arm can’t move, they are permanently affixed at angles becoming of a roach. Kafkaesque?

Why a roach?

This is a one person concept and I was running a little low on time

I wanted to maintain an element of play and bugs are funny.

Also when I was working with the original frankenstein’s monster concept the limbs were affixed at a certain angle in a certain location, very much like a pinned bug.

 

my code: https://github.com/himalinig/frankenstein

all the code is in test.py

I wrote most of it

 

Compositing Time in Space

Product

What are you seeing: a timelapse overlaid onto a slo-mo shot at the same location.

This is not the final version I had envisioned because I made a mistake when I was shooting the scene I had in mind for the final version yesterday and didn’t have time to reshoot. The following are frames of the footage I shot for the final version. I would have myself walking across the frame in slow motion while other people pass by me (along the same path) in a timelapse. It’s the exact same capture machinery as the test footage but the subjects align more with what I want to convey with my work.

Ideation

From an experimental capture perspective, I want to experiment with compositing different speeds of time and temporally distinct occurrences of events in the same video. I was inspired by the visuals of timelapse and hyper-lapse that contain a still subject in contrast with the sped-up subjects (see sample image below. I want to further this idea by developing a method of slowing down the subject beyond real-time and a tool that would) allow me to pick the subject and overlap it with the sped-up subjects as I wish.

From a conceptual perspective, I want to explore our perception of the speed of time’s passage and our linear understanding of the sequence of events. By contracting the past and the future into a microscopic examination of the present (i.e. by condensing the people before me and people after me into the magnified present moment of me walking along the path,) I attempt to question our place and significance when put in perspective with the grand scheme of time and examine the relationship between the past, the present, and the future.

Process

The basic idea behind this is to find a composition where the subjects I want to speed up are in front of the subjects I want to slow down (the reverse would work too but is less ideal due to the current state of AI.) Ideally, there should be no penetration of slow and fast subjects in the 3D space — 2D space is obviously fine as that is the whole point of developing this method (this is something that happened in the test footage where people penetrated the car that I attempted to fix in the second footage.) Then I would have two temporally distinct sequences, with the same composition, one of the slow subjects and one of the fast subjects. I would then rotoscope out the subjects of interest from the footage I want to speed up. Here, I used a semantic segmentation model (DeepLab’s Resnet 101 trained on a subset of COCO that matches PASCAL) to mask out anything that is not human. All that is left to do from here is to alpha-blend the extracted human overlay onto the slow-motion video.

^ your device may or may not support the format of this video above, use your imagination if it doesn’t

^ your device may or may not support the format of this video above, use your imagination if it doesn’t

There are, however, some details one has to deal with when implementing this algorithm. For one, the segmentation result is not perfect (actually, pretty awful.) One way that happens to mitigate this issue is faking a long exposure from multiple frames (i.e. hyperlapse.)  I didn’t spend too much time on this but got some decent results by simply applying a Gaussian filter and a convolution with a horizontal-line filter before blending ~100 frames together by pixel-wise maximum (similar to “lighten” in photoshop.) One could also use background subtraction on the masked region, cluster the pixels and filter out small ones, run Canny edge detection to get a sharper edge, etc. I didn’t really have time to debug those so I chose to use the segmentation results as is. There were also some other issues, such as dealing with the computational cost of Resnet inference on large frames (takes ~16G of vram to process a 3K video in FP16) and figuring out the right equation for the right blending mode but they are not worth discussing in detail.

Can post the code if anyone is interested…