Marimonda – Final Project

 

Breaking up with a robot after an intimate relationship defined by a lack of dancing. 

Rompecorazones (Heartbreaker)

In this video (in case you didn’t watch it), I am wearing a traditional Colombian Cumbia dress as I dramatically break up with a robot.

This project, at it’s core, is a vent piece that contextualizes my failure in attempting to have a robot dance with me.

My Person in Time project, at is core, was creating a robotic interaction that would follow me as I moved or danced with it. But I wasn’t able to get quite there. I spent roughly 2 months continuously debugging a UR5 Cobot (Collaborative Robot), as I attempted to have it follow my face in real time.  Using Handsfree.js, python and a couple of dozen cups of coffee, I was able to partially achieve this daunting task.

Without getting too much into the technical implementation of my robotic interactions, I had a lot of issues wrapping my head around different SDK’s and networking protocols for communicating with the robot with my computer. It’s not easy to make code for the robot to move, but it’s even harder to do so remotely from your computer. So I spent a lot of time trying to figure out how these pieces went together. I eventually hacked it, but I realized that to get to a step where the robot would be able to move smoothly with me, would take a lot more time and effort than I had. At this point I had grown bitter with the robot, verbally abusing it every time I could. Because I knew it would never be good enough to properly dance with me at this rate.

Needless to say, this piece as a whole is a testament to the technical failures that I experienced, technical failures that made me consider whether or not I wanted to drop this class at times. This piece was the process of coming to terms with my lack of technical expertise, but it is also a testament to my learning.

In terms of the video, I did consider using music/a background track (specifically of cumbia beats). But the jovial nature of cumbia sounds went against the tone of the video, and I found that my voice alone was appropriate enough for this project. Like any project, I find a billion ways in which my storytelling could be improved. I spent hours trying to figure out how to film myself interacting with the robot, and there are many distinct motions of the robots (as well as bugs) that I didn’t do justice in this documentation.

Special thanks to Oz Ramos, Madeline Gannon, and Golan Levin for helping me get started with the UR5! We developed a special relationship.

Special thanks to Neve (and Clayton), Mikey, Em, Ana, Will, Benford and Lauren for saving my butt with the narrative/filming/set up of the video!  I am not a video person, so I genuinely couldn’t have made this without everyone who helped me figure these things out. 

marimonda – PersonInTime Update

I am making this post right now, describing a work-in-progress as a means of documenting where I am, since currently my connection with the robot is limited and I am unable to control it real-time through RoboDK (although I think I might get it to work using RDTE or OSC). Additionally, I had very limited access to the Studio as I was out of town for an extended period and there were conflicting events the weekend I came back.

My project is a performance/interaction between a person and UR5, exploring curiosity and mimicry.

Specifically, my goal is to create a robot that looks at and follows a person, by appending a camera to the UR5 and “correcting” the movements to follow a person in real time.

Using Oz Ramos’ method for real-time control of the UR5, through RoboDK, I was able to control the robot pretty well and support interactions through HandsFree.js (which additionally supported some pretty great pose/face tracking capabilities which would allow me to implement my own robot copycat).

Unfortunately, I started having problems with connecting the robot in real time. I attempted to reset my network settings, redownload RoboDK and connect through other methods. But unfortunately it’s getting to the point that trying to connect to the robot through this SDK is proving to be more time consuming that constructing the same program using RTDE or OSC.

I think I easily could have mocked a project either hard coding motions on the UR5 or by using handsfree.js to create an interactive experience. But I am not interested in these two venues, since my goal for this project is to obtain sufficient skill in controlling the UR5 in real time.

Right now I am considering different ways of implementing this project, because I am devoted to making UR5 my baby.

This is me controlling the UR5 with my simulation:

 

 

Lost in Time

 

This project explores individuals in time through slit-scan.

Inspiration

Slit Scan–> I am fascinated by the possibilities of distorting the body, time, and space.

Slit Scan Archive: http://www.flong.com/archive/texts/lists/slit_scan/index.html

Alvaro Cassinelli & Masatoshi Ishikawa

“Alvaro & Ishikawa‘s Khronos Projector is a multi-faceted project exploring space-time representations from video. One manifestation is an interactive spatial browser for time-lapse sequences. Time-lapse photographic sequences are formed by taking a snapshot every minute, hour or day from a fixed camera shooting at a natural or artificial landscape. A ‘Time-Punch’ brings the night as a dark eye in the middle of the sky. More simply (and classically), a Spatio-temporal gradient can be formed on the image by selecting a plane temporal filter.”

System

    1. Obtaining/ editing the footage,
    2. Generating a time displacement Map,
    3. Applying the map to my footage and creating the slit-scan effect in After Effect
Work 1: ME 

Gradient Map:

Final Product (original footage from past work):

 

Work 2: ME + MOVEMENT in SPACE

Displacement Map:

Final Video:

 

Discussion and Evaluation: Slit-Scan in Videos

    • Explore narratives: how stories can be told through distortion of the body
    • Against censorship: how slit-scan might “protect” certain video contents from internet censorship
    • Challenge: After Effect difficulties
    • More possibilities: photography/ videos with more calculation/design
      Derek Burnett,  “In the Spirit of Things” , Kansas album cover (1988)

Person In Time | quiddity in improvisation

This project captures the quiddity of a vibraphonist embedded in improvised music by presenting the music from mallets’ perspectives.

Music improvisation allows musicians to freely express and communicate their feelings and stories at the moment. For a vibraphonist, two pairs of mallets held by their left and right hands are the main vehicle of their feelings and thoughts. Through deliberate controls of the two pairs of mallets, a vibraphonist forms harmony and composes music.

This project delves into the quiddity of a vibraphonist in the process of improvisation by shifting the perspective to these two pairs of mallets. The system tracks the movements of two mallets and fixes the focus on each pair of mallets. This project is inspired by Evan Roth’s White Glove Tracking.

Evan Roth’s White Glove Tracking project

To be more specific, the system works in the following steps. The steps are visualized in the video below.

  1. The software built with OpenFrameworks tracks red and yellow blobs separately as shown in the picture below. In the picture, you can see white bounding boxes  with green labels around the yellow mallet heads. The software uses ofxOpenCv, ofxCv, and ofxGui packages for tracking and parameter adjustment.
  2. Based on the positions of two mallet heads, the software rotates, scales, and translates the video so that the pair of mallets is always centered with the same distance.
  3. I used Adobe Premiere Pro to composite the two red mallet-centered and yellow mallet-centered videos.
  4. For additional exploration, I tried out different visual effects in an attempt to simulate the improvisation process inside the vibraphonist’s head — for example, splitting the screen into left and right cuts, overlaying the two sides, and flipping the videos upside down as if we’re looking through the vibraphonist’s eyes.

 

This mallet-centered view of improvisational music reveals the quiddity of the vibraphonist more than still photos or normal videos.

excap2 postcrit

(note: I turned in a rushed version of this project with garbage documentation and someone in crit felt they needed to absolutely obliterate me, so I spent the next two days putting together something I feel dignified presenting. That’s why the recording was posted two days after the project deadline).

I make music through live coding. Explaining what that entails usually takes a while, so I’ll keep it brief. I evaluate lines of code that represent patterns of values. Those patterns denote when to play audio samples on my hard drive, and with what effects. I can also make patterns that tell the system when to play audio from a specified input (for example, when to play the feed coming from my laptop mic). Bottom line, everything you hear is a consequence of either me speaking to a mic, or me typing keys bound to shortcuts that send lines of code to be evaluated, modifying the sound. I have many many shortcut keys to send specific things, but it all amounts to sending text to an interpreter.

Here’s a rough diagram of my digital setup

neovim – text editor
tidalcycles – live coding language/environment
supercollider – audio programming language/environment
osc – open sound control

If you’re interested in the details, feel free to reach out to @c_robo_ on instagram or twitter (if it’s still up by the time you read this).

Anyway, this setup is partially inspired by two big issues I see with contemporary electronic music.

The first is the barrier to performance. Most analog tools meant for electronic music are well engineered and fully intended for live use, but by their nature have a giant cost barrier associated with them. Anyone can pirate a digital audio workstation, but most digital music tools are made prioritizing composition. If they’re made also for performance like ableton live, they usually have some special view suggesting the use of a hardware interface to control parameters more dynamically, which to me is just another cost barrier implicitly refuting the computer keyboard as a sufficiently expressive interface for making music. This seems goofy to me, given how we’ve reduced every job related to interfacing with complex systems to interfacing with a computer keyboard. It’s come out as the best way to manage information, but somehow it’s not expressive enough for music performance?

The other problem is that electronic music performance simply isn’t that interesting to watch. Traditionally, it’s someone standing in front of a ton of stationary equipment occasionally touching things. In other words, there are no gestures clearly communicating interaction between performer and instrument. Live visuals have done quite a bit to address this issue and provide the audience something more visually relevant to the audio. Amazing stuff has been done with this approach and I don’t mean to diminish its positive impact on live electronic music, but it’s a solution that ignores gesture as a fundamental feature of performance.

400 words in, and we’re at the prompt: electronic music performance has trouble communicating the people in time actually making music. What the hell do they see? What are their hands doing? When are they changing the sound and when are they stepping back? To me, live coding is an interesting solution to both problems in that there are no hardware cost barriers and “display of manual dexterity” (ie. gesture) is literally part of its manifesto. I don’t abide by the manifesto as a whole, but it’s one of a few components I find most interesting given my perspective.

This is an unedited screen recording of a livecoding performance I did, with a visualization of my keyboard overlayed on top (the rectangles popping up around the screen are keys I’m pressing). I spent the better part of two days wrangling dependency issues to get the input-overlay plugin for obs working on my machine. Such issues dissuaded me from experimenting with this earlier, but I’m incredibly happy that I did because it’s both visually stimulating and communicates most of the ways through which I interact with the system.

The sound itself is mostly an exploration of feedback and routing external sound with tidalcycles. I have sound from the laptop going  through a powered amplifier in my basement picked up again going back into the system through my laptop mic. You can hear me drumming on a desk or clicking my tongue into the mic at a few points (to really make sure a person is expressed in this time, cause I know y’all care about the prompt). I also have some feedback internally routed so I can re-process output from supercollider directly (there’s a screen with my audio routing configuration at 1:22).

Not gonna lie, it’s a little slow for the first minute but it gets more involved.

CONTENT WARNING: FLASHING LIGHTS and PIERCING NOISES
(best experienced with headphones)

Person-In-Time | Points & Paths

 

My goal was to explore movement paths and “light painting” through LiDAR points and photogrammetry.

Polycam only allows you to capture using either LiDAR (which creates a triangulated mesh ) or photos (that creates the mesh for you and not ply. points)

I’m exploiting the points detection capture option inside of the home scanner app, because as long as I keep a fixed distance with the lens, then it assumes that my hand is also a static object within the space. That’s what’s creating the trails, but the effect doesn’t work using the other capture options in both Polycam as well as Homescanner App.

 

Experiment #1 by kcotton on Sketchfab

 

 

 

 

 

 

 

Point Cloud Visualizer Documentation: https://jakubuhlik.com/docs/pcv/docs.html#installation

 

I then asked my friends and dog friends to walk for 2 minutes and avoid walking in straight lines as much a possible. Here’s what was captured in that process:


https://3dscannerapp.com/

What 3D Can 3D Scanner App Export to?

Software – Compatible Formats

This following list is export formats that are compatible with the most common 3D Software.

The home scanner app gives you all of these different export options, all of which I tried to import into C4D and failed.

Glitter and rainbows (Person In Time)

I chose to capture my body performing. I performed for myself and the camera in front of a large-scale drawing. I decided to film myself as an extension of my previous project. Also, recording my progress has been a way to research parts of my performance practice. This method allows me to analyze and find areas of strength and weakness.

Takeshi Murata, Monster Movie, 2005, single-channel video, color, sound;04:19 minutes( https://www.youtube.com/watch?v=t1f3St51S9I)  

My process included experimentation with a high-speed camera (Edgertronic).

Phase 1: Using the high-speed camera, record clips at 10 -30 second movement intervals.

Phase 2: Break down the footage further for processing.

Phase 3: process the footage through Datamoshing program.

Phase 4: Export as a GIF

I am interested in the glitch and the experience of queering material. The slowness and movement generated by the camera are visually seductive, adding a layer of distortion.

The camera can capture a short amount of movement. It becomes a compact version of the performance. I am thinking about compression and expansion. The data moshing provides an opportunity to pull apart elements into a different realm besides just a slowed version of the vogue. The slow motion offers me the ability to understand my technique. It has the potential to be a coach to break down movement into its specificity (consider lines, facial expression, posture, etc.) However, Datamoshing gives a new level of play. The distortion becomes a tool of experimentation linked to the vogue’s improvisation. All this is formed into a GIF and animated moving g image that acts as a loop or a vignette. It emphasizes a moment of urgency and repetition.

This process exercises my level of play within my performance and the aftermath. I have discovered a new way for my performance to exist in a digital space. I am also dissecting my vogue elements, creating new forms of movement.

EDGERTRONIC (EDITED PROCESS CLIPS)

GIF:

 

 

 

Person in Time

A few months ago, I encountered this 2-min monologue while randomly browsing youtube. I was first convinced it was an interview because of its tone and cinematic expression, as well as its content since I knew the person talking is an actor and she is talking about what made her want to become an actor. However, this monologue is actually taken from the 1995 fiction film “Last Summer in the Hamptons” which revolves around a family of theatre actors. Once I learned it’s a fictional piece, it’s ts deceptive and reflexive qualities became even more compelling to me – an actor acting an actor talking about what is it to be an actor.

For my Person-in-Time experiment, I created a sort of “broken telephone” game of acting: I asked 4 actors (all students at CMU school of drama) to improvise the mentioned short monologue while seeing and hearing it in real-time, without having any prior knowledge about it. Moreover, one by one, each of the actors was watching and hearing the person before them performing the monologue while improvising (so only the first actor in the chain actually watched the original). For this purpose, I arranged a small set containing a teleprompter and wireless earphones. I manually updated the file for the teleprompter between each session, so that the next actor in the chain will watch the performance of the one that acted before them (I’m aware that there’s a way to automate this process, but since I don’t have any prior experience with the needed tools or coding I preferred to stick to a manual process for this time frame).

All actors performed the monologue twice, and I used their second takes since I wanted my broken telephone experiment to center around their acting choices, rather than the disruption or accuracy of the text. In a sense, this experiment is more of a typology machine than a person in time.

The final video result 

After sharing my conceptual framework with a friend, they immediately brought up the Fibonacci Sequence as a reference. I found the Fibonacci spiral relevant to my experiment since it can be correlated as a loose metaphor for how the actors’ performances are built on top of one another. I, therefore, chose to use the Fibonacci spiral as a loose guide for “blocking” the footage within the frame:

Another version of the editing following the crit.

A short behind-the-scenes of the chain

The setup:

frankenstein’s monster / roach – miniverse

For my project I wanted to do an interactive installation

My initial idea was an app on a large monitor

app would take real time footage of passerbys and crop their body part and assemble a single Frankenstein body from them. Ex: left leg from person A, right leg from person B

concept: have fun with friends in front of this exhibit

However, finding a real time frame to frame stable multi person limb segmenter ML model is difficult. There’s no way for me to really train one   these models are produced by like teams at google.

 So I pivoted to a slightly different concept:

Roachifier!

This is a one person interactive exhibit

Here’s me interacting with this, real time:

 

 

 

 

It roachifies the participant by segmenting out the arms and the legs and then sticking it onto the torso of a roach!

It rotates your limbs so that your thigh and upper arm can’t move, they are permanently affixed at angles becoming of a roach. Kafkaesque?

Why a roach?

This is a one person concept and I was running a little low on time

I wanted to maintain an element of play and bugs are funny.

Also when I was working with the original frankenstein’s monster concept the limbs were affixed at a certain angle in a certain location, very much like a pinned bug.

 

my code: https://github.com/himalinig/frankenstein

all the code is in test.py

I wrote most of it

 

Hunan – PersonInTime

Compositing Time in Space

Product

What are you seeing: a timelapse overlaid onto a slo-mo shot at the same location.

This is not the final version I had envisioned because I made a mistake when I was shooting the scene I had in mind for the final version yesterday and didn’t have time to reshoot. The following are frames of the footage I shot for the final version. I would have myself walking across the frame in slow motion while other people pass by me (along the same path) in a timelapse. It’s the exact same capture machinery as the test footage but the subjects align more with what I want to convey with my work.

Ideation

From an experimental capture perspective, I want to experiment with compositing different speeds of time and temporally distinct occurrences of events in the same video. I was inspired by the visuals of timelapse and hyper-lapse that contain a still subject in contrast with the sped-up subjects (see sample image below. I want to further this idea by developing a method of slowing down the subject beyond real-time and a tool that would) allow me to pick the subject and overlap it with the sped-up subjects as I wish.

From a conceptual perspective, I want to explore our perception of the speed of time’s passage and our linear understanding of the sequence of events. By contracting the past and the future into a microscopic examination of the present (i.e. by condensing the people before me and people after me into the magnified present moment of me walking along the path,) I attempt to question our place and significance when put in perspective with the grand scheme of time and examine the relationship between the past, the present, and the future.

Process

The basic idea behind this is to find a composition where the subjects I want to speed up are in front of the subjects I want to slow down (the reverse would work too but is less ideal due to the current state of AI.) Ideally, there should be no penetration of slow and fast subjects in the 3D space — 2D space is obviously fine as that is the whole point of developing this method (this is something that happened in the test footage where people penetrated the car that I attempted to fix in the second footage.) Then I would have two temporally distinct sequences, with the same composition, one of the slow subjects and one of the fast subjects. I would then rotoscope out the subjects of interest from the footage I want to speed up. Here, I used a semantic segmentation model (DeepLab’s Resnet 101 trained on a subset of COCO that matches PASCAL) to mask out anything that is not human. All that is left to do from here is to alpha-blend the extracted human overlay onto the slow-motion video.

^ your device may or may not support the format of this video above, use your imagination if it doesn’t

^ your device may or may not support the format of this video above, use your imagination if it doesn’t

There are, however, some details one has to deal with when implementing this algorithm. For one, the segmentation result is not perfect (actually, pretty awful.) One way that happens to mitigate this issue is faking a long exposure from multiple frames (i.e. hyperlapse.)  I didn’t spend too much time on this but got some decent results by simply applying a Gaussian filter and a convolution with a horizontal-line filter before blending ~100 frames together by pixel-wise maximum (similar to “lighten” in photoshop.) One could also use background subtraction on the masked region, cluster the pixels and filter out small ones, run Canny edge detection to get a sharper edge, etc. I didn’t really have time to debug those so I chose to use the segmentation results as is. There were also some other issues, such as dealing with the computational cost of Resnet inference on large frames (takes ~16G of vram to process a 3K video in FP16) and figuring out the right equation for the right blending mode but they are not worth discussing in detail.

Can post the code if anyone is interested…