What are you seeing: a timelapse overlaid onto a slo-mo shot at the same location.<\/p>\n
<\/iframe><\/p>\n<p>This is not the final version I had envisioned because I made a mistake when I was shooting the scene I had in mind for the final version yesterday and didn\u2019t have time to reshoot. The following are frames of the footage I shot for the final version. I would have myself walking across the frame in slow motion while other people pass by me (along the same path) in a timelapse. It\u2019s the exact same capture machinery as the test footage but the subjects align more with what I want to convey with my work.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-1723\" src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-640x343.png\" alt=\"\" width=\"640\" height=\"343\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-640x343.png 640w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-1024x549.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-768x412.png 768w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-1536x823.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture-1200x643.png 1200w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/Capture.png 1575w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<h2>Ideation<\/h2>\n<p>From an experimental capture perspective, I want to experiment with compositing different speeds of time and temporally distinct occurrences of events in the same video. I was inspired by the visuals of timelapse and hyper-lapse that contain a still subject in contrast with the sped-up subjects (see sample image below. I want to further this idea by developing a method of slowing down the subject beyond real-time and a tool that would) allow me to pick the subject and overlap it with the sped-up subjects as I wish.<\/p>\n<p>From a conceptual perspective, I want to explore our perception of the speed of time\u2019s passage and our linear understanding of the sequence of events. By contracting the past and the future into a microscopic examination of the present (i.e. by condensing the people before me and people after me into the magnified present moment of me walking along the path,) I attempt to question our place and significance when put in perspective with the grand scheme of time and examine the relationship between the past, the present, and the future.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-1726\" src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/5.webp\" alt=\"\" width=\"480\" height=\"270\" \/><\/p>\n<h2>Process<\/h2>\n<p>The basic idea behind this is to find a composition where the subjects I want to speed up are in front of the subjects I want to slow down (the reverse would work too but is less ideal due to the current state of AI.) Ideally, there should be no penetration of slow and fast subjects in the 3D space \u2014 2D space is obviously fine as that is the whole point of developing this method (this is something that happened in the test footage where people penetrated the car that I attempted to fix in the second footage.) Then I would have two temporally distinct sequences, with the same composition, one of the slow subjects and one of the fast subjects. I would then rotoscope out the subjects of interest from the footage I want to speed up. Here, I used a semantic segmentation model (DeepLab\u2019s Resnet 101 trained on a subset of COCO that matches PASCAL) to mask out anything that is not human. All that is left to do from here is to alpha-blend the extracted human overlay onto the slow-motion video.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-1852\" src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-640x346.jpeg\" alt=\"\" width=\"640\" height=\"346\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-640x346.jpeg 640w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-1024x554.jpeg 1024w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-768x415.jpeg 768w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-1536x830.jpeg 1536w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-2048x1107.jpeg 2048w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/8715A060-2E55-49B1-9CF7-0A92B5AB4301-1200x649.jpeg 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<div id=\"kgvid_kgvid_0_wrapper\" class=\"kgvid_wrapper\">\n\t\t\t<div id=\"video_kgvid_0_div\" class=\"fitvidsignore kgvid_videodiv\" data-id=\"kgvid_0\" data-kgvid_video_vars=\"{"id":"kgvid_0","attachment_id":"1857","player_type":"Video.js v7","width":"600","height":"330","fullwidth":"true","countable":true,"count_views":"start_complete","start":"","autoplay":"false","pauseothervideos":"true","set_volume":"1","muted":"false","meta":true,"endofvideooverlay":"","resize":"true","auto_res":"automatic","pixel_ratio":"true","right_click":"on","playback_rate":"false","title":"luminance_trail_demo","nativecontrolsfortouch":"false","locale":"en","enable_resolutions_plugin":false}\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><meta itemprop=\"embedUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/?attachment_id=1857&kgvid_video_embed[enable]=true\" \/><meta itemprop=\"contentUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/luminance_trail_demo.mp4\" \/><meta itemprop=\"name\" content=\"luminance_trail_demo\" \/><meta itemprop=\"description\" content=\"Video\" \/><meta itemprop=\"uploadDate\" content=\"2022-11-03T14:15:39-04:00\" \/>\n\t\t\t\t<video id=\"video_kgvid_0\" playsinline controls preload=\"metadata\" width=\"600\" height=\"330\" class=\"fitvidsignore video-js kg-video-js-skin\">\n\t\t\t\t\t<source src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/luminance_trail_demo.mp4?id=0\" type=\"video\/mp4\" data-res=\"330p\">\n\t\t\t\t<\/video>\n\t\t\t<\/div>\n\t\t\t<div style=\"display:none;\" id=\"video_kgvid_0_meta\" class=\"kgvid_video_meta kgvid_video_meta_hover \">\n\t\t\t\t<span class='kgvid_meta_icons'>\t\t\t\t<span id='kgvid_kgvid_0_shareicon' class='vjs-icon-share' onclick='kgvid_share_icon_click(\"kgvid_0\");'><\/span>\n\t\t\t\t<div id='click_trap_kgvid_0' class='kgvid_click_trap'><\/div><div id='video_kgvid_0_embed' class='kgvid_share_container'><div class='kgvid_share_icons'><span class='kgvid_embedcode_container'><span class='kgvid-icons kgvid-icon-embed'><\/span>\r\n\t\t\t\t\t<span>Embed: <\/span><span><input class='kgvid_embedcode' type='text' value='<iframe src='https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/?attachment_id=1857&videopack[enable]=true' frameborder='0' scrolling='no' width='600' height='330 allowfullscreen allow='autoplay; fullscreen'><\/iframe>' onClick='this.select();'><\/span> <span class='kgvid_start_time'><input type='checkbox' class='kgvid_start_at_enable' onclick='kgvid_set_start_at(\"kgvid_0\")'> Start at: <input type='text' class='kgvid_start_at' onkeyup='kgvid_change_start_at(\"kgvid_0\")'><\/span><\/span><\/div><\/div>\n<\/span>\n\t\t\t\t<span id='video_kgvid_0_title' class='kgvid_title'>luminance_trail_demo<\/span>\n<\/div>\n\t\t<\/div>\n<p>^ your device may or may not support the format of this video above, use your imagination if it doesn’t<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-1859\" src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-640x352.png\" alt=\"\" width=\"640\" height=\"352\" srcset=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-640x352.png 640w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-1024x563.png 1024w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-768x422.png 768w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-1536x845.png 1536w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-2048x1126.png 2048w, https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/277-1200x660.png 1200w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/p>\n<div id=\"kgvid_kgvid_1_wrapper\" class=\"kgvid_wrapper\">\n\t\t\t<div id=\"video_kgvid_1_div\" class=\"fitvidsignore kgvid_videodiv\" data-id=\"kgvid_1\" data-kgvid_video_vars=\"{"id":"kgvid_1","attachment_id":"1862","player_type":"Video.js v7","width":"640","height":"352","fullwidth":"true","countable":true,"count_views":"start_complete","start":"","autoplay":"false","pauseothervideos":"true","set_volume":"1","muted":"false","meta":true,"endofvideooverlay":"","resize":"true","auto_res":"automatic","pixel_ratio":"true","right_click":"on","playback_rate":"false","title":"segmentation_demo","nativecontrolsfortouch":"false","locale":"en","enable_resolutions_plugin":false}\" itemprop=\"video\" itemscope itemtype=\"https:\/\/schema.org\/VideoObject\"><meta itemprop=\"embedUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/?attachment_id=1862&kgvid_video_embed[enable]=true\" \/><meta itemprop=\"contentUrl\" content=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/segmentation_demo.mp4\" \/><meta itemprop=\"name\" content=\"segmentation_demo\" \/><meta itemprop=\"description\" content=\"Video\" \/><meta itemprop=\"uploadDate\" content=\"2022-11-03T14:20:05-04:00\" \/>\n\t\t\t\t<video id=\"video_kgvid_1\" playsinline controls preload=\"metadata\" width=\"640\" height=\"352\" class=\"fitvidsignore video-js kg-video-js-skin\">\n\t\t\t\t\t<source src=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-content\/uploads\/2022\/11\/segmentation_demo.mp4?id=1\" type=\"video\/mp4\" data-res=\"770p\">\n\t\t\t\t<\/video>\n\t\t\t<\/div>\n\t\t\t<div style=\"display:none;\" id=\"video_kgvid_1_meta\" class=\"kgvid_video_meta kgvid_video_meta_hover \">\n\t\t\t\t<span class='kgvid_meta_icons'>\t\t\t\t<span id='kgvid_kgvid_1_shareicon' class='vjs-icon-share' onclick='kgvid_share_icon_click(\"kgvid_1\");'><\/span>\n\t\t\t\t<div id='click_trap_kgvid_1' class='kgvid_click_trap'><\/div><div id='video_kgvid_1_embed' class='kgvid_share_container'><div class='kgvid_share_icons'><span class='kgvid_embedcode_container'><span class='kgvid-icons kgvid-icon-embed'><\/span>\r\n\t\t\t\t\t<span>Embed: <\/span><span><input class='kgvid_embedcode' type='text' value='<iframe src='https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/?attachment_id=1862&videopack[enable]=true' frameborder='0' scrolling='no' width='640' height='352 allowfullscreen allow='autoplay; fullscreen'><\/iframe>' onClick='this.select();'><\/span> <span class='kgvid_start_time'><input type='checkbox' class='kgvid_start_at_enable' onclick='kgvid_set_start_at(\"kgvid_1\")'> Start at: <input type='text' class='kgvid_start_at' onkeyup='kgvid_change_start_at(\"kgvid_1\")'><\/span><\/span><\/div><\/div>\n<\/span>\n\t\t\t\t<span id='video_kgvid_1_title' class='kgvid_title'>segmentation_demo<\/span>\n<\/div>\n\t\t<\/div>\n<p>^ your device may or may not support the format of this video above, use your imagination if it doesn’t<\/p>\n<p>There are, however, some details one has to deal with when implementing this algorithm. For one, the segmentation result is not perfect (actually, pretty awful.) One way that happens to mitigate this issue is faking a long exposure from multiple frames (i.e. hyperlapse.)\u00a0 I didn\u2019t spend too much time on this but got some decent results by simply applying a Gaussian filter and a convolution with a horizontal-line filter before blending ~100 frames together by pixel-wise maximum (similar to \u201clighten\u201d in photoshop.) One could also use background subtraction on the masked region, cluster the pixels and filter out small ones, run Canny edge detection to get a sharper edge, etc. I didn\u2019t really have time to debug those so I chose to use the segmentation results as is. There were also some other issues, such as dealing with the computational cost of Resnet inference on large frames (takes ~16G of vram to process a 3K video in FP16) and figuring out the right equation for the right blending mode but they are not worth discussing in detail.<\/p>\n<p>Can post the code if anyone is interested\u2026<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Compositing Time in Space Product What are you seeing: a timelapse overlaid onto a slo-mo shot at the same location. This is not the final version I had envisioned because I made a mistake when I was shooting the scene I had in mind for the final version yesterday and didn\u2019t have time to reshoot. … <a href=\"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/hunan\/11\/03\/hunan-personintime\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> “Hunan – PersonInTime”<\/span><\/a><\/p>\n","protected":false},"author":15,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[],"_links":{"self":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/posts\/1722"}],"collection":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/comments?post=1722"}],"version-history":[{"count":8,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/posts\/1722\/revisions"}],"predecessor-version":[{"id":1866,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/posts\/1722\/revisions\/1866"}],"wp:attachment":[{"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/media?parent=1722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/categories?post=1722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/courses.ideate.cmu.edu\/60-461\/f2022\/wp-json\/wp\/v2\/tags?post=1722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}