MODULE 4- FILM MONTAGE, VIDEO REMIX + ROTOSCOPE ANIMATION

I. Introduction

This module will pick up from the last module, and continue to look at time-based artworks and technologies, primarily through the design fundamentals, techniques, and technologies used to create film montage, video remix + rotoscoping animation. These three art-forms are very tied to their base technologies and mediums {film, video and animation}, and also overlap with one another in technical, formal, and conceptual ways.

One reason for exploring these three formats in this module is that they also provide a more specific and focused way of approaching these larger fields - whole classes and majors and masters and doctorate programs can be dedicated to film, video or animation, and while someone in a Ph.d film program might focus on rotoscope animation in their dissertation, it is still specific enough to use as one potential point of entry and expansion for both animation and film / video for this course. The examples in this module barely scratch the surface of any of these topics in terms of history, theory or technique, but they are one way {out of many} for introducing some important design basics and considerations when starting to work in these formats.

Another reason for focusing on rotoscoping, video remix and film montage is that these three forms of artwork / media were very linked to - and grew out of - artistic experimentation and interaction with their physical / digital production technologies. These experimental forms, in turn, shaped both additional new media / art forms, and the forms they originally emerged from. This is a recurring theme in this course, both in terms of understanding how current digital design technologies are linked to older creative technologies, as well as understanding how current media forms, formats and artworks are partially shaped and informed by previous creative experiments.

1. Time, Film + Montage

Film is, in many ways, a sequence or collection of still photographs displayed at a fast enough rate to create the illusion of motion using various capture and projection technologies. There is a reason why early films were described as “motion pictures” - it was pretty literal. When film was first growing as a medium, the concept of filming and then combining two different filmed sequences together had not yet been explored. This was something that came with both creative experimentation and technological advancement - in this case, the ability to cut and connect different strips of film, and ensure they were able to project seamlessly.

1.1 Editing + Time

The use of editing is, by its very nature, a way of working with and manipulating time and space in film {and later video}. True, unedited film and video can only document events in “real time” - the only way to compress or expand these timeframes are to speed up or slow down the playback speed, or the amount of time each frame is displayed. As soon as a long block of footage is cut in 2 or more places, the possibility to communicate a similar narrative on a different timescale opens up. The middle “clip” of footage could be removed, and the 2 existing blocks joined together so that, when they are played back, the filmed image appears uninterrupted and the overall timeframe is shorter. Where these cuts are made, and what is removed, becomes an important part of this process when things like ideas and narratives are important.

Time, space and meaning can be further manipulated with editing once multiple blocks of different footage are cut and combined together. With this editing process, it is possible to communicate a narrative or information that appears to be occurring at the same time, but in reality the footage could have been filmed hours, days, weeks, months or years apart and the footage also could have been shot in completely different places. Editing presents an opportunity to convey and make meaning not only with the filmed images / subjects themselves, but the order of shots and images in relation to one another, and the speed and rhythm at which these shots are combined.

1.2 “Stop-Motion” Film

To pick up from last week, and jump into this week with editing, I would like to discuss a hybrid “time-lapse-animation-video” below by photographer Noah Kalina. This is what I would call “extreme time-lapse” - it is essentially a “frame animation” stringing together the selfie images taken by an individual once a day for TWENTY YEARS. I have a hard time doing anything once a day for more than twenty DAYS, but, that’s more of a me issue…anyway…

The resulting moving image is a combination of a frame animation, video montage and time-lapse. It is technically heavily edited, as it combines at least 7,000 separately captured individual images together - but since each photograph’s framing and subject are intentionally similar, they produce the illusion of a “time-lapse” and a frame animation. This form of editing and compilation produces a video where the first and last frame are taken 20 years apart. Please watch through this entire film - and note your emotional response to seeing a person age 20 years in about 8 minutes. Also think about the other elements of the images {such as changes to the subject’s surroundings and interior setting, the lighting, additional people / figures in some shots, etc} and how these also communicate meaning and information about the passage of time and a potential 20 year narrative.

Noah Every Day for 20 years

2. Early Montage + Editing Experimentation

Below is one of the earliest films to be projected in a theater-like setting. Louis and Auguste Lumiere captured this footage in 1895 and were able to project it in 1896 to a theater audience. This 50 second film contains no edits or camera movement, but creates a sense of depth, scale and an illusion of editing by working with a large, quickly moving initial subject, and then multiple subjects that enter and exit the frame. As these subjects travel closer or farther away from the camera, the frame composition changes enough that it feels as though there are separate shots.

This type of unedited shot is standard for these first projected films, however, the ability to physically cut, splice together, and then continuously play different strips of film was realized quickly. As early as 1903, Edwin S. Porter utilized an editing technique known as cross-cutting in the short western, “The Great Train Robbery”. This method uses two different clips played in succession to convey that the actions portrayed were occurring simultaneously in the film’s narrative. This film also exhibited one of the first close-up shots, with the actor turning their focus directly to the audience.

2.1 Sergei Eisenstein + Lev Kuleshov

As film editing technology and techniques continued to develop in the 1910s + 1920s, Russian filmmakers began to truly explore the sense of meaning and emotion that could be conveyed through edits themselves. The 2 filmmakers discussed below, Lev Kuleshov and Sergei Eisenstein, experimented with the meanings they could form by combining different images in succession, and how context could influence the viewers’ understanding of the overall film based on the order of images presented. Using this approach, Kuleshov especially developed a theory of forming meaning via different images based on not only their individual subject matter / content, but the order in which they were shown. This is known as the Kuleshov Effect - where the meaning inferred from one scene might shift based on the scene before and after it - and that this is a constant point of interaction / tension for film viewers.

The video below is required to view, and is covered by 1 question on the Async Lecture Quiz.

This effect is similar to something that we first discussed in this class in the collage module, and will be a recurring theme throughout this module. In the case of 2D collage, the original context and the new images both influence meaning. In this module, both video remix and rotoscope also work with this idea of creating meaning not only through the visuals presented, but in how the images might be altered or changed via the images displayed next to them, and/or the comparison to the original images they might be based on.

Additionally, Eisenstein worked with time in relation to editing as another way to influence and make meaning. He developed a theory of Five Types of Montage that could be used to communicate ideas to viewers - Metric, Rhythmic, Tonal, Overtonal + Intellectual. The video below describes these different types in detail, so, I will let them do most of the explaining, just be sure to note that there is overlap between all of these, especially metric + rhythmic and tonal, overtonal + intellectual. This video is covered by 2-3 questions on the Async Lecture Quiz - these questions can be answered even when skipping both of the sections noted below in the content warning.

CW#1: Eisenstein’s Five Methods of Montage below contains a graphic sequence from Requiem From a Dream, which depicts extreme physical and sexual violence and traumatic medical procedures. These begin at 1:17 and continue to 1:31 - please feel free to skip this part of the video, as it is the second example of Metric Montage.

CW#2: Scenes from the Godfather films depicting gun-violence are discussed from 4:09 to 4:20. This is less graphic than the content listed above, but is still fine to skip through.

These filmmakers and approaches were considered experimental and avant-garde {or extraordinary / unconventional } at the time. As demonstrated by the video above and the modern and contemporary examples it uses - these techniques were widely adopted by many filmmakers over time and incorporated as dominant methods of meaning-making and expression via editing, collage / montage and rhythm in film. Filmmakers continue to experiment and push these methods, gradually shifting the overall form, but the influence is still crucial to the overall development of film - and then video - as a critical form of expressive, narrative-based media.

3. Video Remix + Collage

When video was first introduced for consumer usage, it was somewhat more accessible and available to artists because of the more streamlined - but still analog - capture, editing and playback technologies. More critically, there was no longer a need to work with or develop physical film - users were able to view videos more immediately, and were able to collect video footage more easily, and also incorporate sound. Videos could also be re-written and reused, albeit with some loss in quality.

With the next round of technological advancements, video cameras were more handheld, analog effects were easier to apply than they were with physical film, and some basic editing and/or playback could be done through the camera itself, versus an analog editing bay. Similar, but smaller in scale, to the shift to digital photographic cameras, this created a creative space where many more people were recording videos, compiling / collecting videos, and developing their own media via video forms. This new media landscape led to experimentation, technological advancements, and impacts on mainstream film and visual culture.

Skipping many years of video media, design and art {this is part of the field being waaaayyyy too big to cover in a single module} this shift continued into the digital space. Not only did digital video become more accessible and more available to more people via digital video cameras, mobile phone technology, and digital video editing tools, digital video sharing and streaming platforms and social media applications created new distribution channels. All of these technologies continued to push the conventional forms of film and video, and also generated new forms of expression.

3.1 Digital “Montage” - YouTube, TikTok, Stories, + other Video Sharing Apps

Outside of more conventional time-based film and video artworks, new media and digital media formats also engage viewers and communicate meaning - sometimes unintentionally - based on the principles of film montage and editing discussed in the previous chapters. Many of these apps work with the concept of video collage or video remix, both in very intentional and much more random - or algorithmically generated - ways.

Most social media apps algorithmically generate video montages that can create or influence meaning in similar ways to the films discussed above. This was something introduced in the collage module, but from a more 2D perspective - what it means to be viewing digital journalism media, where multiple stories and visuals are competing for attentions, along with textual information, ads, notifications from other apps, other screens, and other media sources are also constantly in view.

Unlike most of the films and artworks covered in the previous sections, these “edits” or “cuts” are determined not by an artist or filmmaker, but by an algorithm that loads images and videos in a particular order. Each individual viewer’s preferences and personal settings partially influence this algorithm, however, they are also based on viewing and search history, along with many other factors, and are ultimately controlled by the platform or application. In these situations, especially with the shorter videos posted to TikTok and SnapChat, the algorithm is essentially playing the role of the editor / artist / designer, influencing the meaning and overall information that a user might draw from their scrolling experience. Additionally, and especially in cases of targeted content / advertising, these types of algorithms can dictate a user’s navigation.

Consider the last time you were scrolling on your platform of choice - how often do the images or videos in a post above and a post below another post with a video and/or image influence your understanding of all three? How would a different order change the way you use these apps? What happens when sponsored content interrupts your feed? How would you design these feeds if you were able to organize them completely on your own - and would you want to?

These are all important social stakes to consider, and I always spin out a bit thinking about how editing techniques and montage theory first developed 100 years ago, is still incredibly relevant in our everyday lives, and also that some or a lot of our media landscape is determined by black-box AI {artificial intelligence developed using deep learning, where training data and outcomes are controlled / validated by the designer, but the logic the system uses to come to conclusions is unknown}. What are the implications of leaving design decisions that are known to create / form / shape meaning up to something that is not totally understood or controlled? We might be seeing / experiencing these impacts as we try to understand the world that digital media and social media have monumentally changed in the past 5, 10, 15 and 20 years.

YouTube itself, as a platform, is also a video montage like TikTok and SnapChat. As soon as it starts to “Autoplay” videos based on previous views and user data, recommend videos, determine which ads to play, or what other videos to list in the user view, it is algorithmically generating a montage / video collage of some kind.

3.2 Youtube as an Archive / Medium - Intentionally Designed Video Montage

Many artists and filmmakers have also utilized YouTube and other Video Sharing Platforms to source and generate intentional montages, a few forms of which I’d like to focus on a bit more below. These are different from the examples above because there is an artist - not a program - designing the edits, the timing, and the visuals and content of the remixed or collaged original footage. The examples below also work with specific editing techniques to communicate and explore ideas about time and documenting time - bringing us back to another main theme of this module.

Life In A Day

Life in a Day (2011) directed by Kevin MacDonald is a feature length film compiled entirely of videos shot and uploaded to YouTube on July 24th, 2010. As you might have guessed, this film is available on YouTube in its entirety.

1SE - One Second Every Day

While the two are not directly related, when the 1 Second Everyday (1SE) was released, I immediately thought of Life in a Day - its kind of like the inverse method working with similar concepts. This app cuts together one second of video from each day of the year.

BeReal

This is a more recent social app that collages photos and videos of users from both the front-facing and back-facing cameras on mobile devices, at random times throughout each day. This type of app / sharing creates a more “authentic” space, and uses the minimum two-camera tech of most mobile devices to create “collaged” images that provide additional context, as well as uncertainty / synchronicity / surprise / chance to a visual social media landscape that can be very curated / designed.

3.3 Video Streamers and VTubers

This is a newer category of video, video collage / remix and in some cases animation / rotoscoping that I’m using as a point of transition into rotoscoping animation in the last chapter of this module. VTubers are performers on YouTube and Twitch that interact with a wide audience via video streaming, often interacting with other forms of media.

These streams are highly complex examples of digital media collage and interactive media, games and performance. The visuals typically include at least two main streams of content - the Vtuber performer, interacting with directly commenting on another form of media, usually a video, film, or game. Additionally, there is a constant stream of interaction via the audience through text-based chat, which also contributes to the visual, narrative, and/or interactive landscape. On top of all of this, many VTubers perform via animated / motion-capture avatars, another, more evident / visible level of digital mediation between the artist and audience, as well as another layer of visual design influencing and informing the meaning of everything else within and/or adjacent to the frame.

From a video remix standpoint, these are layers of interaction, collage, remix and meaning making that shape and form new meanings. Even just considering the two examples below, how does the VTuber’s commentary or gameplay, the juxtaposition of their avatars, and the interaction of the chat / commentary, all work to form new meanings around the original film or game being featured? These are media forms and artworks that I do not believe could be imagined in the 1910’s or 1920’s, but are still being highly influenced by art movements like Dada-ist 2D collage, film montage, surrealism, and performance video artworks from the 1970’s and 1980’s.

Moving forward in this module, I also believe a connection could be drawn between this use of avatars and rotoscoping, or, at least between this idea of incorporating animation and video to change, shape and/or inform meaning.

4. Rotoscope + Remix

Rotoscoping is one technique used by animators and artists to easily capture, trace and animate movement and motion filmed in "real-life" {IRL}. Conventionally, rotoscoping is based on photographic frames of video, film, or even still image sequence photographs - and the new meanings that can be formed through this transformation process. I believe it is less interesting to rotoscope something that is already a 2D animation in most cases, unless it is being done to intervene with a conventional animation style or set of conventions, and, along similar lines, it would be interesting to hand draw or trace 3D animation frames, programmatically generated sprite animations, or even generative AI video.

4.1 Rotoscoping Foundations

Rotoscoping is a technique originally explored in the 1900’s, where artists would trace photographic images by painting them on a plane of glass. This was originally so that artists could project those “moving” photographic images, since the film technology had not yet been developed. I described this process a bit more in the first module, but this was the process that Muybridge needed to utilize in order to first project / exhibit his images of a filmed horse in motion. An artist was hired to paint over the photographic images on a disc of glass with more transparent paint, so that the images could actually be projected via the technologies available at the time.

Rotoscoping continued to be utilized as an animation form after the development of film projection technology, as a way to accurately depict and animate realistic motion, however, it did not become a widely adopted technique for animators or animation studios. While animators still base motion and animations on real-life observations, they are primarily used as only a starting point for developing the more stylized movements of animated characters.

4.2 Analog Rotoscope

Before the rotoscoping process was digitized, which made it a much more accessible process to more artists, filmmakers and animators, there were a few notable examples of rotoscoped animations that were created for mainstream audiences. From the examples below, consider why the animator or film maker decided to work with rotoscoping as a technique, and what impact this chosen style has on the final piece.

Below a quick fast-forward to the recent Weezer music video that might look familiar….

For the examples below from the 1990's Prince of Persia series, focus on the animation and action sequences, not the content, which is questionable in terms of cultural appropriation and representation. The use of rotoscope produced animated visuals that were very lifelike and unique at the time - it completely blew my mind, even playing it on a small 8-bit Gameboy LCD screen.

A related {non-analog} process was used for a much much more recent game, Motion Twin’s 2018 Dead Cells. For these visuals, the assets were rendered and animated using 3D software, and then an automated pixel-art texture map / shader was applied to the 3D forms. These assets were then rendered and framed via a camera setup that simulated 2D movement, and the those frames were used to create the sprite sheets for the animations in the final game.

4.3 Rotoscope in Modern Film

1990's to 2000's

In the late 1990’s, rotoscope technology entered into the digital space with automated processes that were much more accessible to artists and filmmakers. This included the ability to automatically convert video frames, which are pixel-based, into a sequence of vector-based image files which could be further manually processed, edited, and adjusted. These vector based files are more visually similar to animation than photographic frames, and produced an interesting visual effect that, for the time, was very unique.

This digital process was applied to film footage in order to produce the 2001 feature-length film Waking Life. The film’s subject matter explored various philosophical concepts, including the distinction between the “reality” and “imagination” and the use of the rotoscope technique worked well to represent these ideas visually. The captured footage, including the camera movements and the faces, expressions and movements of the actors was photorealistic, and rooted in reality, while the visual style, and the abilities to augment these images - sometimes subtly, and other times drastically - produces uncanny, surreal and fantastical images and situations.

Just a few years later, the film A Scanner Darkly picked up where Waking Life left off in terms of the progression of more mainstream films using the roto-scope technique, this time utilizing the medium to work with a Sci-Fi theme. Again, the rotoscoping processes produced visuals that were both realistic AND fantastic, lending themselves to the imagined science fiction world.

Contemporary Examples

I would argue that as this automated technology became more and more accessible and more widespread, the impact of the visuals themselves on the audience has changed. For these earlier films, just the effect alone was enough to evoke a sense of wonder and awe among viewers {along with some reports of motion sickness} similar to how early 3D animations were so captivating even though they were often of simple objects. As the technology has improved, the visuals, detached from content, have become more commonplace, so, animators and filmmakers have had to develop different styles and approaches to the medium and the process in order to return the viewer to that place of the uncanny, powerful image.

This can be seen in the examples below of films that utilize rotoscoping techniques and push the medium beyond just the technical process or visual enhancement. Consider how these artists work with rotoscoping, but also apply strong, unique styles to their cinematic visuals. Consider how these types of transformations might change or impact the meaning for the audience, compared to the earlier digital rotoscope films that worked with a more consistent style heavily connected to the available technology?

Last Day of Freedom

Below is a clip from Dee Hibbert Jones' short Last Day of Freedom (2015). This Oscar nominated documentary short uses roto-scope techniques for both stylistic and conceptual reasons.

Below is a link to a short interview where Jones talks about this technique in relation to race and representation in film - this article is optional, but highly recommended.

https://www.fullframefest.org/2015/11/race-representation-and-rotoscoping-dee-hibbert-jones-and-nomi-talisman-on-last-day-of-freedom/

ESPN “30 x 30 Hoodies Up” Preview Spot

The animation below {click on image to link to full animation} is different from most rotoscopes in that the figures are highly abstract and simplified, and the motion that is captured is primarily that of the “camera”, which is actually achieved in an automated process called keyframe animation. The reason why I am showing this as an example, however, is because the figures are based on original photographs and “real-life” video footage, and then identifying characteristics and details are obscured to emphasize the film’s overall narrative, which focuses on violence against communities of color by police and other law enforcement agencies, both public and private. This is another example of how the technical aspects of rotoscoping can be used to influence meaning in animations beyond just visual style and aesthetics.

Loving Vincent

The trailer below is for the 2017 film Loving Vincent, centered around the life and artwork of painter Vincent van Gogh. This animated film worked with a rotoscope process that was designed to either directly reference his specific paintings in some cases, and general visual painting and drawing style throughout the entire film. This more recent example of a rotoscope technique demonstrates how artists and filmmakers are pushing beyond previous technological limitations with this digital technology to produce more unique and intentionally designed aesthetics that fully inform other aspects of the film and its overall meaning.

4.4 Rotoscope Design Considerations

As the previous examples all begin to demonstrate, there are a lot of ways to work with the rotoscope technique in animation, and use it to explore, transform, and or recontextualize the motion and visuals captured by film and video. Below are a few different design approaches / affordances that animators might work with when developing rotoscope animations. This list is in no way extensive, and there are many areas of overlap. These could be some starting off points to consider when planning out your rotoscope project for this module.

Accuracy and Level of Detail / Fidelity

The traces can be very precise and true to the photographic quality of the stills, or they can be more loosely or abstractly applied. This accuracy level can vary or fluctuate throughout the animation’s duration, or remain consistent. Some designers use rotoscoping as a way of animating complex IRL effects, textures, motions, etc in a high-level of detail or fidelity. Others might actually use rotoscope to simplify high-fidelity visuals.

Visual Style

There are a multitude of styles that can be applied to rotoscopes. Subjects can be composed by points, lines or shapes or a combo of all three. The properties of these elements can also be altered - lines can be thick or thin, neat and precise or more messy and sketched out. Shapes can be geometric or organic, complex or simple, can look more like a realistically rendered painting or like a stencil or a more graphic image. The more stylized something is, the less “realistic” it appears, but something that is stylized can still be high-fidelity, or it can be lo-fidelity, and have less details.

Color, Lighting, + Tone

Color and tonal effects can be applied to rotoscopes in many different ways, and lighting effects that were not present in the original video footage can be added to rotoscope animations. A designer can decide to reduce the number of colors in the original footage, apply a completely different color palette / scheme to the animation or remove color and add things like atmospheric fog or haze.

Edit / Transform Original Content

When working with original video or filmed footage, the animator can decide which elements and subject matter to include in the final animation. This type of decision has the potential to greatly influence or change the narrative meaning of the rotoscope animation, in part because that transformation, exclusion or change might be evident. This is another way that rotoscope can be directly connected to the themes of collage and remix.

Envision a rotoscope where only the shadows of moving subjects were traced, or one that redraws a static background ”around” the negative space of a moving figure or subject. Or a rotoscope that works with a well-known visual scene, and removes a subject or inserts a totally different subject into that space.

Camera Movement and Complex Cinematography

For some rotoscoping animations, if the animator is only interested in capturing the moving subject, the composition of the frame does not matter as much, since the drawings can be composed and placed into whatever frame dimensions are desired. However, rotoscoping can also offer the opportunity to translate complex camera movements into an animated form, which can be very challenging with traditional frame animation. This can be something to consider when sourcing out video or planning on taking your own videos for your projects.

4.5 Rotoscope Comparisons

The two video / rotoscope pairings below are excellent examples of the many different directions designers can go when working with rotoscoping. For each of these, watch the original video first, then the corresponding rotoscope animation. Note how the rotoscoped animations differ from the original videos, and also how each rotoscope artist works with rotoscope in intentional ways, some similar, and some very different. How do these design decisions inform the meaning of each individual animation, and how do they possibly change / transform or recontextualize the meaning of the original video?

Each of these comparisons is covered by one question on the Async Lecture Review Quiz, for two total. These questions also refer to the Design Considerations listed above, in Section 4.4

Sia’s “Chandelier” Music Video {2014} + “Chandelier” Rotoscope Animation by Unknown Artist

Twenty One Pilots’ “Level of Concern” Music Video {2020} and “Level of Concern Lyric” Music Video - Rotoscope {2020}, by animator Pinot Ichwandardi

Before creating this music video in 2020, Ichwandardi first went viral in broader internet culture by re-creating rotoscoped scenes from Childish Gambino’s dance choreography from the “This is America” music video using a similar approach. Instead of using cutting edge rotoscoping technology to produce all of these visuals, Ichwandardi instead works with older creative technologies, such as the 30 year old Apple SE running what I think is animation software that eventually turned into Macromedia / Adobe Flash.

Through this approach, Ichwandardi is using animation and rotoscope as a way to not only recontextualize the visual styles of video / filmed subjects, but to also remix their creation / production technologies from high-resolution video to very low fidelity, minimal, yet highly impactful and expressive visuals that also communicate a sense of time via those visual cues. I will hopefully be able to cover this artist more in future modules, but for now, check out Ichwandardi’s Twitter - his animation experiments are incredibly inspiring {optional}.

5. Module Wrap-Up

This will be the last module extensively covering time, motion, film + animation, so I wanted to end with a relatively short video essay that I believe offers another little window / jumping-off point into all of the design topics we have covered over the past week. The essay below breaks down a scene from the film Dune: Part 2, and within this visual analysis, covers aspects of montage, collage, and motion design, and also connects to the larger mega-topics film, video, and animation.

The tools the video essay’s author uses to create this video and aid in the frame by frame breakdown of this scene also relate to rotoscoping and remix, and, finally, the platform that is used to distribute and share this video, also connects back to remix and author-to-audience interaction via the comments and other use feedback, and also how these types of videos overall change the way we might understand the original film. This short video is required to watch, and is covered by 1 question in the async lecture content quiz, but technically the answers are also contained within the paragraph above. I mostly just wanted to share this video and the visual design that goes into film, and hopefully relate this back to the last two modules as we continue to understand all the different design choices and decisions we make as artists, and how they inform, influence, and create meaning.