Clint Enns

Navigating Algorithmic Editing: Algorithmic Editing as an Alternative Approach to Database Cinema

“Navigating Algorithmic Editing: Algorithmic Editing as an Alternative Approach to Database Cinema.” Millennium Film Journal 56 (Fall 2012): 66-72.

Ideas are not separable from an autonomous sequence or sequencing of ideas in thought that Spinoza calls concatenatio. This concatenation of signs unites form and material, constituting thought as a spiritual automaton.

D.N. Rodowick, The Virtual Life of Film

Database cinema, as introduced by Lev Manovich in The Language of New Media, is a new media form that takes advantage of the computer’s ability to manipulate, analyze, organize and arrange multimedia data. Being less than efficient, traditional video editing software is not the ideal platform for producing database cinema. Despite the fact that video editing software systems allow for direct access to any frame without requiring the sequential navigation through adjacent footage, they are still heavily rooted in a film-based editing paradigm. Database cinema borrows one of its key concepts from computer science: namely, how the computer accesses its database – algorithms. Many artists are already learning from and exploiting the computer’s relationship to the database through the use of a technique called algorithmic editing.

As with most new media, algorithmic editing is not new and its roots can be seen in the earliest attempts to formalize/ theorize the practice of cinematic editing. Programming software to interact with and manipulate the digital file and the database provides the artist with direct access to and insight into the files themselves, naturally connecting algorithmic editing to the aesthetic tradition of materialism. In addition, theorizing about algorithmic editing offers new critical cultural insight and the practice of algorithmic editing offers the potential to address, re-invert and subvert the medium.

Algorithmic editing is a term that was first coined by Lev Manovich in an artist statement for Soft Cinema (2002), a collaborative project with Andreas Kratky that attempted to navigate the database in new and innovative ways. In the artist statement, Manovich theorizes about algorithmic editing without providing a concrete definition. Explicitly, algorithmic editing refers to any method of editing based on direct procedural approaches. In other words, algorithmic editing can be seen as a technique for cutting and reassembling raw footage by following a schema or score. Here is an example of a simple, one line algorithm that could be used to algorithmically edit a film. Sequentially use every odd frame from one sequence of film and every even frame from another sequence of film to assemble a new film which alternates between the odd and the even frames. The resulting algorithmically edited flicker film would rapidly alternate between the two sequences. Creating this film using two filmstrips would be difficult without the labored use of an optical printer. On the other hand, producing this film from two video sequences would be difficult without the use of a script or specially made plug-in; nevertheless, a script for this algorithm would only consist of two or tJiree lines of code.

In a broader context, algorithmic art is produced by following an algorithmic process, that is, it is art produced by following a finite list of well-defined instructions or by following a procedure. Although the use of computers is usually associated with algorithmic art, computers are not an essential part of the process. On the other hand, algorithms are essential to computer operation. That is, computer software is merely a collection of computer programs, and computer programs are simply computer algorithms that process and manipulate data.

Despite the fact that algorithms follow a stepby-step procedure, the user cannot expect the same output every time; this is because algorithms often contain random or pseudo-random processes. Equally important, the computer can be programmed to produce results that are unexpected, a feature that is often exploited by artists creating algorithmic art. Finally, algorithms are often designed to require input from the user in order to perform their tasks, allowing the user to maintain at least the semblance of control and providing the computer artist with a sense of authorship – which some might suggest has been lost in the transition from film to the digital.

Though Manovich coined the term, it can be argued that algorithmic editing traces back to Soviet montage theory and was later developed through the work of structural filmmakers in the late ’60s and early ’70s. By treating film as a countable and measurable entity, filmmakers such as Sergei Eisenstein and Dziga Vertov used rhythm, as well as formal content, to develop simple editing structures. In the late ’60s and early ’70s, many experimental filmmakers began to use simple schema to edit their films. In addition, filmmakers were beginning to experiment with the optical printer, a device that, in some cases, allowed for the creation of slightly more complex schema through the use of a programmable sequencer.

Editing chart employed by Dziga Vertov.

Currently, through the use of a computer, artists are able to create database works using fairly complex schema by combining algorithmic editing with found footage practices. Algorithmic editing allows the artist to take advantage of the computer’s other functions and abilities, in addition to the enormity of digitally available databases. Moreover, many artists are designing software to algorithmically edit footage, both original and found. In this article, the advantages of algorithmic editing, relevant examples of algorithmic editing software, as well as database cinema created using digital algorithmic editing will be thoroughly examined and discussed.

In The Language of New Media, Manovich asks, “how can our new abilities to store vast amounts of data, to automatically classify, index, link, search and instantly retrieve it lead to new kinds of narratives?”1 In order to answer this question, I propose that we need an intermediate step – a method for converting the database structure into a narrative structure. As Manovich has suggested, “once digitized, the data has to be cleaned up, organized, and indexed. The computer age brought with it a new cultural algorithm: reality -> media -> data -> database.”2 By developing Manovich’s cultural algorithm further, I suggest expanding this diagram to: reality -> media -> data -> database -> algorithmic editing -> a new narrative form. To Manovich, cinema “is the intersection between database and narrative,”3 therefore, die expansion of the database must lead to more innovative and complex narratives. By using a computer, the artist is able to create editing schema that is far more complex than those of his/her predecessors including filmmakers like Eisenstein, Vertov, Kubelka, Sharits, Iimura, and Kren.

As early as 1974, Malcolm Le Grice noted that computers “are ideally suited to dealing with complex relationships of data precisely and very rapidly, and they are being developed towards highly efficient indexing and retrieval capability.”4 In fact, this trend has continued and is precisely the reason computers are ideal for creating database cinema. Currently, we are at the point where computer users have the ability to retrieve multimedia information from enormous, well-indexed databases. In Experimental Cinema in the Digital Age, Le Grice conjectures that the three most important functions performed by the computer in relation to cinema are “systems of incrementation, permutation and random number generation.”5 To this I would add, the computer’s ability to access large amounts of data; its ability to manipulate and analyze data; and its ability to efficiently copy and paste. On the other hand, efficient and effectively made cinema isn’t necessarily cinematic. This brings me to a question raised by Barbara Lattanzi in her essay of the same title, namely, “what is so cinematic about software?”6

Lattanzi is a new media artist who develops her own original, open-source software to algorithmically edit films from a database. Lattanzi uses software consisting of “simple, dynamically-modifiable algorithms” to encode and emulate editing techniques of seminal avant-garde films.6 Her software, AMG Strain (2002), HF Critical Mass (2002) and EG Serene (2002), emulates the editing schema of Anne McGuire’s Strain Andromeda The (1992), Hollis Frampton’s Critical Mass (1971) and Ernie Gehr’s Serene Velocity (1970) respectively. By referencing other work, Lattanzi observes that “the simulation of film structure in a software algorithm – where the software becomes referential to a specific film experience – paradoxically registers a narrative in the algorithm, a narrative with concrete reference within an abstraction.”7 I would argue that reference to another film’s structure is not enough to produce narrative (although it is enough to enter into a cultural dialogue with the original artists). On the other hand, the clips the software is referred to – the database – are extremely important and play a key role in contributing to the narrative or content of the film. In other words, as Lattanzi observed, the “Software is not narrativising in itself.” She continues, “Software is not about something. Software performs something.”7 In the case of Lattanzi’s software, it performs something to a specific video clip from a database of video clips, and the choice of the clip is significant. For instance, Critical Mass, Serene Velocity and Strain Andromeda The are considered works of cultural significance precisely because the artists took careful consideration of the content to which they applied their editing schema. The editing schema in and of itself was not enough.

One of the first works explicitly considered to be algorithmically edited is Manovich and Kratky’s Sofi Cinema, which can be viewed as a theoretical investigation into the different types of narratives database cinema offers. Despite being one of the first artworks to explicitly make use of algorithmic editing, the work ultimately fails due to the seemingly arbitrary nature of the editing. Although the clips are associated through keywords that describe and account for their content and formal properties, it is impossible for the viewer to decipher the underlying logic being employed, thus making the clip selection appear arbitrary. On the other hand, experimenting with database cinema allowed Manovich to theorize about the potential of algorithmic editing:

Different systems of rules are possible. For instance, one system selects clips closest in colour, or type of motion to a previous one; another matches the previous clip in content and partially in colour, replacing only every other clip to create kind of parallel montage sequence, and on and on.8

In this description, Manovich alludes to the computer’s ability to edit according to specific analyzable properties in the clips, for instance, color, motion, sound, etc. An example of an algorithmically edited work that takes advantage of this ability is Cory Arcangel’s Drei Khvierstücke op. 11 (2009).

Cory Arcangel’s Drei Khvierstücke op. 11 (2009).

In Drei Klavierstücke op. 11, Arcángel humorously reassembles Arnold Schoenberg’s 1909 Drei Khvierstücke, Op. 11 from various videos of cats playing piano found on YouTube. Drei Klavierstücke is considered to be among the very first true pieces of atonal music – a form of music that does not conform to any particular key signature. Despite the technical knowledge required to both perform and compose atonal works, to the unsophisticated listener, atonal works may sound like a cat playing the piano, an act that is undeniably cute and hence requires recording and uploading to YouTube. Ironically, the act of creating an atonal work from videos of cats playing the piano is a fairly sophisticated process which Arcángel informally described on his website as follows:

So, I probably made this video the most backwards and bone headed way possible, but I am a hacker in the traditional definition of someone who glues together ugly code and not a programmer. For this project, I used some programs to help me save time in finding the right cats. Anyway, first I downloaded every video of a cat playing piano I could find on YouTube. I ended up with about 170 videos. Then I extracted die audio from each, pasted these files end to end, and then pasted this huge file onto the end of an audio file of Glenn Gould playing op. 11. I loaded this file into Comparisonics. Comparisonics, a strange free program I found while surfing one night, allows users to highlight a section of audio, and responds by finding “similar” sounding areas in rest of the audio file. Using Comparisioncs [sic] I went through every “note” (sometimes I also did clusters of notes) in the Gould. I then selected my favourite “similar” section Comparisonics suggested and wrote it in the score. After going through die 1000’s of “notes,” the completed scores were turned into a video by some PERL scripts I wrote which are available here if you wanna do something similar.9

Cory has since improved this technique for his most recent video Paganini’s 5th Caprice (2011) – a video composition of Paganini’s early 19th century musical composition Caprice No. 5 reconstructed from hundreds of different instructional videos for guitar found on YouTube. By re-using his code for Drei Klavierstücke op. 11, Arcangel demonstrates the importance of code reusability to algorithmic editing. As previously observed, these works also demonstrate another important aspect of algorithmic editing, die computer’s ability to analyze clips. Arcangel analyzes the sound found in various YouTube clips and edits the appropriate clips together according to a schema, namely, a musical score. Expanding on this, it is possible to analyze clips from extremely large databases, like YouTube, for other properties and then algorithmically edit the clips together according to a schema.

Finally, Doug Goodwin’s Mersenne Devil Twister (2011) is a simple video sketch that edits together four to twelve frame sequences selected randomly from a clip in a process Goodwin describes as “desequencing.”(Doug Goodwin, “mersenne video twister,” cairndesign: since 1996 (March 4, 2011).)) By writing code to algorithmically edit a video segment, in this case a clip from Peter Yates’ Bullitt (1968), Goodwin produces a segment whose movement is jarring and unusual yet strangely memorizing, compelling and beautiful. The title itself refers to Nic Collins album Devil’s Music (1985), which was created using a similar technique applied to sound clips and the Mersenne twister – an algorithm for generating pseudo-random numbers. In using this title, Goodwin reveals both his influence and the process involved in the making of the video. Finally, Goodwin made the code available on his website, thereby encouraging others to further develop this technique.

As observed by Amos Vogel in Film as a Subversive Art, the “avant-garde offers no solutions or programmatic statements, but a series of intricate challenges, hints, and coded messages, subverting both form and content.”10 Despite involving programmatic statements in the form of code, algorithmic editing in the digital age can be seen as one technique extending contemporary avant-garde practices and concerns. I’d like to explore some of the contemporary issues that artists are engaging with, either implicitly or explicitly, by applying the techniques of algorithmic editing to the database.

There are some theorists who believe that the interfaces imposed by digital practices restrict creativity. Although this may be true of commercial editing software, it is certainly not true for artists writing their own software and experimenting with algorithms and code. The computer, like the optical printer, is a powerful tool that in the hands of a creative artist can be used to generate engaging work. It would be considered ridiculous to argue that the interface imposed by the optical printer or the Bolex restricts creativity, and it is equally ridiculous to argue that all digital interfaces restrict creativity.

Through writing code and video editing tools the artist is able to critique industrial modes of filmmaking, both in terms of the tools they employ and in terms of content they are generating. Most commercial video editing software attempts to hide the algorithms it employs and is unmodifiable. As Barbara Lattanzi states in an interview with artist Keiko Sei:

I would rather make my own software (what I term idiomorphic software), because the commercial software that I use comes at a price. That price has less to do with money and more to do with a different process of abstraction: the active framing of my work within considerations dictated by irrelevant practices of Design. I make clear with students that I am not interested in their Design clarity and precision, but in their discovering productive ambiguities.11

In this statement, Lattanzi points out the role that errors and mistakes play in the artistic process, something commercial software tries to eliminate. This implies that, in spite of embracing a systematic approach to film, algorithmic artists are also embracing errors and imperfection. This simple act can be seen as subversive since our society historically and presently strives for perfection. In fact, this is one of consumer myths that capitalism is based upon, namely, the myth that newer and sleeker is better.

Knowledge sharing is also an important part of the culture. As observed by Tom McCormick, “… glitchers seem eager to share their strategies. Part of this probably has to do with the fact that many new media artists are code junkies who come directly out of the open source movement; but then the open source movement may have equal roots in functional programming and media art.”12 This sentiment is reenforced by Arcángel and Goodwin’s eagerness to share their processes and Lattanzi’s strictly open source policies. Conceptually, this act carries with it all of the political motivations of the open source movement, however, it also reveals the importance of content since everyone potentially has access to the same processes and techniques. Open source as a pragmatic methodology, is inherently a subversive practice because it promotes cooperation, collaboration, community and removes profit incentives.

Despite the cheapening of process-based work due to the reusability of code, the database takes on a new and heightened value since the content of an artwork is, at least partially, dependent on database choices. Database cinema is planted firmly in the continuum of found footage filmmaking. Although there are many positive aspects to this – as Michael Zryd, for instance, suggests in his article Found Footage as Discursive Metahistory: “the etymology of the phrase [found footage] suggests its devotion to uncovering ‘hidden meanings’ in film material” – it also raises questions about copyright/ ownership of the sources being employed.13 Many artists blatantly ignore copyright issues. For instance, it can be assumed that Arcángel does not ask individual users for permission to use their clips when he uses YouTube as database, though he does acknowledge his source videos, transforming the original authors into unknowing collaborators.

In our society, algorithmic editing as an approach to the digital database is actually constantly being used in invisible ways. For example, it is the current template for many television news channels. News stations bombard the screen with information obtained from different databases. Current world news, in the form of text, runs across the bottom of the screen, in addition to information about the weather, time and the stock market. It can be assumed that the station is accessing this information from various databases and that the station does not research all of the stories they are broadcasting, despite presenting these stories as news. Finally, many news stations are potentially accessing the same databases, thus the news being provided only represents a single perspective.

Algorithmic editing techniques are also being applied to internet search engines in an attempt to provide user-specific content. In The Filter Bubble: What the Internet is Hiding From You, Eli Pariser developed and explored a controversial concept sharing the same name. This concept addresses some of the negative effects of generating user-specific content based on our past viewing behaviours. Through the filtering of information, determined by capitalist interests, Pariser is suggesting that a bubble is formed around individual users, which inhibits intellectual growth by not exposing the user to ideas conflicting with their own ideology and by not necessarily providing die user with the most accurate information.

By experimenting with algorithmic editing, the artist is investigating a concept that is informing and framing the culture in which they live. Through this exploration, artists are able to provide insight into these processes, and, at the very least, are able to reveal and demystify them. By understanding algorithmic editing, the artist is able to provide social and cultural critique.

As noted by Barbara Lattanzi:

The Cultural Producer who samples form the raging flows of media detritus – endless satellite feeds, cable and broadcast transmissions, and the sedimentary layers of these through the past 25-50 years – becomes the heroic Luther, wresting deconstructive (re)form(ations)s out of the desultory, formless industrial wasteland. Deconstructive film and video-making demonstrate the inherent formlessness of mass media by making it into the “New Nature.”14

To expand on this, it is not only the database that artists are deconstructing, it is also the techniques used to access the database. Furthermore, by technically understanding algorithmic editing, artists can renvent and subvert the role that it plays in traditional applications.

Conclusion

Algorithmic editing as an alternative approach to the digital database is still in its infancy and the examples explored in this paper demonstrate the potential these techniques have, not only in terms of aesthetic considerations, but also in terms of social and cultural critique. By experimenting with algorithmic editing, observing the use of algorithmic editing as it functions in the world around us, and theorizing about algorithmic editing, especially in its relationship to the digital database, artists and theorists are be able to offer new critical insight into the effects of algorithmic editing upon our society. By addressing the social effects of the medium and by understanding the medium itself, artists possess the ability to transform our society by demystifying, recontextualizing and potentially reinventing the medium itself.

  1. Lev Manovich, The Language of New Media (Cambridge: MIT Press, 2001), 237. [↩]
  2. Ibid., 224. [↩]
  3. Ibid., 237. [↩]
  4. Malcolm Le Grice, Experimental Cinema in the Digital Age (London: BFI, 2001 ), 220. [↩]
  5. Malcolm Le Grice, Experimental Cinema in the Digital Age (London: BFI, 2001 ), 220. [↩]
  6. Barbara Lattanzi, “What Is So Cinematic About Software?” (Presented at “Connectivity,” Connecticut College, New London, CT, March 31 -April 1, 2006). [↩] [↩]
  7. Barbara Lattanzi, “Critical Mass, the Software,” (Presented at “Gloria! The Legacy of Hollis Frampton,” Princeton University, Princeton, NJ, November 5-6, 2004). [↩] [↩]
  8. Lev Manovich, Soft Cinema (Karlsruhe: TKM, 2002-2003), 5. [↩]
  9. Arcangel, Cory. “Drei Klavierstücke Op. 11 (2009).” Cory Arcangel’s Internet Portfolio Website and Portal (2009). [↩]
  10. Amos Vogel, Film as a Subversive Art (London: Distributed Art Publishers/CT Editions, 2005), 308. [↩]
  11. Sei, Keiko. “Productive Unclarities: Interview with Media Artist Barbara Lattanzi.” Springerin Magazine, no. 4 (December 2001). [↩]
  12. Tom McCormack, “Code Eroded: At GLI. TC/H 201 0, RHIZOME, Oct. 2010,” in GLI.TC/H READER[ROR] 20111, eds. Nick Briz, Evan Meaney, Rosa Menkman, William Robertson, Jon Satrom and Jessica Westbrook (Chicago: Unsorted Books, 2011), 16. [↩]
  13. Michael Zryd, “Found Footage as Discursive Metahistory: Craig Baldwin’s Tribulation 99,” The Moving Image vol. 3, no. 2 (Fall 2003), 41. [↩]
  14. Barbara Lattanzi, “We Are All Projectionists,” Millennium Film Journal, No. 39/40 (Winter 2003), 84. [↩]