Role: Creative Director & Creative Technologist
Timeline: 1 month

Curated compositions trigger AR image targets to reveal hidden intent in film.

Augmented Cinema is a collaborative research project by Faith Kim and me, at the intersection of media arts, cinema and interaction design. We were interested in studying AR and its more common uses, to ultimately expand it to a previously unventured realm: Cinema. Prior work has been done with AR in theater, where the audience was prompted to reveal objects through their phones, however nothing yet with film. This prototype is a technical effort to embed new user experiences within traditional filmmaking.

Historically, AR has worked by detecting a specific set of image targets. Nowadays it is preferable to create content with floor detection or personally tailored images. Nevertheless, technologies surrounding AR are still novel and rarely offer a subtle way of merging image targets and content.

These are the main research questions we dealt with in this project. How can we create a richer experience in film? How can we make a more genuine and less intrusive interaction out of augmented reality? With these questions in mind, we constructed physical prompts within the aesthetic language of film by creating embedded visual cues. These cues, built from frame composition, color and choreography, trigger AR sequences seamlessly without removing from the beauty of cinematography.

Video for proof of concept. Technology, aesthetics and animations are still in preliminary prototyping phases. 


There is an enormous underlying narrative potential for augmented reality in cinema. AR could be used by directors to create sub-plots and internal dialogues or to express certain moods and feelings. It could even prompt an audience to superimpose their own perceptions to the film and share them with each other. In film, AR could serve as a tool to add complexity to an intrigue, but first and foremost, it would allow viewers to engage with cinema in a refreshing and meaningful way.

Hypothetical use of AR to suggest a character’s dreamscape
or to insert clues into the narrative


1.0 FIRST DEMONSTRATION OF TECH: Enhancing Pre-filmed Scenes through Augemented Reality using the Vuforia Ground Plane.




After a successful trial without using image targets (i.e. use of ground plane), we decided to expand our project to a more complex targeting system. We started prototyping with multiple targets, which would be timed perfectly with specific shots. We achieved this by curating very specific layouts within our short film in order to generate significanlty different images which could then be used as the targets themselves. One great advantage of this technique is the subtle use of image targets, which is still uncommon in augmented reality.

We instructed our actress, Sarah Ssemakula, on the specificities of how she should be moving the objects within the scene. Our short film depicts a woman’s morning routine in a diner: Drinking coffee, reading the newspaper, considering breakfast choices. The scenes are cut quickly and do not represent a linear narrative. Music by the amazing Conner Harden.

Filming at Ritter’s Dinner (Bloomfield, Pittsburgh)

First technical success with image targets


Technical debugging

IN CONCLUSION, our project ended on a successful note, giving us the opportunity to further explore the posibilities of augmented cinema. Although the technology worked statisfactorily, there is now a need to consider the user experience of augmented cinema. What would make this experience more valuable than classical cinema? How can we provide deeper meaning in plot through augmented reality? These questions will be addressed over the course of the next year for my senior capstone project.

© Copyright 2020 Chloé Desaulles. All Rights Reserved.