Augmented Reality and Nonlinear History in “Ghost Game”

by Ray Dweck                              Published November 25, 2015


“Ghost Game: Spiritus Chronomatix” is an interactive storytelling game developed by multimedia artists at Tisch School of the Arts and the Polytechnic School of Engineering, showcased last month at Tisch in commemoration of the school’s 50th anniversary.  The game, a detective-like investigation of real-world environments aided by digital technology, invites players to uncover secrets from a tragic yet significant episode of Tisch’s prehistory dating back to 1911, when nearly 150 garment workers perished in a massive fire at the nearby Triangle Shirtwaist Factory building (currently, New York University’s Silver Center of Arts and Science). 1 2


Triangle Shirtwaist Factory building (New York)

Taking place in both real and virtual environments, “Ghost Game” engages participants in a digitally-enhanced perception of their surroundings by way of “augmented reality” technology.  In augmented reality (AR), real-world objects are viewed through a video screen that alters their appearance using superimposed computer-generated imagery (CGI) in real-time.  Distinct from virtual reality (VR), in which users perceive a self-contained CG environment separate from the real world, AR appears to integrate CG objects into the already-existing world to achieve a hybrid mode of real and virtual perception.

Tisch’s showcase of “Ghost Game” invited visitors to experience augmented reality within the lobby of its Production Center using Google’s Project Tango, an AR-capable tablet that generates on-screen CGI in response to images on its live video feed.  Holding the Tango at eye level, players hovered their screens over and around display tables adorned with mock historical artifacts from the Triangle Shirtwaist Factory, including sewing spools, women’s blouses, and antique flat irons, as well as photographs and reproductions of old newspaper clippings. As players scanned across the tables, artifacts arranged in front of them generated on-screen CG animations of ghostly silhouettes, accompanied by pre-recorded voices emulating from the Tango’s speakers.

Factory artifacts

Over the course of “Ghost Game,” players uncovered fragments of a larger story about the Triangle Shirtwaist Factory through their encounters with virtual ghosts, who imparted biographical vignettes about ex-factory workers via spoken dialogue.  While the spirits’ messages served to sway the course of players’ investigations — piquing their interest as to which objects and spaces to investigate and explore next — players were ultimately free to explore the lobby and investigate the artifacts in any manner or order they chose, shaping their experience of the story based on the trajectory of their movement throughout the lobby and, thus, engaging in a “spatial” and “non-chronological organization of (the) narrative.” 3

As an interactive storytelling game, “Ghost Game” is actually neither highly interactive nor explicitly game-like.  Strictly speaking, players were not tasked with “surmounting increasingly complex obstacles” in order to achieve “mastery,” 4 and were furthermore unable to “achieve actual control over the content” of the story, apart from deciding whether to accept or decline reception of predetermined and finite narrative events (e.g., activating or ignoring the messages of spirits by tapping or not tapping the Tango’s touchscreen). 5 The game’s arguably low rate of interactivity is reinforced in that its AR permits players to merely observe a digitally mediated setting without the ability to alter the digital imagery superimposed over the images of that setting (as is possible in other AR applications using touchscreen devices).

Ghost Game 2

Real-life portrait of a factory worker circa 1911 (top left), alongside the “Spiritus Chronomatix” time-travel machine (right)

Ultimately, “Ghost Game”’s use of AR is striking in that it merges not only real and virtual environments, but present and past timeframes, enabling players to toggle between live video images of their present-day environment and CG animations representing elements of its past.  In the game, a chronological timeline labeled “2015 → 1911” appears at the bottom of the Tango’s touchscreen, prompting players to swipe across it to activate the spirits’ messages disclosing information about the history of New York University’s campus grounds.  Once players swipe across the timeline, a box-shaped contraption named the Spiritus Chronomatix (pictured above) receives an electrical signal as the Tango loads on-screen CGI, triggering the hum of the Chronomatix’s internal motor as players travel through time virtually.

While “Ghost Game”’s use of AR is currently limited to toggling between the years 2015 and 1911, and in designated sections of the Tisch building, one could imagine the possibility of a future iteration of the game in which players are permitted to toggle between a greater multiplicity of timeframes in a greater multiplicity of settings, with the Tango functioning as a ‘window’ through which to view and interact with the storied, digitally-rendered past of a myriad of real-world locations.  In this sense, “Ghost Game” points to the potential of using augmented reality to experience and create nonlinear historical narratives and, consequently, to help cultivate a nonlinear understanding of the historical past as an alternative to chronologically-based histories (for better and/ or worse).  Indeed, the employment of AR for the purpose of historical education seems more than ideal when considering the technology’s most basic function of heightening and informing our knowledge of our environment.

Word Count:  795


Works Cited

1 “175 Facts about NYU – Brown Building.” New York University. <>

2 “Brown Building (Manhattan).” Wikipedia. <>

3 Hassapopoulou, Marina. “Cine-Games & Interactive Music Videos.” Interactive Cinema and New Media. New York University. New York. 27 October 2015. Lecture.

4 Huhtamo, Erkki. “Seeking Deeper Contact — Interactive Art as Metacommentary.” 1992. p. 1

5 Bucy, Erik P. “Interactivity in Society: Locating an Elusive Concept.” 2004. p. 376