My Decoding the Digital presentation

I delivered my presentation at University of Rochester’s Decoding the Digital conference today, and it seemed pretty well received. So I present the text in full below, and you can see the accompanying Prezi here. At some point I’ll see if I can do something to add a synced recording to the Prezi, but for now, I’ll make it a quiz. Can you work out which frames apply top which bits of the talk?

Regular readers will find some parts of this familiar, which shouldn’t surprise, as this presentation is a synthesis of pretty much everything I’ve been thinking about so far. There was an interesting presentation today from Kathleen Fitzpatrick, during which she said that one result of “Digital” is that we academics should share more of our drafts, so I hope this blog does what she means.

There were lots of interesting presentations over the last three days, but I’ll have to blog about them at a later date.

The Interpretation Game: Research into digital narratives and cultural heritage interpretation

Matthew Tyler-Jones
Digital Economy USRG
University of Southampton

In his book, Designing Games, Tynan Sylvester says:

“If we look around, we find interactive narrative everywhere. Museums and art galleries are interactive nonlinear narratives where visitors explore a story or an art movement in a semi directed, personal way. Ancient Ruins and urban graffiti tell stories…

“These interactive forms – museums, galleries, real spaces, and life – should be [games designers’] first touchstones as we search for narrative tools. These older forms address our most fundamental challenge: creating a story that flexes and reshapes itself around the player’s choices, and deepens the meaning of everything the player does.”

Sylvester points game designers towards the heritage industry (and elsewhere) in an attempt to dissuade them from focusing on cinema as the sole source of narrative instruction. My own studies arise from wanting to look beyond linear storyforms (text, film) which exert a strong influence on museum and heritage interpretation designers looking to engage visitors’ emotions. Games do a great job of getting players to care about watching mathematical algorithms choose what colour each pixel on a screen is.

So, thank you (on behalf of my profession) for the kudos, Mr Sylvester, but I think the learning can be two-way.

To that end I’ve been looking, these past few months, at three digital narratives (or computer games) that each take a different approach to create so-called “open worlds,” three-dimensional virtual story spaces around which the player can wander with apparent freedom. All succeed in creating emotionally engaging (story-worlds or) diageses.

The first, Dear Esther, is described by its creator as an interactive ghost story. This was designed as an academic experiment in digital narrative, and has seen two incarnations, first as a “mod” or player-created content for Half Life 2, and subsequently as a standalone game, with a number of improvements.

The next, Red Dead Redemption is at the opposite end of the commercial scale, a big budget production from international production company Rock Star. It’s a western themed adventure that received a very positive critical and popular reaction.

Finally, I’ve been looking at is The Elder Scrolls V: Skyrim. An open world adventure with a traditional European fantasy theme. This offers a more dynamic narrative that either of the other two, and enables a greater degree of agency for the player.

The designers of these three games experimented with form as they explored story telling within each game’s virtual spaces. Cultural heritage institutions, including museums, built heritage, historic and ancient sites and heritage landscapes, have long been telling stories in three dimensions. Where it’s done well, visitors to those sites, and players of the best games, can make an emotional connection with the stories that they co-author as they make choices  about what to look at first and subsequently, and how deeply they want to explore individual points of interest.

So what drives emotional engagement in digital narratives, and what can cultural heritage institutions learn from them to improve interpretation?

I’m going to take a few minutes to run through the emotional drivers I’ve identified in the games I’ve played, and with each, explore equivalents in real-world cultural heritage.

Spectacle and sensation

When I’d just started playing Red Dead Redemption for this research, I spotted this tweet. (Prezi)

Hennigan’s Stead is one of the locations in Red Dead, so I followed the trail to this blog entry (Prezi), which demonstrates the power of visual spectacle to drive emotional engagement

The visual spectacle of all three games that I’ve been looking at is frequently lauded by players and in reviews.

Emotion through spectacle/beauty is something that my own organisation recognises. The full corporate name of the National Trust is, after all, “National Trust for Places of Historic Interest or Natural Beauty.” Respondents to surveys at the Trust’s most beautiful places, like Ightham Mote, for example, do report a higher emotional impact than elsewhere.

But does the National Trust as an organisation rely too much on the beauty inherent in our places? Good museums know how to manipulate spectacle, by creating “wow” moments, often at the threshold of galleries. These can involve impressive exhibits, multimedia “shows”, interpretive “set design” or even the design of the spaces itself, such as the Great Court at the British Museum.

Presence

But the virtual worlds that these games create are not simply beautiful. They are immersive, creating environments which themselves tell stories. The way the long grass sways as your avatar walks through it, the shadow of a bird that you notice a moment before you sight the bird itself, the changing weather, all add to your immersion in the diagesis. The architectural details, textures and ephemera of your surroundings all have the power, to inform the story.

Dan Pinchbeck, the academic and creator of Dear Esther calls this “presence,” and indeed, Dear Esther was created as an experiment in the manipulation of narrative and presence. Of course one might argue that Pinchbeck’s definition of presence thus includes all the emotional drivers that I’m discussing. But I’ll think about that and save it for a later paper.

Museums and archaeological sites often build reconstructed environments to encourage visitors’ sense of presence among the exhibits. And many historic houses are presented in such a way as to suggest the visitor has walked in on a room only recently vacated by the building’s historic inhabitants. Perhaps those that manage presence most successfully are the living history museums such as Plimouth Plantation

Society

There’s still an unfair perception of computer gamers as solitary types with no friends, but of course most games, not just team sports, are ways of bringing two or more people together. Whether it’s gathering around the screen in the amusement arcade, bedroom-coding or simply comparing high scores, computer games have always been social. But of course fast, cheap data communications have resulted in all sorts of social gaming from Words with Friends to World of Warcraft. So, how do my three games do?

In fact, only Red Dead Redemption has a multi-player mode, and that only after you download an expansion. Dear Esther’s isolation is part of the appeal of that game, and Skyrim offers stilted, repetitive, scripted conversations in place of interactions with real human beings.

So this is one area where, it seems, cultural heritage is way ahead of computer games. People use museums and heritage sites as social spaces, to spend quality time with friends and family, without any intervention from the cultural intuition itself. Guides and docents have long been a part of the heritage infrastructure, and part of the enjoyment of visiting a place like Plimouth Plantation is navigating the sometimes archaic language and social mores of the costumed interpreters.

Of course, people are an expensive asset for cultural organisations, and even volunteers are not free, so there are experiments inspired by gaming technology. A team working to interpret Monserrate Palace in Sintra, Portugal, have been experimenting with what they called Embodied Conversational Agents. The idea is that the virtual character would capture the visitor’s interest with a non-interactive animated opening scene, in the manner of a cut-scene on a video game, but then would open up a real time conversation that would immerse the visitor with realistic “face movements, full-body animations and complex human emotions.”  The conversation would be more sophisticated than a simple question and answer system, by being “context aware,” breaking up the knowledge base into modules, to make interactive responses more possible.

Acquisition

Computer games often simulate the acquisition of wealth, equipment, or simply points. High totals are rewarded with new abilities, or unlocked levels and new play experiences. However the emotional value of simply beating your best score, or getting to the top of the high scores table should not be ignored.

In Skyrim, you can pick up, buy or steal almost every object you see, and a new player’s character will quickly get weighed down with useless cheap tableware and other ephemera, before making more rational choices and building a useful supply of weapons, potions, spell books and apparel. Eventually your character will be able to acquire one or more houses in which to keep the stuff you accumulate.

In Red Dead Redemption, apart from the usual money and weapons, players can devote time to completing various quests which are rewarded with new outfits for their avatar to wear.

Of the games I’ve researched, only Dear Esther eschews acquisition as an emotional driver. But of course Dear Esther is all about loss.

Cultural Heritage sites often relegate acquisition to children’s trails. Many of which are “I spy” checklists, sometimes rewarded with a sticker to wear. The National Trusts “50 things to do before you are 11 ¾” campaign encourages families to acquire 50 fifty life experiences (like making a mud-slide or flying a kite).

I was also intrigued to recently discover this question (which measures an acquisitive impulse in visitors of all ages) among those evaluating a mobile guide app.

Challenge

As we’re talking about games, the most obvious emotion driver should be the ludic one (which is to say, the one all about play). Games test the player’s dexterity, pattern learning, and puzzle solving ability before rewarding him or her, not just with a sense of accomplishment, but progression within the game. Both Red Dead Redemption and Skyrim use the traditional (to video games) challenge of having to kill people to get where you need to be. Deconstruct computer game conflict however, and what the players do is very similar to what they do playing Tetris or Candy Crush Saga – the core of the challenge is to point at as many icons as you can before the time runs out.

The National Museums of Scotland have experimented with introducing a time-based, ludic element to the cultural heritage experience. In Capture the Museum “visitors download an app to their smartphone and sign up to either the Red or Blue clan. The two sides plan their strategies then spread out across the National Museum of Scotland. A map that updates in real-time shows which clan owns which ‘territories’ – the differently themed galleries in the Museum. Players scan into territories using their phone’s camera, where they prove their understanding of the exhibits to earn the high score. After 30 fast and furious minutes the clan with the most territories is crowned the winner.”

Visitors coming to the museum’s programme of late-openings appear to enjoy the game, but time limits and territory capture may not be compatible with the every-day visitor’s wants and needs. However Sylvester argues that challenge is not as essential to games as it might appear, citing Dear Esther as a game that can “create powerful emotions without players struggling.”

Learning

All three games put learning at the core of their gameplay. And I don’t mean simply learning the patterns of the Space Invaders’ irresistible advance. Books describing the history and culture of Skyrim are liberally scattered around that game’s world. The player of Red Dead Redemption learns uncomfortable truths about their character, and the whole point of Dear Esther is to piece together seemingly random memories to reveal a tragic story. “If a lesson is obvious,” says Tynan Sylvester “there’s not much buzz in finally getting it because it was always fairly clear.” Instead, he advocates a moment of insight, where everything that has come before “clicks into place and reveals the shape of the whole.”

I’m not sure how successful cultural heritage sites are at shaping that moment of insight.

A couple of decades ago (which is in itself a scary fact) I was at a conference like this, sitting where you are now, listening to Judy Rand describe a model of interpretation which starts off with the main message, or theme, which she describes as the “single most important idea you want people to leave with.” With the theme in place, and informing all the subsequent decisions, all the other story elements or messages fall into one of three categories, primary, secondary, or tertiary, in decending order of importance.

Rand explains that this process is more than an arrangement of the relative value of the messages, it suggests a floorplan, with primary messages becoming sub-divisions of the exhibition and secondary messages indicating the contents of individual exhibits. Tertiary messages are only found by the most interested visitors.

It’s a sensible model, and one you can see in exhibition galleries right around the world, most obviously those that begin with an introductory video to put everything else in context. But it front-loads the moment of revelation, of insight.

The best attempt to subvert expectations I’ve seen recently was the Life and Death in Pompeii and Herculaneum exhibition, recently hosted by the British Museum. The introductory video was still there of course, but the final exhibit was an opportunity to remember that the reason we have discovered so much about the Roman way of life was a terrible intimate catastrophe.

Character Arc

Playing Dear Esther, you are intrigued to discover snippets of insight into a number of characters shared by a nameless narrator that may, or may not, be your avatar in the virtual world. Though there is an element of randomness in what you hear, the little world you inhabit is so tightly structured, that there is a very definite character arc.

Similarly the apparent open world of Red Dead Redemption, seems to offer a wide range of player choices, at first. But the choices are illusionary, you can’t change the story. As play progresses, you are forced towards your avatar, John Marston’s inevitable fate.

So Red Dead Redemption takes a more cinematic approach to narrative than Skyrim. In that game, you have more control over who your avatar is to begin with, and the avatar you design will affect how the game-world reacts to you to some degree. There’s far less control on where you go than in either Red Dead or Dear Esther, and a far wider choice of stories to explore.

In fact, each story turns out to be a lot more linear than Red Dead Redemption. Yes you can hop between stories, and tackle challenges in a wider variety of ways, so your avatar’s personal narrative may well be unique. But… the ending of Red Dead Redemption is powerful, and in Skyrim, I’ve saved the world twice already, and yet I’m still waiting for, longing for, the end credits.

You’ll see that, of the two, Skyrim’s narrative structure is closest to Rand’s model of interpretation. It just fizzles out when we get to bored to enjoy it any more. So my challenge to cultural heritage, is can we structure our stories to end with something closer to Red Dead Redemption and Dear Esther’s emotional punch?

The Holburne Museum in Bath recently experimented with a ludic interactive narrative form of interpretation. The project, called Ghosts in the Garden, involved giving groups a steam-punk listening device, which would allow twenty-first century visitors the opportunity to listen in on early nineteenth century conversations, taking place around the museum’s gardens, which were once a popular “pleasure grounds” attraction. At every location, users were given a choice about where to guide their nineteenth century interlocutor next, in the knowledge that they choice they made would change the outcome of the story, just like a “choose your own adventure book.”

I was interested in the emotional impact this innovative story-telling method might have, and offered to help evaluate the project with a user questionnaire. I received the data only recently, and I’ve not yet had time to analyse it properly. But one result I’ve already noticed is an apparent scepticism from users that they were actually changing the story.

So I’m just beginning to wonder if making a story interactive, is of less value than creating a strong character arc?

Music

And finally, we come to music.

Tynan Sylvester has this to say about music in games “Nobody ever gives it the credit it deserves because nobody consciously pays attention to it during play. But even though the conscious mind is oblivious, the unconscious is still processing the music into a continuous flow of feeling. You can tell because music is easily separable from the rest of the experience. Listen to a game soundtrack by itself, and you’ll feel much of what you felt during play. Play the game in silence, and you’ll be surprised at how hollow it feels.”

I know that’s true, the song that plays over the end credits of Red Dead Redemption has the power to bring a tear to my eye.

I’m going to lay my cards on the table here and say I think cultural heritage uses music incredibly badly (at least when music isn’t the subject at the centre of the heritage experience). Museums and Heritage sites too often resorting to clichéd soundtrack choices or only-just-appropriate royalty free generic music.

There’s an opportunity to use music far more intelligently in heritage spaces, Cohen (Prezi) explains how computer games use music to indicate breaks in the narrative, direct attention to particular spaces, communicate meaning, and trigger moods. The Dear Esther soundtrack is exemplary in this regard. The power of the music to invoke memories or “prepare the mind for a type of cognitive activity” is well recognized in advertising and sonic brands such as those created for Intel and Nokia. Why is the leitmotiv, such a useful tool in operatic storytelling for hundreds of years, so seldom used by museum spaces?

Conclusion – where next?

The sample size for my research around Ghosts in the Garden is too small, and of course, I’m sure I’ll discover I’ve asked the wrong questions. But I’ve got the opportunity to improve on the method with a similar project taking place at Bodiam Castle in Kent. I want to use that to test the validity of the emotional drivers I’ve defined today.

I also want to explore in more detail in the impact of, for want of a better word “designed music” on emotional engagement with cultural heritage sites. There’s something about how the games I’m examined use music (and silence) dynamically, that I’d like to find collaborators to help me test.

And I’m curious that participants of Ghosts in the Garden didn’t believe that their interaction really changed the story. It reminds me of what Aylett refers to as the Narrative Paradox  – allowing your audience to interact with, and change the story, reduces its cohesion. Colleagues at Southampton wrote a paper last year on the Narrative Braid, which attempts to tackle the narrative paradox in documentaries. I’m convinced that combining their ontologies, and concepts of narrative atoms, molecules and threads, with location and object based orientation, could make for exciting possibilities in heritage interpretation.

I wonder if interpretive narratives can become, if not interactive, then dynamic, in a similar way to computer game music, sticking to a well written engaging narrative arc, but layering different elements, tones and details into the story, depending on the agency of the visitor and where they choose to go with the heritage space.

Thank you.