Time Detectives: The Mystery of the Mary Rose with Charlotte Mikkelborg

Fred: In this interview, we chat with Charlotte Mikkelborg, director of the new multisensory augmented reality game Time Detectives: The Mystery of the Mary Rose, available now on Apple and Android. Well, Charlotte, we just checked out the trailer for this awesome-looking game, available on Android and Apple iPhone. Wow. How did this whole concept come about, and how did you get linked up with the Mary Rose Museum?

Charlotte Mikkelborg: Well, how did the concept come about? I’m in the business of making immersive experiences, and in a way, each thing I make is different from the last, because we’ve made immersive installation experiences, we’ve made interactive VR, and I’ve made conventional 2D film and also 360 film that you experience in headsets. So a lot of different things. The last piece we made pre-COVID was a very interactive VR piece called Fly. And then COVID hit, and there weren’t going to be as big budgets available to make massive experiences, but we didn’t want audiences to suffer. How could we make an experience that’s going to be as immersive, but maybe doesn’t require quite the amount of investment that VR does?

AR was kind of a natural solution to that, because obviously you’re not building an entire new world. You’re building elements of that world: characters, objects. So we started thinking about AR, and then I was like, “Well, do we have to limit ourselves to just the audiovisual just because we’re doing AR, or could we still incorporate other senses?” Because a lot of my work incorporates other senses, whether that be haptics or scent. So we were thinking, what could we do with that? We came up with this idea of creating the world’s first multisensory augmented reality game, which would incorporate the audiovisuals through the phone, but also scent. Originally, we were doing these little scent badges, and in the end it turned out to be a backpack that releases scent at certain interactive moments in the story, in the game.

So yeah, that’s kind of where it was born, I guess. It was a COVID baby, for sure.

Fred: And why did Henry VIII’s story speak to you in particular?

Charlotte Mikkelborg: When we started developing it, we were asked to come up with new immersive ways to highlight, or bring new audiences to, some of the really incredible cultural heritage sites that we have in the U.K. As part of that program, we were looking at a few different sites, like the Houses of Parliament, and we looked in more detail at Oatlands Palace in Surrey, which is one of Henry VIII’s palaces. When we got the funding for the first full game, we were looking at partners. We were speaking to different people, like the Tower of London and Shakespeare’s Birthplace, because obviously there are just so many amazing sites to choose from here. But because we were funded by something called Innovate UK, which is a government-run body, or at least part-funded by them, we had quite a limited time window.

We had to make the game within six months. Cultural heritage institutions are notoriously not fast-moving, so it was also about who could work that quickly. And so the Mary Rose Museum… and the Mary Rose is one of… oh, I was about to say one of Henry VIII’s wives. No, no, no, complete mistake. I’m having Friday night brain freeze. The Mary Rose sounds like one of Henry VIII’s wives, but it was his favorite warship. She successfully sailed for several decades before sinking in 1545, just as they were about to enter battle with the French. So not great timing. Almost all of the 500 sailors and soldiers on board died in that sinking, but it’s remained a mystery.

They don’t know why it happened. They have different evidence for different theories, and there are lots of clues, but they don’t know definitively. So it was a chance to give you agency in the game. You become the investigator investigating the sinking, and see what you make of the evidence. I love that period of history. I hate to say there’s an element of me that loves it for its brutality. You cannot get your head around the punishments that they would come up with. Some guy who sabotaged the guns on board had both his ears cut off. We have a great series in the U.K. called Horrible Histories. It’s mostly for kids, but I love it. It shows you the most gruesome, disgusting things about the Tudors, the Vikings, the Romans.

So I think the Tudor period fascinates me because it’s not that long ago, but it is pretty different to today.

Fred: You’re a creator at heart, a filmmaker at heart, a director at heart. You have this great technology you can work with, but at times it can also be your worst enemy and handicap you in certain ways. What kind of challenges did you face trying to tell this great story while also working within the confines of this technology, and of course the deadline?

Charlotte Mikkelborg: It is interesting, isn’t it? Because with film, obviously, you’ve got your established pipelines, and you know what tools are at your disposal and what lenses you can use and everything else. With new technology, there are always unexpected problems that emerge. One of the key innovations in this game, other than the scent I mentioned, is that we wanted to feature photorealistic 3D characters. If you’ve ever played Pokémon Go, you’ve seen these simple anime characters that pop up. Ours is also augmented reality, so in that sense it’s like Pokémon Go, but we wanted you to feel like actual humans from the past had entered your presence. There’s a process called volumetric capture, I’m sure you know about it, and basically that involves a very complex system of cameras to capture somebody really authentically in full 3D.

But that’s never going to work within an app-based game, because it’s so data-heavy. So we worked with a simpler form of volumetric capture to capture our actors, so that it was more data-light. We had a single camera plus Depthkit, so very different to the full volcap setup, but actually the end experience is not that different. From the front of your characters, you get a very realistic 3D impression. If you try to go around behind them, they start to distort a little, and you don’t have full 3D. They’re essentially more hollow behind, unlike a full volumetric capture character that you could potentially put into a VR game. But you don’t lose a huge amount through that, and what you gain through feeling like they have just appeared in front of you is substantial.

So there were challenges like that, and weird things as well. We had a funny bug where the audio and the video for our volcap characters would fall out of sync, which was throwing everyone off. And there were other technical challenges, like you say, given the timeline, which was stressful. I think if we’d had a bit more time… You could always work these things out, can’t you, given time and money? But we had a limited supply of both. I think what came out was good in the end, though, for sure.

Fred: You’ve definitely become synonymous with this space of augmented reality, VR and experimental technology when it comes to your work. How long did it take you to embrace all that? You see a lot of directors and creators who are a little apprehensive about it. Maybe they’re like, “I don’t want to touch it. It’s killing the art of filmmaking.” You look at it a different way and say, “No, this can only enhance what we’re doing.”

Charlotte Mikkelborg: Well, I think it’s also sort of where I was coming from. I came out of journalism, actually, in 2010, I’m going back in time, and I was in filmmaking for a few years. I started in documentary, and I progressed on to also doing fictional work, but I hadn’t been in it for decades. I was clearly not Scorsese trying to then make my move into XR storytelling. So in a way, it was quite nice, because I discovered the medium at the right time, where I’d honed my skills to a degree as a filmmaker, but I wasn’t so established that I didn’t want to give up my hold within that industry. It’s obviously a very competitive industry.

And then I was actually meeting with the producer of Lucy Walker, who’s an Oscar-nominated filmmaker herself, and I asked the producer, “What’s Lucy working on?” And he said, “Oh, she’s working on this 360 film.” And I was like, “Oh, explain that to me,” because I didn’t know what that meant at the time. He showed me this, and while the technology clearly still had a long way to go, I was like, “Yes, this makes so much sense.” Because why would we experience our visual medium like this, when we can experience it like we experience the world? So it just made a lot of sense to me. Of course it’s going to be more entertaining if we can fully immerse you in something, whether that be a thriller or whatever it might be.

And then if we can involve your other senses in that, in a way it’s almost too powerful. I don’t think I want to experience a horror film like that, if I’m honest. But there are certainly a lot of things that I would like to experience like that. So I came at it from that point of view, and I was able to carve out a niche for myself in this industry while it was still evolving, and that’s been really exciting.

Fred: When it comes to Time Detectives, were you a gamer prior to working on this, or did you slowly immerse yourself in the space as you were working on this project?

Charlotte Mikkelborg: I would say more the latter. I have definitely played games, but I wouldn’t consider myself a gamer, because when I think about all the incredibly dedicated gamers out there who’ve played everything, if I have a conversation with one, they will lose me within a few minutes. We might have played a couple of the same games, and then they’re off, and I’m like, “Oh no, I haven’t played that.” So no, I wouldn’t have described myself as a gamer. But what I am is very excited about any new method… For me, the medium is not as important as how much we can engage audiences in story. For me, life and work is one big, very fun experiment in, can this next project make people feel even more deeply involved in the story I want to tell them?

That, for me, is the excitement of this whole area, including the gaming area, but still with that augmented reality. I think pure gaming, without augmented reality or virtual reality, is perhaps of less interest to me, because I don’t feel it’s pushing the envelope as much. But yeah, that’s where I am.

Fred: Long term, would you like this story to have some sort of port or companion piece on, say, PlayStation VR or some of these other bigger platforms when it comes to virtual reality and augmented reality? The phone is great. It’s super convenient, and there are a lot of powerful phones that can run this. But I would assume the next step would be to get onto more platforms, so you’ll be able to educate people or pass along whatever message.

Charlotte Mikkelborg: Honestly, I haven’t thought about that a huge amount yet, because we’ve really just completed this first game in what I think will be a Time Detectives series, and obviously this is the first of those, The Mystery of the Mary Rose. You’re starting to get the juices flowing in my brain now, because really taking you and fully immersing you in that with VR, or PlayStation VR, could be a great deal of fun, and I could see a lot that you could do with that. But I think we’ve been thinking more along the lines of how we could use this incredibly immersive game style to open up these incredible places that we have to new audiences, because younger audiences often don’t engage with them as much as older audiences.

And the fear with that is, if they don’t, will these places get lost in the future? So I’m quite passionate about engaging young people, really showing them how exciting these places are, and using technology to do that. How exciting not just these places are, but these stories. Because you look at some old ruins and, I mean, that’s not interesting. But when you see a murder that was committed there… okay, maybe that’s a bit too gruesome. But when you see some of the stories that have unfolded within those walls, that becomes different.

Fred: You’ve of course had success in traditional filmmaking, and you’ve had success in advertising. You’re more than likely going to have a lot of success with Time Detectives as well. What’s the best piece of advice you’d give for success?

Charlotte Mikkelborg: Honestly, there’s a lot of hard work involved. I know someone wants me to say, “I’ll tell you what, there’s a magic cure.” I wish there was. It’s a lot of hard work and a bit of luck, I think, always, along the way. And probably putting yourself out there to a degree. I know other successful people who certainly networked their way to success. I can’t say I’m particularly one of them. I have gotten better at doing that, but it’s really about being in the mind of somebody who might be making a decision on commissioning something, and obviously if you’re not out there speaking to people, then you’re not necessarily there in their mind. So I think hard work, a bit of unavoidable networking, and probably a bit of luck.

But you make your own luck, because you try to put yourself in the right place at the right time, or have the right conversation. And I also think helping people. I know that sounds really cliché, but people take different attitudes to things, right? Some people take this attitude of, “This is my idea, and I can’t talk to anyone about it, because someone will steal it.” Or there are people who are just very competitive, so you ask them for a hand with something and they’re like, “Oh no,” and they make you feel very awkward even asking. Whereas I actually think it’s better to have a more open view. Ultimately, ideas are ideas, and if you’re an ideas person, they come and go.

You’re going to have more of them. You don’t need to be too worried about one getting stolen. If that does happen, you’re unlucky. But I think if you can do a bit of giving, you get back as well. So I think it’s a mixture of all of those factors, probably.

Fred: Why should people play Time Detectives?

Charlotte Mikkelborg: Oh, why? Well, you should play Time Detectives if you fancy the idea of being transported 500 years back in time to experience the sounds and smells of a Tudor warship, and to eavesdrop on the conversations. And apart from anything else, if you’re a girl or a woman, there were no women allowed on board, so get on there, because that’s your only chance. But no, I think it’s fun. It’s different to a lot of games. It’s not a game that you’ll go back to and play over and over again, because it is a story-based game. You play it once. Actually, you could probably play it a second time and it would still be interesting, because you can follow the perspective of two main characters.

But yeah, it’s fun. It’s something you will not have seen before, that much I can almost guarantee, because photorealistic augmented reality characters in games are new. So I hope you enjoy it.

Fred: Awesome. Well, we’ve definitely enjoyed it so far, and we hope everybody else does as well. Looking forward to seeing more of your work. Charlotte, thank you so much for your time. Before we let you go, where can we find you online?

Charlotte Mikkelborg: Personally, on Twitter, I’m @CMikkelborg. The spelling of my name is a tricky one, but hopefully you’ll find me. And I’m the same on Instagram. My company, slightly oddly, is Picture This Productions, but actually we’re Pic This Ltd, which is how you name a company here in the U.K. In terms of new things we’ve got coming out, Insta is probably a good place to follow us. And I think we’re also Picture This Productions on Facebook. I say “I think” because there might be a couple, but I think we actually did manage to get our proper title, Picture This Productions, on Facebook, if I remember right. We’ve got a cool circular dragon logo, so hopefully you’ll find us.