Earlier this summer, we released  Pharos AR, a mobile app that takes Childish Gambino fans on a virtual, hallucinogenic journey through the cosmos starting from wherever they stand in the real world. Fans and press alike were impressed by the artist’s bold foray into virtual space.

VentureBeat was excited by the long-term value of the app, for example: “Between the cool visual effects and Glover’s music—which the app will apparently update with new songs over time—there’s certainly enough here to merit a download for fans.” Variety, meanwhile, applauded how well it integrated within the larger Childish Gambino universe: “The whole thing is very spacey, and stays true to Childish Gamibo’s other ‘Pharos’ projects” that have expanded across performances.”

As we see interest build in augmented and mixed reality over the interim, we’re looking back on the project’s development and how multiple parties and team members came together to employ best practices in UI and design, helping everyday users ease into a mysterious–and perhaps technologically overwhelming–new interface without limiting space for play and exploration.

Made in collaboration with Google Zoo, Unity and MediaMonks, it’s the world’s first shared augmented musical experience, allowing multiple users to enjoy a unique, artistic experience together. “This app is a breakthrough for AR,” says Thomas Prevot, Sr. Producer at MediaMonks. “It serves as another outlet for Childish Gambino’s creativity, letting him update his fans with future song releases over the cloud,” says Prevot, explaining how the app fits within the larger Childish Gambino brand.

Pharos AR also showcases the power of the ARCore platform and how it can enable immersive, social storytelling experiences. In particular, it shows off the capabilities of Cloud Anchors, which lets multiple users interact in a shared virtual space–which also makes it fairly unique among AR experiences. The tech’s newness can be intimidating, though: how can apps cultivate an interactive, exploratory experience for those new to AR?

Integrate Brand Familiarity & Digital Ecosystems

Childish Gambino fans will recognize Pharos AR’s light-particle silhouette from a Pixel 3 TV ad, and scenes projected within an enigmatic pair of monoliths at this year’s Coachella festival. These elements are purposefully and artfully executed across touchpoints, and this integration with a pre-existing digital ecosystem helps to make the process of designing for emerging tech more intuitive.

From the Pharos Festival to a TV spot to Coachella and beyond, you’ll find the same trademark, psychedelic motifs true to the nature of the artist. This goes to show the potential of AR as a powerful channel to not only engage audiences in an immersive way, but to bridge together the Childish Gambino gospel as well.

Get Intuitive

With AR still being fairly new to some, developers must think carefully about onboarding new users. “Experiences like Pharos AR provide an exciting opportunity for us to help make AR and VR more accessible to wider audiences,” explains Justina Sung, UX Designer at MediaMonks. “If people habitually use their phones a certain way, how do we break out of that to teach them new behaviors for emerging tech like AR?”

Inspired by Childish Gambino’s spirit of ambiguity and minimalism, the app integrates a sensory communication method, instead of verbal directions, to onboard. “Most app tutorials task the user with swiping through instructional cards, but we made it feel like the title sequence of a movie,” says Alex Otto, Associate Creative Director at MediaMonks. When users first open the app, for example, a pink laser beam dynamically snakes through space and into icons that illustrate next steps. “It’s designed to give off a mysterious feeling that builds up layers of suspense,” says Otto.

Direct Users Through Environmental Storytelling

Because AR rewards users through exploration, designing for it requires a careful balance between providing freedom versus direction. “It’s important that the design gives fans room to explore on their own to find meaning, using subtle environmental cues that nudge users in the right direction,” says Sung.

In Pharos AR, the scene gradually dims as users discover hidden glyphs on the cave walls. This nudges them to shift their focus to the altar, which lights up as users discover more paintings—a bit like a progress bar. Other cues include haptic feedback and the gradual, sonic buildup to Childish Gambino’s song “Algorythm,” which released through the app.

AR is a powerful platform for brands to tell their stories and engage directly with key audiences. In the case of Pharos AR, the technology provides users a chance to embark on a wild, immersive journey that encapsulates Childish Gambino’s message of enlightenment. As the first interactive, shared musical experience in AR, the experience truly pushes the limits on what can be achieved—inspiring brands and artists alike to consider what’s possible.

Eager to experiment with easily approachable AR?

Raising awareness or building compelling content around destinations that are newly offered, remote or struggle with mixed perception can be a challenge for travel brands. But through emerging tech and improvements in mobile platforms, brands can provide experiences that let consumers explore far-flung locales without having to leave their homes, letting them intimately learn about a place through immersive storytelling. 

According to a report from Expedia Group, one third of Latin American Gen-Z and millennial travelers perform the entire travel shopping journey via mobile devices. The finding showcases the importance in offering mobile experiences that streamline the decision-making process through value adds, attention-seeking imagery and tie-in to social platforms.

Know Your Audience & Grab Their Attention

In essence, brands can better reach these young travelers through mobile moments: the points at which users turn to their devices to seek inspiration or answer a question. Brands can achieve this by adopting a mobile-first mindset when designing user experiences. Start by thinking about the context through which consumers engage with their devices.

Next Destination: Our 2019 Travel Trends Report

When Edelman sought to promote Hawaii as a destination, they targeted busy New Yorkers on their daily commutes—a key moment in which users are staring at their phones.  With an attractively redesigned MetroCard directing users to an elegant mobile site, the campaign intervenes with commuters’ daily grind to inspire a sense of wanderlust. Produced by MediaMonks, the mobile site lets users discover each of the archipelago’s islands in stunning visual detail before entering a drawing to win a free trip. In addition to its visual design, another strength of the campaign is how well it “gets” its local New York audience and situates itself within their daily routine.

Innovate through Scalable Experiences

With competition shifting its focus to mobile as well, it can be a challenge for travel brands to stand out. But investing in emerging technologies or supporting new social platform features provide opportunities for brands to reach consumers in fascinating and noteworthy ways. Take airline brand KLM for example, which is no stranger to using mobile AR to transport users to another place.

MediaMonks worked with the brand to innovate even further by offering AR advertising directly within Facebook’s newsfeed—a feature that isn’t yet available to the public. The ads invite users to open a virtual door as they scroll past, which activates a 360-degree photo allowing users to view landmarks from different angles by moving their device. The immersive content focuses on new or lesser-known destinations serviced by the airline brand, making use of the technology to drive awareness and allow users to almost literally stumble into a brand-new environment.

A similar example is STC’s “Unveil Saudi” series of VR experiences. STC isn’t a travel brand—the content serves to showcase the strength of its network—but it does an excellent job of letting users inhabit remote landmarks in stunning 360-degree video. Users can enjoy the content by strapping their device to a VR headset, or simply drag their cursor to control the view on desktop. This versatility showcases an important consideration for immersive storytelling: it’s easily scalable and accessible to audiences.

Reap the Full Potential of Mixed Reality

The strategies above are great for raising awareness about destinations or pique consumers’ curiosity about a place. But what better way is there to showcase a travel destination than by giving consumers the opportunity to really feel as if they were there?

A best-in-class example of bringing the destination directly to consumers is the 4D trishaw ride in VR that we developed with TBWA for the Singapore Tourism Board. Users strap into a real trishaw equipped with a VR headset, which transports them to Singapore’s hustle-and-bustle. The trishaw makes for an excellent vehicle—excuse the pun—for such an experience, as participants inhabit the role of a passenger.

By focusing their gaze in different directions, users can choose the path they want to take. Along the way, spatial sounds, vibrations and scents provide a multi-sensory, immersive experience that makes users feel like they’re really there. This type of experience won’t be practical for every brand; as an installation, it makes a better fit at trade shows, for example. But travel brands that want to make an event out of highlighting a destination (like a tourism board) can benefit greatly from the technology.

Improvements in emerging tech and consumers’ favorite social networks provide new opportunities for brands to connect with audiences and inform them about their services or destinations. By letting users inhabit or explore an immersive location—whether it be just a quick moment on their phone or a lengthy drive through a city in VR—brands can raise awareness in compelling ways and increase their digital maturity.

While an exciting game and sports-stars-turned-celebrities alone were once enough to raise an audience for broadcast sports, today’s leading brands strive to provide premier, engaging digital experiences that reach users through personalization and emerging tech. And we know, because MediaMonks recently went for the gold and made it onto the Hashtag Sports Engage 150 list, which features the top partners engaging consumers through sports and fan culture today.

Through our sports-related work—encompassing platforms, creative content and technical innovation—we’ve formulated a game plan for brands to better engage and deliver upon the needs of fans through sports. This includes more effective utilization of user data and investing in emerging tech solutions, allowing brands to bring their A-game to enhance spectatorship for fans both near and far away from the game.

Transform Spectators into Active Participants with Emerging Tech

We all know the joke of those who buy the biggest TV screen available before an important game, only to return it after hosting a viewing party. But today’s digital media allows brands to do more than just provide a larger-than-life picture; through emerging tech, they can provide entirely new experiences that significantly improve spectatorship through heightened immersion.

A premier platform achieving this is the 2018 Webby Award-winning Red Bull Air Race for Google Daydream, which takes users on a thrilling ride aboard a virtual airplane that follows actual flight paths from real races. Giving users full control to look wherever they please, the experience makes them feel as if they’re really there.

But VR lets you do more than let users feel present in the moment. “The benefit something like AR has over 360-degree video is that it’s a platform,” says Robert-Jan Blonk, Senior Interactive Producer at MediaMonks, noting the amount of on-screen data present to the user. “In this case, we built a platform with pilot info, race standings and different locations where races are held, which users can go back and replay.”

This added level of autonomy is powerful for fans who want to be part of the action. “Sports fans want to feel like they have a role in the game and aren’t just bystanders,” says Emily Veraart, Senior Digital Strategist at MediaMonks. While Red Bull Air Race users don’t influence the race directly, interactive toggling of the camera lets them experience the event in their preferred way—whether it be the middle of the fray or at a safe, omniscient distance.

For MediaMonks Operations Manager Donny Hofman, this level of interactivity is integral to the experience.  “The most interesting thing about an experience like this is the freedom you can give to users,” he says. “You can potentially get anywhere in the action that you want—and where you can’t place a camera in real life, you can in VR.” That latter point describes how emerging tech can enhance sports spectatorship through entirely new perspectives: “It’s a rediscovery of something you’re familiar with,” says Hofman.

Reach Fans with Targeted Content and Personalization

The sports industry encompasses several different parties, each of whom has their own fans and content: teams, federations, individual players and industry-adjacent influencers. This makes the industry ripe for producing always-on content.

During the Rio Olympics, we produced with Google Zoo a platform to deliver content to sports fans throughout the Games. Designed for both online fans and those visiting Rio de Janeiro to attend in-person, the platform integrated various forms of content (like timely updates and summaries from some of the region’s most influential YouTube content creators, the Castro brothers) onto a map of the city. In addition to providing relevant spatial information in this way, the platform surfaces up personalized content responsive to how they interact with the platform, like recommending content based off behavioral habits and viewing preferences.

The benefits to better understand fans are measurable. According to MightyHive’s Data Confident Marketer report, data-confident marketers’ success “is attributed to becoming more customer-centric: they’re able to apply first-party data in ways that help them understand who their customers are, what motivates them, and how digital advertising plays a role in their purchasing decisions.” Such confidence in their data allows brands to identify which information fans seek, where and when—and which partners make the best sense for engaging with fans through supplementary content.

This level of personalization, paired with forging partnerships with content creators and influencers, is ideal for delivering upon changing user expectations for how to consume sports. “With sports, there’s a bigger generational divide in user behavior,” says Veraart. “Baby boomers and millennials are used to watching sports with their family, but Gen Z doesn’t have that relationship with how they watch sports.” According to Veraart, much of this shift is informed by the constant discussion happening on social media, as well as an abundance of statistics and data available for the most avid fans to track. Brands can meet this need with a content strategy that offers a sliding scale of data and content tailored to fans’ individual preferences.

Engage User Needs through Contextual Design

When developing a digital platform for sports consumption, carefully consider the context in which users will interact. Consider micro-moments that prompt users to engage in the first place: the sports tourist attending a big gaming event in town, the stats-obsessed fan, the user who simply wants to see what games are playing at the height of the season. “How you guide the user through a schedule is a key example of the types of challenges a sporting federation may face,” says Joeri Lambert, Business Monk at MediaMonks. “Another is how you apply the data that you have to alert the user of games or information that they want to see.”

One crucial consideration in how to best support user context is whether the platform is accessed via mobile or desktop. With the All of Brazil Plays platform, for example, the mobile experience was tailored more toward location-based information for those attending the Games who needed to know where to go, and when. On desktop, meanwhile, users were treated to their personalized newsfeed of recap content, perfect for those catching up on a day’s many events.

Marrying data with digital creative—whether it be a digital content platform or emerging tech—is a smart strategy for brands to reach sports-obsessed and average fans alike. From helping users keep up with their favorite team to placing them right into the action with immersive tech, digital content transforms spectatorship from a passive experience to an active one, letting everyone revel in a good sporting victory.

With omnipresent technology on the horizon, are we headed toward a cyberpunk dystopia?

That was the big question MediaMonks posed to the audience at its Fast Co. Fast Track session, offering attendees food for thought to mull over with drinks. Rejecting a future in which pop-ups and banner ads vie for our attention literally everywhere we look, speakers Eric Wagliardo and Jouke Vuurmans offered a more optimistic view. By using emerging technology like VR, AR and artificial intelligence to delight audiences, Wagliardo told the audience, “we can use creativity unconstrained by the past and the laws of physics.”

The theme of the talk anticipates a near-future in which wearables or ubiquitous AR will revolutionize the way we interact with media and technology, just like how the iPhone ushered in the era of smartphones. As these technologies become more consumer-friendly, brands might be biting their nails wondering if they’re ready for the new digital landscape. Are you?

The focus of your VR or AR experience doesn’t have to be grand to be impressive.

One of the more compelling details from the talk was about a humble plank of wood used in the Jack Ryan Experience, a 60,000 square-foot activation that pulled together some of the most cutting-edge VR technology to provide a 4D, immersive experience. Through a VR headset, the plank transformed into a startling chasm for participants to cross in virtual reality. That such a basic material in reality could become an intimidating environment to traverse demonstrates the power to create compelling experiences from just a simple object. With so many elements and gorgeous tech to compete with, who would expect a plank of wood to steal the show?

VR can be tricky, though: without a proper “hero device” like the iPod or iPhone to put the tech in almost everyone’s pockets, VR experiences must often be tethered to a specific environment for participants to don a headset and enter your world. This can be a big investment in terms of reserving a space as well as providing a rig to host the experience, though it’s well worth it if you’re hosted within a venue that allows you to reach your audience in a meaningful way. For some brands, a simpler solution might be to create an AR experience that participants may enjoy via nearly any mobile device.

Another benefit to AR is that it can enable a charming connection with objects or pre-existing campaigns that wasn’t possible before. One of the demos featured at the Fast Track event invited attendees to scan an ad for Kiwi shoe polish featuring a Van Gogh self-portrait, which prompted the artist to leap from the poster, ready to try on several pairs of shoes with the user.

The AR app was a remix of an award-winning Ogilvy campaign that “completed” various famous portraits by adding views of the subjects’ shoes. Partnering with Ogilvy and Google, MediaMonks was able to bring the Van Gogh portrait to life by stepping into the real world and responding to different shoe styles with animations and props. The experience of helping the famous artist try on shoes is humorous and helps the user relate to an esoteric character they certainly recognize, but may not have previously understood.

AR and VR are more consumer-friendly each day, offering several ways for you to reach them with ease.

 A large-scale project with emerging tech can be intimidating. While it can be difficult to identify where to begin planning bigger projects, there are several levels of engagement to design for in VR and AR, allowing brands who want to dive into emerging tech to start small and provide increasingly immersive experiences from there. One way to provide an engaging experience is to focus on one simple, primary mode of interaction. For example, the Kiwi case demoed at the Fast Track event required only a poster and a mobile device to render the character on the screen, along with some options for interacting with that character. This simple template could be useful for brands new to the emerging tech space, who also want to provide engaging, emotional experiences.

Seeing is believing.

And just like how the app brings the print ad featuring Van Gogh to life, you can use your app to remix existing assets from current or past campaigns, thereby boosting their ROI—not bad for a budget-strapped brand who wants to dive in on AR.

If a business has a little bit of extra room in their budget, they can elevate the same experience detailed above by staging it in an appealing way. While the primary mode of interaction might be to bring a character, object or other experience into the real world via AR, offering a set for participants to play with the experience can further immerse them. In the case of the Kiwi app, MediaMonks constructed a tiny room at Cannes Lions International Festival of Creativity decorated to look like a city street you might find in a Van Gogh portrait, allowing participants to step into the artist’s world (without the need for a VR rig and other expensive tech).

If one theme is clear across the Innovation Festival, it’s that the quickening pace of new media and platforms are challenging brands to think outside the box in providing new, experiences to enchant and inform their audiences. We were happy to allow guests into the hallowed chambers of the MediaMonks monastery to impart a secret or two about our process. But can the students surpass the master? We look forward to seeing what you create!

If you thought that mind control was only real in Marvel’s cinematic universe then think again.

Vodafone announced the launch of a 4G network on the moon in 2019—and since it’s now possible to swim with sharks in VR using 4G, why wouldn’t we try to raise the bar even higher to transmit brainwaves via 4G?

Together with Vodafone, ACHTUNG! mcgarrybowen, and Made.For.Digital. we made it possible for anyone to feel like Charles Xavier and control a race car in virtual reality (VR) — and real life — with their brain.

Stage One

The first stage of the project, dubbed Formula Brain (the pun works a tad better in my native Dutch tongue), launched earlier this year in twelve Vodafone stores across the Netherlands.

Once store visitors took place in the driver’s seat, they were fitted with Emotiv EEG headsets, which allows for the brainwave frequencies to be registered and transmitted. The headset calibrates four types of brainwaves ranging from low to high frequency — Theta, Alpha, Beta and Gamma.

The frequency measured by the EEG headset is then filtered into code that was used to control the car. In short; higher focus means a higher frequency and so higher acceleration. To complete the life-like feel of the experience, we used Gear VR headsets that immersed racers in a circuit based on the Red Bull Ring.

Stage Two

Over the course of three weekends, participants competed for a coveted ticket to the second stage: a race event for the five best racers at a local circuit. Using the same EEG headsets for measuring brainwaves, the real life race event offered a more realistic driver’s experience — complete with Vodafone suits, live telemetry data reflecting the car’s performance, and a direct camera feed from inside the car for the racers to focus on.

Inside a custom-built dome, placed in the circuit’s center, the racers claimed the driver’s seat once again — this time controlling a physical race car on the track outside. Vodafone’s 4G network was the connecting factor in this stage of the game, making sure the racers brainwaves were transmitted to the car on the track — all in real time. In the end, one racer was crowned the king of Formula Brain and won a ticket to the 2018 Grand Prix of Austria at the Red Bull Ring.

Brainwaves + 4G Network = Dream Team

Keeping in mind that Formula 1 drivers go through years of extensive training to maintain focus for seventy laps at soaring speeds, it’s even more exciting that we allowed regular people to have the same experience with barely any training — all using the power of their brain.

Adding our experience with VR and real time telemetry data handling to the equation — think of projects like Red Bull: Air Race Live VR, the Old Spice: S.Q.U.I.D. and more recently Nike Game of Go — promises more exciting possibilities for the future.

Maybe down the road we’ll see what else these technologies holds for creating some high-octane brand experiences — on this planet and beyond.

This is one of our favorites – and that says a lot when you consider the reach Amazon’s Jack Ryan Experience had in terms of buzz and coverage. Here, VR Scout covers our work with Amazon Studios at Comic-Con –

Rappelling from a helicopter and zip-lining in VR will have your heart pounding.

What better way to tease the world-wide release of the new Tom Clancy’s Jack Ryan series on Amazon Prime then to drop thousands of Comic-Con fans into the boots of Jack Ryan himself.

Dropping down in a conflict zone under fire made what happened next all seem like a quick blur. I “crossed” a ridiculously long unsteady plank of wood to enter another bombed out building. I picked up a weapon and began engaging with enemies, ducking behind crates that were both physically and virtually there.

Walking further down a hallway I came to a balcony. I grabbed onto a zip-line and literally just walked off the side of the building. I land on what could only be safety mats and was quickly ushered into a vehicle where I had to drive myself to safety. What the hell just happened?

Keep in mind I was physically walking, grabbing, and flying with a VR headset on, the entire time.

Did I mention that this was all outside under direct sunlight?

Created in collaboration with MediaMonks, a team of over 300 worked non-stop over a few months to get this ready for Comic-Con. Quite an accomplishment considering the public will get to experience it this week only. This really should be its own theme park.

The immersive Jack Ryan Training Field pushes you to uncover if you have what it takes to become a field operative. And I can tell you, I barely had what it took. It’s a nerve-racking experience that had me questioning my reality in front of me and forced me to push any fear of heights I may have had aside.

The Jack Ryan Training Field was also streamed live on Twitch, where viewers could interact and throw challenges in the way of me or others running through the course.

On top of the VR Training Field, Amazon also erected a massive escape room that will throw you into your first field assignment. Created in collaboration with AKQA and Unit9, you can dig deep to thwart an extremist conspiracy, uncover a plot of double-crossing, and obtain classified intel. The Dark Ops escape experience is run as a live drama, with actors, voice technology and immersive set pieces.

If you’re heading to Comic-Con this week, you’ll find Tom Clancy’s Jacky Ryan experience right outside Comic-Con at the corner of MLK Promenade & 1st Street. The season premiere airs August 31st on Amazon Prime.

This article was originally published on VR Scout on July 19, 2018.

Taking over an entire 60,000 square foot city block, Amazon created a massive event park that places you in the heart of the Middle East. Featuring one of the most extravagant attempts at an end-to-end warehouse-scale VR experience I’ve ever seen—it had everything. Repelling, zip-lining, plank walking, and even a car chase.

Over the years, I’ve grown accustom to VR taking center-stage at Comic-Con, experiencing everything from an immersive Blade Runner ride to a Mr. Robot simulcast at Petco Park. But for the most part, the VR demos were usually as simple as putting on a headset and enjoying the ride.

Not for Jack Ryan. Jack Ryan loves action—and stomach drops.

Upon entering the training park and receiving your Analyst ID badge, the first thing you’ll notice is the Jack Ryan Training Field, an overbearing obstacle course with a life size Bell Huey military helicopter propped a couple floors off the ground.

Before entering the immersive “training field,” I got the privilege of watching UFC fighter Ronda Rousey breeze through her run. Geez, I have to follow in her footsteps?

Slowly making my way up multiple flights of stairs to gear up in the cabin of the helicopter precariously perched atop a bombed out building made me realize how unordinary this VR experience was about to be. I strapped on a rappelling harness, HP Omex X VR backpack PC, a modified Oculus Rift VR headset, and hand foot trackers. My heart began to beat faster.

Then virtual reality happened. Staff members dressed as soldiers guided me to the edge of the sliding cabin door. In VR, I could see my limbs in front of me as I stutter stepped to the edge. It was clear that I was now flying high over a war torn city. I took a seat and nervously watched my virtual legs dangle in the air off the chopper side. The next thing I knew, I was rappelling out, actually hoisted by a crane down from the safety of the cabin. My heart was racing.

We’ve been called on many times to bring truly crazy ideas to life. From creating a robotic squid to a sandbox that turns into a VR race course, to bringing a rainforest to life with the biggest AR experience ever created.

And those were only this year.

We take great pride in being hailed as the industry’s go-to player for ambitious digital ideas, but we never set out to be the most innovative. We don’t have a “Chief Innovation Officer” or an “Innovation Department.” It’s never been our goal and yet, after 16 years in operation, having made the first Cannes’ Lions Innovation List in 2013 and with over 100 Lions to date, something seems to be going right.

Many ask us how it’s done and the answer is pretty simple: it’s saying yes to crazy requests for almost two decades.

We’d never built a remote robotic squid before, but we had built over 100 VR installations and all sorts of connected game experiences. So, when our friends at Wieden+Kennedy called on us to build a giant squid for Old Spice that would invite users to remotely control a tentacle to complete random “quests,” we had the confidence we could do it and threw everything we could at it.

There wasn’t a single part of this project that didn’t push us to our limits. From creating squid ink and inflatable respiratory sacks to creating a platform that could allow remote players to control a single tentacle to perform tasks in real-time. A lot of this was new for us, but the teams from across multiple offices and disciplines collaborated to push themselves to the limit and make it happen.

We’d never turned a museum into a rainforest, but Google Zoo challenged us to create the largest augmented reality experience on the planet, so we said yes. Making sure the walls aligned with the trees and the corridors with the forest’s paths wasn’t easy, but through a process of experimentation we developed the second ever experience created using Tango, Google’s new technology platform for computer vision, and transformed 10,000 square feet into a endangered rainforest.

So I guess it was only a matter of time before we were asked to launch a sandwich into space. Right? At this point there’s not a lot of wild ideas that feel out of scope. The sandwich launched this week. On the cover of the New York Times, it was featured in an article called “A Chicken Sandwich Hitches a Balloon Ride to the Stratosphere.” To call this ambitious is an understatement.

Why do we keep getting these briefs? When it comes to some of the more bizarre ideas, it’s often the case that although we haven’t delivered the exact project before, our clients know we can draw from our wealth of experience to give it our best shot.

We work with some of the world’s best agencies, and even for them some of these borderline insane projects might be a one-off experience. For us, we’re learning, practicing and refining our craft on a daily basis. The more shots into the stratosphere we’re asked to take, the bigger the chance of getting us, and our clients to the moon.

This article originally appeared on The Little Black Book on July 3, 2017.

VR and AR are the future, or so they say. With headlines like “2016 Will Be the Year That Sets the Stage for Virtual Reality” and “How VR Is Starting To Become Our Reality In 2017” taking over the hyper-saturated blogosphere, it might seem like VR and AR are the only technologies worth investing in.

But, as a virtual and augmented reality creative technologist, I’m constantly telling clients and colleagues to question this sentiment.

I work on some of the world’s most forward-thinking VR and AR projects every day and wholeheartedly believe in the power of these technologies to alter, integrate, or create new experiences and memories. At the same time, it’s also my job to think critically about technology and what new tools are best suited to meet client objectives.

VR struggles with two things: sharing and distribution. Since digital advertising lives or dies by social, the question many brands face is, how do we share cutting-edge VR and AR experiences? The struggle is that these experiences are not inherently shareable. VR hijacks your perception of the world by creating an illusion for you and your eyes only. Unless the VR experience is broadened through another channel, such as a teaser video on Youtube for example, if you want to share it with your friends, they have to be as big as a VR geek as you.

So if you’re a digital strategist or brand manager stretching yourself to explore how you can engage with these tools, I encourage you to stop, take a breather, and first read this post. It might just be that the best tool for the job is another technology entirely.

The Next Big Thing — Social VR Integration

Currently brands are only interested in creating their own VR experience, instead of exploring opportunities surrounding the VR hype. Brands creating their own one-off VR or AR experiences in hopes of creating a new channel for brand awareness is like creating an entire branded social network or community. This was very popular in the early days of social media advertising, but now you advertise within these social networks instead of trying to replace them. The same will happen with AR and VR.

For example, while Facebook is working on their experimental VR social platform Spaces, there already exists social platforms like the Rec Room and Altspace where users can virtually join others and talk, play games, and create things. If you’re hell-bent on creating VR and AR content for your brand, this is the trend to watch. The social iterations of VR will have an infrastructure designed for you to tell brand stories. This will be much cheaper and easier than creating your own application, and you can see the beginnings of this in Facebook’s AR studio. And similar to the way much of our current work is focussed on creating content for existing platforms, we’ll be delivering VR and AR in this way.

Experiences Unique to VR & AR

It’s undeniable that VR and AR can create unique experiences and express creative ideas that are not possible with any other technology. So if you’re dead set on creating a VR or AR experience, then make sure that you play to the medium’s strengths.

Take for instance this mixed reality case: ASM: Into the Wild, the world’s largest mixed reality experience to date. Using AR markers to place tiny virtual objects or characters on your table has been around for ages, but this is different. Using Google’s Tango technology, a museum was augmented into a living rainforest. Walls were transformed into trees and corridors into forest paths and guests were given tablets which they could walk around and interact with endangered animals, something they can’t do in real life. The magic of AR is not just bringing virtual animals to your dining room table, but to your entire environment.

Or perhaps my favorite example, Google Earth VR. Imagine having the whole planet at your fingertips, one moment deftly flying through the skyscrapers of midtown Manhattan, and the next sitting peacefully at the top of Mount Everest. Taking data from satellite imagery and 3D photogrammetry, the environments in Google Earth VR are majestic, and are some of the most presence inducing in all of VR. There’s nothing more mind blowing than virtually standing outside of your apartment looking up at your window, knowing that you’re actually inside, decked out in VR gear. Furthermore, the experience really gets you excited about the VR’s future. If this is what it looks and feels like now, what will it look like 10 years from now?

The Future

But what about other VR experiences that could well and truly be useful for real life? In the automotive industry you could allow potential customers to drive through impossible test drives, while changing features and testing during the same experience.

Or, think about how VR can provide an amazing tool for training new professionals in technical fields.

What if you could train to be a wind turbine technician by running through a variety of possible scenarios before you ever step foot in one? Or, imagine learning a language. How useful would it be to simulate the feeling of language immersion by placing someone learning french in a Parisian cafe, and they can only navigate the experience by correctly pronouncing various phrases?

Or museums! Imagine going to the MOMA and seeing Jackson Pollock via an AR tablet ferverishly throwing paint at one of his canvases!

The possibilities are endless, but that doesn’t mean that every possibility is right for your brand. So before you spend a whole lot of your (or your client’s) money, ask yourself this. Why do you want to create a VR or AR experience? Do these technologies really provide your brand something better than other technologies? Or is your idea just a gimmick? If you’re looking for reach or engagement, then maybe wait a while before reaching for a VR headset. As the VR and AR markets mature, the channels for telling your brand stories will mature with them.

This article originally appeared on Shots on July 5, 2017.

In February 2017, together with World Wildlife Fund, ArtScience Museum and Google Zoo, MediaMonks launched a large-scale mixed reality experience “Into The Wild.”

All backed by the effort to help people in Singapore experience the devastating effects of deforestation and learn more about some of the world’s most endangered species and their habitats.

It was the world’s first Tango-enabled smartphone Lenovo Phab 2 Pro, and guided visitors through personalised digital adventures, which started with augmented reality (AR) on the ground floor of the exhibition space, before transitioning to full virtual reality (VR).

The end of the experience shifts back to AR, where users go up to the fourth floor for an experience that includes planting a virtual tree.

Transforming over 1,000 square meters of the Singapore ArtScience Museum into a virtual, interactive rainforest, making it the largest AR experience in the world, and second ever AR museum experience developed using Google Tango.

And it wasn’t easy. From a technical perspective, we faced the massive challenge of how to accurately and smoothly map a virtual rainforest onto a physical and dynamic museum space, making sure the walls aligned with trees, corridors with the forest’s paths, and that we worked our way around the museum’s existing exhibitions and staging.

With this, we can complete the alignment. And since Tango reports the device’s coordinates back in ecef coordinates, we can easily calculate the corresponding unity coordinates. Effectively we update the Unity camera with every Tango update we receive using this transformation.

What’s more, for every virtual tree planted, a real tree was planted in Rimbang Baling, one of the last pristine rainforests in Sumatra where the endangered Sumatran tiger lives. 5000 new trees were pledged in the project’s first month.

I hope by sharing this we can inspire the imagination of current and aspiring developers to build even more exciting AR/VR experiences that map to the real world. Go forth and conquer!

This article originally appeared on TechWorld on July 18, 2017.

So How Did We Do It?

To start with, if you’re augmenting the real world with virtual objects, it’s important that the device rendering your view (such as a smartphone, monitor, CAVE or head mounted device) is exactly aware of where it is in the real world.

For this, a device needs to know its position and orientation in a three-dimensional space.

In the case of Tango, where the augmentation happens on a camera feed, the position and orientation of the rendering device needs to be in real world coordinates. Only if the position and orientation of a Tango device is reported accurately, and fast enough, proper augmented reality is possible.

The fact that Google Tango does this for you is very cool because it allows developers to augment real world locations within their own virtual world which is different from Snapchat-like AR which, for example, augments bunny ears to your head.

With real world bound augmentations, you can potentially create shared AR experiences that revolve around and involve landmarks.

In this case, it allowed us to transform the ArtScience Museum into a lush virtual rainforest and from the user’s perspective, exploring the rainforest becomes as natural as exploring the museum itself because every corridor or obstacle in the virtual world matches a corridor or obstacle in the real one.

Google Tango Coordinates

We used Unity3D to create our virtual world. To begin, we assured our Unity developers that they wouldn’t have to worry about alignment and were free to design the virtual world using whichever position or orientation they liked, as long as it was true to scale.

Developers familiar with Geographic Information Systems (GIS) know there are a lot of coordinate systems out there called “datums”. Historically, a lot of institutes developed their own, but since the introduction of GPS, the US developed WGS84 which is the most often used for commercial devices.

The great thing about this coordinate system is that it is Cartesian, it calculates in meters, and uses the centre of the earth as its point of origin. This is important because, in a properly mapped environment, Google Tango can give you its exact position and orientation on the globe and gives you these in WGS84.

Google Tango calls these coordinates ecef coordinates, so, we’ll call it ecef also.

Determining the Correct Approach

The next step is to ensure our Unity world overlaps with the real world so we can achieve augmented reality. Two approaches to solve this come to mind.

1. Transform (move+rotate) the Unity world to sit on top of the ecef coordinates of the museum.
2. Transform (move+rotate) the ecef Tango device coordinates into Unity world coordinates.

The approaches are 80 percent the same, as in both cases you have to calculate the transformation from virtual (Unity) to real (ecef). The difference, however, lies in whether you transport the virtual world onto the real one (approach one), or whether you transport the real camera onto the virtual world (approach two).

To determine which approach is best, we had to see what these coordinates look like in a real use case. Here are some examples of how Unity coordinates look:

Object A: [10.000, 63.250, -11.990]

Object B: [-92.231, 33.253, -62.123]

By contrast, below are two examples of how ecef coordinates look:

Hilversum MediaMonks HQ 2nd floor near the elevator: [3899095,5399920414; 353426,87901774078; 5018270.6428830456]

Singapore ArtScienceMuseum in front of cashier shop: [-1527424,0031555446; 6190898,8392925877; 142221,77658961274]

Obviously, the ecef coordinates are quite large numbers. In fact, it’s clear that single-precision floating points (or floats) are going to have a lot of trouble with these.

Without going too much into detail about floats, it’s important to note that performing arithmetic with numbers around 10–6 with numbers around 106 means that you significantly lose accuracy.

In addition, there’s also no way of getting around the fact that a lot of 3D programming is done around 10–3 to 103 (think of transformation, model, view, or projection matrices).

To understand this further, I recommend watching this video as it demonstrates this point perfectly. It shows a fighter jet taking off from around the origin [0; 0; 0] with a camera following it and, as its own position gets larger and larger (as well as the camera’s position), the floating point calculations become less and less accurate.

Imagine then what the error would be if the coordinates of your camera are like the ecef coordinates shown above? You would be combining fine scaled rotation values with very large position values. The error in the result will be enormous.

Add to this the fact that Unity is hard coded to work with floats (rather than double-precision floating points, or doubles), and the fact is that we can’t afford any large errors in AR. It’s therefore clear that approach one is unfeasible. This is because the camera needs to stay relatively close to the origin to avoid precision errors.

So, we proceed with approach two which is to transform the ecef Tango device coordinates into Unity world coordinates.

Find the Transformation

Transformations between coordinate systems in 3D graphics usually entail finding translation (positional), orientation and scaling values.

Each of these three concepts act in 3D space, so they must describe their positioning, rotation, and scaling for each of the three axis (x, y and z). This gives us nine values to find.

The nine unknowns are a hint of how many equations you need to find these nine unknowns. This is important when determining how many real world coordinates are needed to anchor our virtual world to.

Our initial idea was to create a transformation that would deal with all three concepts (translation, rotation, and scale). However, due to difficulties, and the fact we were able to design our virtual world true to scale, we decided to drop accounting for scale and focus on translation and rotation only.

This meant that effectively we now only need to find six unknowns.

Calculate the Transformation Matrix

At this point in solving this challenge, we’re down to finding a transformation matrix that only accounts for location and rotation. Luckily, this problem has been solved a million times already by Computer Science students.

If you simply search Google, you’ll find countless examples of how to transport a rigid body from one coordinate system to another. This is one example that will get you there 90 percent of the way.

Finding a transformation matrix revolves around minimising the sum of squares error between two sets of data points. The following method is tailored for this problem since it deals with rotation and translation separately.

Conceptually, we approach this by picking a point in the real world, and we say that that point corresponds to another point in the virtual world.

Basically, the worlds are anchored to each other on that point. However, as you can imagine, choosing a single point as an anchor still allows the worlds to pivot around the anchor, in which case they will be misaligned most of the time.

Therefore, to place the virtual world squarely on top of the real world perfectly, you need at least a few anchors. Depending on the number of unknowns you’re trying to find, you need an equal or more amount of equations.

Equations can be derived from known pairs (in this case ecef and Unity coordinates). In this case, a total of 3 pairs (or anchors) is enough to allow us to find a full 3D transformation matrix.

The idea is that you choose N amount of points (at least three) in the real world, and find their ecef coordinates. Then, you go into the virtual world and place a point on their corresponding virtual locations (so N in total). For the museum project, we used 10 easy to find landmarks at the base of each pillar inside the museum.

For this, we used a third-party library called Math.Net that allows us to do linear algebra with doubles. You only have to run this code once at the start of the program.

The result is that now have 10 ecef coordinates and 10 Unity space coordinates shaped in a circle, resulting in 10 pairs of coordinates. The next step then is to apply the steps discussed in this article and find a transformation matrix that allows us to transform a point from ecef space to Unity space.

We ran into a few problems while implementing this. For example, Unity is a left-handed coordinate system, while ecef is right-handed. And the article we referenced above also used row major ordering, while the library used column major ordering.

This makes filling, transposing, and multiplication ordering of matrices different. We eventually overcame all of these problems through careful reasoning, and not trying to take too many steps at the same time.

Apply the Transformation

Following the previous step, we have a transformation matrix we could call ecefTunity, (or unityTecef depending on how you calculated the matrix). So, transforming a point in ecef space to unity space becomes as trivial as:

MediaMonks Games unlocked its own achievement this November, as the most playful proposition in the global creative production company celebrated turning ten.

During this time the industry has gone through its own transformation. From Flash bursting onto the scene, to the rise of mobile and gaming for the masses, it’s been a decade packed full of developments.

Flash Back

The games industry looked a lot different pre-2006. There was very little advertising and branded games weren’t that common. Early attempts weren’t always on point. Two in particular spring to mind — Pepsiman, which was so out-of-the-box that it wasn’t comparable to anything out there.

The second is Crazy Taxi, which paved the way with high production value and included some sublimely subtle branding. Around 2006 Flash came into play, which opened up both opportunities for branded games and helped the industry grow as a whole. Along with Flash came the opportunity to make games for more web browsers, and because games were now available on more web browsers, it opened up a whole new audience for the industry.

Typical of the Flash era were games like Dofus and Habbo Hotel. Overall it became far easier to both make and play branded games. And because more audiences could be reached, the interest in branded games from advertisers grew.

iPhone Changes the Game

A few years after Flash came another game changer — the iPhone 3G. By offering licenses for $99 (instead of thousands) the games platform was suddenly affordable. The iPhone proved to be a liberator for games, as developers — who were no longer at the mercy of a select few platforms — could themselves create games for the app store. I consider it to be the industry’s Braveheart moment. With affordable licenses came clamorous cries of freedom.

Entertainment developers switched their focus to making games for mobile devices, whilst the advertising industry focused on targeting mobile phone users. The advertising and entertainment industries suddenly found themselves together on the same platform and both industries had to compete for consumers’ time. In those days, I think the advertising industry was lagging a bit behind the entertainment industry in understanding what motivates people to play games. Back then we prioritised the message over the fun and, as Candy Crush emphatically showed, it didn’t work. No matter how many brands made variations of Candy Crush, none were ever half as popular or successful as the entertainment version.

For us in the industry it became clear we had to think about games in a different way. We needed to think how we could make amazing games that were appealing, yet still incorporated branding.

From Niche to the Masses

Around 2011 — as iPhones became synonymous with cool — there was also a cultural shift in how game players were perceived. Gamers were no longer seen as nerds; gaming was now the norm. Within a relatively short period there was now the audience and cultural perception that made games a primary medium for brands to interact with consumers. From there, the industry started thinking about how games could be integrated into other mediums and experiences.

We moved from the likes of Pepsiman and co. to producing quality branded games. It was clear to see how far the industry had evolved. At MediaMonks Games, we produced Macy’s Wish Writer which sums up the progress perfectly. Within the app we merged research, tradition, and technology, packed it with mini-games, narrative and activities, all with seamlessly integrated branding.

Here’s a look at a few examples from our 2016 Games Reel–

In the wider industry, games like Pokemon and LEGO Star Wars were flawlessly designed to bring consumers into contact with both the brand and the product. Advertisers had perfected the art of reaching consumers via games.

Golden Times Ahead

It used to be that as a games developer you either wanted to be associated with Hollywood AAA or Indie groups. Over the last few years we’ve seen the rise of a third group; the branded game developer. They are perhaps less vocal, but they’re now more relevant than ever. And importantly, they’ve produced some brilliant games.

Right now advertising games are more creative and immersive. Developers and creatives haven’t been tied down by the limitations of only being able to produce games for consoles. I think you can really see that in some refreshingly creative games for mediums like AR/VR (check out the VR game Accounting). The creative progress in advertising has definitely been noticed by the entertainment industry too. Giants like Sony, who have traditionally been console-only, are now branching out into other mediums like streaming services to compete creatively.

What this means for the future only time will tell. It’s been a great ten years and it’s a great time to be involved in the industry. Whatever happens, I have no doubt the new platforms, creativity, content and battle for people’s attention will only make better games. And in my opinion, it’s only a win-win for consumers!

This article was originally posted on HuffPost on September 12, 2016.