The camera is the new browser. Augmented reality has steadily improved over the years, allowing for entirely new, immersive experiences that remove the friction between brand offerings and their audience. The tech’s maturity is of particular interest for brands to add to their marketing mix thanks to its reach: through AR, any part of the user’s direct environment becomes a surface to advertise and build relationships.

So when I say the camera is the new browser, what I actually mean is that this emerging technology requires organizations to rethink how they tailor their message to their audience, just like with social media before it. We’re very bullish on AR, believing it will change the way people interact with the world and with each other, quickly and at scale. Brands should be a part of this process, helping to create those experiences—and lucky for them, there are ample opportunities for AR to create real business value. If you’re just wrapping your head around the new possibilities of AR or have considered investing in the tech for some time, consider these three questions early in your design process.

Why Do Consumers Love AR, and Where Do We Fit?

Augmented and virtual reality hit mainstream consciousness around the same time, making it easy to confuse the two. Throw in mixed reality and the differences between the platforms becomes even fuzzier. Which reality fits your idea depends on whether you want to tell a story to the user, or prefer to give them the tools to tell their own. If the first option is your goal, then VR or mixed reality is best. But when it comes to the second, AR makes the perfect environment for expression and experimentation.

While web AR still hasn’t launched, we were able to create a bit of a stopgap solution with our Dragon Spotting experience to promote Disney’s film Pete’s Dragon. While the web-based experience doesn’t load a model directly in the space around users, it prompts them to seek out specific environments—the family TV, for example—and pulls from the Google Cloud Vision API to identify whether the user has found it. If there’s a match, an animation of the dragon triggers to show that the user was successful. When Web AR really kicks off, such AR experiences will become much more sophisticated and allow for mass adoption.

Augmented reality is an exciting platform that asks both users, brands and developers to really reconsider the way they interface with machines. The technology is ever evolving, and brands shouldn’t wait to invite the platform into their strategies. By understanding what users love about AR, what the current limitations are and where it’s going next, organizations remain ready for this next big revolution in tech.

AR isn’t the only big trend to look forward to this year.

The most effective uses we’ve seen recently are with social platforms like Facebook and Snapchat, which have integrated AR filters into their apps’ respective camera features. This provides fun, out-of-the-ordinary opportunities to photograph and share (think funny facial effects or whimsical monsters joining them on the sofa) that fit seamlessly within the platforms. We’ve seen time and time again in this environment that AR is an instinctual medium: people react quickly and are comfortable in exploring, trying out and testing its features without the feeling of frustration or confusion that can be typical of entirely new interfaces.

As far as user experimentation is concerned, AR is very effective in mapping out information and allowing the user to explore or dive deeper however they choose. Why look at a tough-to-read diagram labeling out info when you can view it directly hovering above the object right in front of you? We set out to achieve this with an AR app that helped customers get acquainted with the hottest features of Weber’s Pulse grill. The app provided a virtual layer of informational hotspots that explained the grill’s parts and showed how they worked with animations. For enterprise use, this speaks to the original promise of AR: that users can easily see contextually relevant information without having to use their hands (i.e. like a car’s parts labeled in AR as a mechanic works on it).

On the consumer side, this method makes for a memorable example of persuasive marketing. With the Weber Pulse app, users had the ability to grill a virtual meal via AR first, alleviating any concerns before trying the real thing. By mimicking the real-life act of grilling BBQ, their experience with the product “sticks” better in their mind than if they’d just watched an ad or seen the grill on a shelf. This improves the chance of conversion or whatever KPI you’re tracking.

How Do I Anticipate and Capitalize on AR’s Limitations?

AR may have settled into broad appeal over the years, but it’s still an emerging technology with room to grow. Early in the process of concepting your experience, take the time to understand and embrace the limitations of the platform you choose. This is when you need to assess the specific platform that best supports the AR experience you want to provide: should you develop for a Facebook or Snapchat filter? Is a custom, fully-featured mobile app a better choice?

There are pros and cons to everything. Photo filters on Facebook and Snapchat both enjoy robust engagement and wide reach thanks to their integration with social platforms. AR in this environment is usually only mildly branded, where the focus is less on the product or brand and more on fun, shareable content that somehow aligns with it. But these filters’ ability to instantly load comes at a cost: a restrictive file size that limits functionality. Initially, this seems like a challenge. If you recognize this limitation and design a very focused experience around it, you can end up with an enthralling, snackable experience that fits the platform so well.

A dedicated AR app, meanwhile, provides greater functionality and creative opportunities through its integration with ARCore and ARKit in a mobile OS. While this platform allows you to do much more (like the Weber grill app discussed above), it’s costlier in resources and has a higher barrier of entry for consumers who may be less willing to download an app. That said, dedicated AR apps make excellent tools for retail events, trade shows or public installations, like the narrative AR experience we made with Santander.

Where is the Technology Headed Next?

New platforms for hosting and developing AR experiences will aim to reduce friction between brands and users, which is really the big promise of AR: providing a direct and intuitive interface that blurs the line between physical and digital. Brands who wish to advertise through AR will need to evolve with new possibilities and formats that emerge.

To get an example of an AR platform emerging right now, check out Google Playground, which is directly integrated with Google’s Camera app—at least for some users. The Playground feature offers a new type of interaction: rather than simply pose with a filter, users can freely place virtual objects in their environment before snapping a photo to send. They’re a lot like the stickers that are common in several photo editing apps, except these objects (called “playmoji”) are three-dimensional and stay in place as users move the camera around.

AR is still maturing, a point I touched upon at CES panel on where the AR experience is going. The next big leap for augmented reality will be web-based AR, something that Google has shown but isn’t yet widely available. In the meantime, the startup 8th Wall has tried to carve out a unique space in being the only platform for developing true, web-based AR experiences. Loading up AR experiences directly in a mobile web browser will dramatically reduce friction for the user, doing away with the need of downloading apps or navigating through a library of effects. Instead, you need only point them to your website to activate. With 5G on the horizon set to unlock connection speeds, users will soon have unprecedented access to more immersive experiences.

With omnipresent technology on the horizon, are we headed toward a cyberpunk dystopia?

That was the big question MediaMonks posed to the audience at its Fast Co. Fast Track session, offering attendees food for thought to mull over with drinks. Rejecting a future in which pop-ups and banner ads vie for our attention literally everywhere we look, speakers Eric Wagliardo and Jouke Vuurmans offered a more optimistic view. By using emerging technology like VR, AR and artificial intelligence to delight audiences, Wagliardo told the audience, “we can use creativity unconstrained by the past and the laws of physics.”

The theme of the talk anticipates a near-future in which wearables or ubiquitous AR will revolutionize the way we interact with media and technology, just like how the iPhone ushered in the era of smartphones. As these technologies become more consumer-friendly, brands might be biting their nails wondering if they’re ready for the new digital landscape. Are you?

The focus of your VR or AR experience doesn’t have to be grand to be impressive.

One of the more compelling details from the talk was about a humble plank of wood used in the Jack Ryan Experience, a 60,000 square-foot activation that pulled together some of the most cutting-edge VR technology to provide a 4D, immersive experience. Through a VR headset, the plank transformed into a startling chasm for participants to cross in virtual reality. That such a basic material in reality could become an intimidating environment to traverse demonstrates the power to create compelling experiences from just a simple object. With so many elements and gorgeous tech to compete with, who would expect a plank of wood to steal the show?

VR can be tricky, though: without a proper “hero device” like the iPod or iPhone to put the tech in almost everyone’s pockets, VR experiences must often be tethered to a specific environment for participants to don a headset and enter your world. This can be a big investment in terms of reserving a space as well as providing a rig to host the experience, though it’s well worth it if you’re hosted within a venue that allows you to reach your audience in a meaningful way. For some brands, a simpler solution might be to create an AR experience that participants may enjoy via nearly any mobile device.

Another benefit to AR is that it can enable a charming connection with objects or pre-existing campaigns that wasn’t possible before. One of the demos featured at the Fast Track event invited attendees to scan an ad for Kiwi shoe polish featuring a Van Gogh self-portrait, which prompted the artist to leap from the poster, ready to try on several pairs of shoes with the user.

The AR app was a remix of an award-winning Ogilvy campaign that “completed” various famous portraits by adding views of the subjects’ shoes. Partnering with Ogilvy and Google, MediaMonks was able to bring the Van Gogh portrait to life by stepping into the real world and responding to different shoe styles with animations and props. The experience of helping the famous artist try on shoes is humorous and helps the user relate to an esoteric character they certainly recognize, but may not have previously understood.

AR and VR are more consumer-friendly each day, offering several ways for you to reach them with ease.

 A large-scale project with emerging tech can be intimidating. While it can be difficult to identify where to begin planning bigger projects, there are several levels of engagement to design for in VR and AR, allowing brands who want to dive into emerging tech to start small and provide increasingly immersive experiences from there. One way to provide an engaging experience is to focus on one simple, primary mode of interaction. For example, the Kiwi case demoed at the Fast Track event required only a poster and a mobile device to render the character on the screen, along with some options for interacting with that character. This simple template could be useful for brands new to the emerging tech space, who also want to provide engaging, emotional experiences.

Seeing is believing.

And just like how the app brings the print ad featuring Van Gogh to life, you can use your app to remix existing assets from current or past campaigns, thereby boosting their ROI—not bad for a budget-strapped brand who wants to dive in on AR.

If a business has a little bit of extra room in their budget, they can elevate the same experience detailed above by staging it in an appealing way. While the primary mode of interaction might be to bring a character, object or other experience into the real world via AR, offering a set for participants to play with the experience can further immerse them. In the case of the Kiwi app, MediaMonks constructed a tiny room at Cannes Lions International Festival of Creativity decorated to look like a city street you might find in a Van Gogh portrait, allowing participants to step into the artist’s world (without the need for a VR rig and other expensive tech).

If one theme is clear across the Innovation Festival, it’s that the quickening pace of new media and platforms are challenging brands to think outside the box in providing new, experiences to enchant and inform their audiences. We were happy to allow guests into the hallowed chambers of the MediaMonks monastery to impart a secret or two about our process. But can the students surpass the master? We look forward to seeing what you create!

It’s easy to see that selfie lenses are one of the most accessible applications of AR available: they’re intuitive, provide instant gratification and load on almost any mobile device.

Still, some brands may find the tech too intimidating to support. Our new guide to designing and developing Facebook Camera Effects is written with these brands in mind, providing readers with everything they need to know about this popular use of AR, including key design insights, technical considerations and what to expect from the tech’s future iterations.

We understand apprehensions around building AR experiences; investing in any emerging tech can feel like a gamble, especially for smaller to mid-size organizations. But developing AR lenses means subscribing to a “less is more” philosophy: they require less budget, fewer assets and lower barrier of entry. This mix of benefits makes the medium uniquely accessible to brands of any size that want to dip their toes into a fast-moving, emerging field of tech. The result? Snackable, interactive experiences that prompt users to share their stories and forge deeper, more personal connections with their favorite brands in the process.

Learn why you should make your first Camera Effect—and how.

If you have any interest in tech, chances are you’ve watched an episode or two of Black Mirror, Netflix’s massively popular series that dives into uneasy relationships between people and tech (and if you haven’t, we suggest you start). While the drama sets its sights in the near future, its power—and proficiency for inducing anxiety in viewers—stems from the fact that the questions it explores are just as applicable to the present day.

If you take solace in the fact that Black Mirror’s terrifying tech hasn’t yet (or never will) come to fruition, don’t get too comfortable: the future is now, they say, and the mad minds at MediaMonks Labs are releasing a web app that brings the show one step closer to reality.

In the season 2 episode “White Christmas,” a neural implant called Z-Eye provides an augmented reality interface within a user’s field of vision. Among its features is the ability to block other people in real life, which makes them appear as gray silhouettes with muffled speech. Inspired by the feature, MediaMonks Labs built an AR filter that likewise renders a user’s body as a pixelated, black-and-white shape (try it here–best enjoyed on a tablet or laptop).

While it’s not quite as advanced as what you see on Black Mirror—it doesn’t require an implant, thankfully, and works with only one person in the view—it demonstrates how technology featured in speculative fiction is closer to reality than audiences might anticipate. In this case, the technology is so accessible that it requires little setup for users to play with. “What makes this so cool is that it’s running in your browser and any out-of-the-box webcam,” says Joe Mango, Creative Technologist at MediaMonks Labs, who built the tool. “It’s clickable and usable at any time.”

Block Out Anyone—No Implant Required

The filter is powered by BodyPix, an open-source machine learning model that enables body-part segmentation. “Segmentation” is a process in which a machine takes an image or video and separates the pixels that belong to a person versus those that do not. It’s the same idea behind using a green screen, no screen necessary.

What makes BodyPix unique is that it not only separates a body from its surroundings, but can also segment that body into 24 specific parts; for example, the left side of the face versus the backside of a subject’s right arm. Such technology could have several applications, like creating precise, body-tracking augmented reality filters or image editing.

With BodyPix as an underlying machine learning model, the Labs-developed body-blocking tool isn’t TV magic. And even though the technology that makes it work is quite complex, the concept behind body blocking is relatedly simple.

Rendering formless, pixelated bodies in real time, the Labs-developed body-blocking tool isn’t movie magic. The tech powering it is quite complex, but the concept behind body blocking is rather simple and relies on editing two identical video feeds together. On one end, the tool takes the camera feed and applies a shader that distorts and pixelates it. On the other end, a separate version of the video is segmented to cut out the user’s body—as if you took the magic wand tool image-editing program to select an object and delete it. This version of the video is pressed on top of the distorted one, resulting in a pixelated, black-and-white body surrounded by an otherwise normal environment.

Finding the Science Behind Science-Fiction

While the tool isn’t totally accurate to the technology as seen in the show, its underlying mechanics can help viewers understand and envision a technology that previously may have seemed possible only in the realm of science fiction. It also demonstrates the value of trying to make the impossible possible through cutting-edge and emerging technology.

“It may seem unusual that a creative and production company like MediaMonks is doing such in-depth technological research,” says Mango, though that openness to experiment can help brands find new opportunities to take experiences available in their worlds and replicate them for fans in ours. When Mango stumbled upon BodyPix in GitHub, for example, “My mind immediately went to the Black Mirror episode.”

The pitch to build upon the software to reproduce the startling Z-Eyes feature was surprisingly easy: “When I approached [Head of Labs Sander van der Vegte] with the idea, he said, ‘That sounds cool, let’s do it!’” recounts Mango. “In a nutshell, it’s that simple to get sign-off on a project thanks to the open, creative environment.” But considering Black Mirror is often a case study in troubling uses of technology, this zeal to experiment shouldn’t be viewed as flippancy. “We’re able to peer into Pandora’s box to see the interesting possibilities that are available—and the questions they raise,” says Mango.

And that ties back to how such projects can help end-users better apply fictitious, futuristic tech they see in the media to their own lives—and to anticipate or wrangle with some of the ethical dilemmas that inspire speculative fiction. And of course, such projects are just plain cool, too. In essence, it all boils down to making a connection that breaks down barriers between the real and imagined.

There’s no denying that online retailers make shopping more convenient, but in some ways the digital shopping experience falls short. For one, you can’t try on clothing or fashion accessories before having them shipped to your door, which means you might discover after days of waiting that something doesn’t fit. To get around this, customers might buy several versions of an item in different sizes, which can be costly to them in the immediate term while contributing to scarcity of high-demand items—for example, sneaker releases, which have generated $20 billion in U.S. sales alone in 2017 and have a resale market valued at over $1 billion.

There has to be a way to instill purchasing confidence in consumers, which is why one of the latest projects to come out of MediaMonks Labs, our internal R&D team, is an AR tool that uses computer vision to accurately measure shoppers’ feet. While it may not sound like the most exciting use of AR (no, it won’t transport virtual monsters to your living room or render a shoe directly on your feet), “it speaks a lot to what else we do beyond the crazier stuff,” says Joe Mango, Creative Technologist at MediaMonks Labs. “We explore things where there’s need, and see how feasible—or infeasible—it is. There’s high interest in exploration and discovery.”

Sander van der Vegte, who leads the Labs team, discusses the importance for more practical uses of the tech in both its adoption and signaling value to consumers: “Everyone’s pushing AR, but why is it useful? This tool is designed purely to solve a specific problem, and is functional in doing so.”

Improving CX with a “Form Follows Function” Approach

While most people are aware of their shoe size, fits aren’t so cut and dry; shoppers often get some shoes in one size and some in another. “Shoe shopping becomes more complicated when different brands and shoes have different sizing charts, making it harder to gauge what fits,” says Mango, who brought the tool to life. It can get around this issue by automatically translating a unit measurement into a specific brand’s sizing chart, giving the perfect size recommendation as users shop online. Ideally, the tool could exist within a retailer’s app, seamlessly aligned with the browsing process.

The tool has practical value for both consumers and retailers alike: shoppers know their purchase will fit before they buy it, and retailers don’t have to foot the bill of processing returns. This also means customers don’t have to worry about being blacklisted from making frequent returns, either—a fail-safe measure some retailers have used that erodes consumer trust and confidence. But perhaps most importantly, the tool serves as example for how retailers can use emerging technology to give consumers an obvious reason to provide their data. In this case, the length of their foot improves and optimizes the customer’s journey by enabling them to make the right purchasing decision.

Improving Measurements with Computer Vision

It’s worth noting that measuring via AR isn’t an entirely new capability: Apple, for example, integrates an AR “tape measurer” into the current version of iOS. While the feature is great for measuring perfectly straight lines or edges, it’s not the most accurate tool of measuring the length of a foot: there’s a lot of user error in moving the phone and finding the right angle to measure the foot.

The tool built by MediaMonks Labs is powered by OpenCV (a programming library for computer vision) and takes a different approach. Starting with a top-down photo of the user’s foot, it uses edge detection to determine and outline its shape. It then compares the size of that shape to a reference object whose size is defined and absolute—for example, a ubiquitous coin—which results in a consistently accurate measurement. While the output is in units of inches or centimeters, the tool has the potential to translate these measurements to brands’ custom sizing charts, taking the guesswork out of determining what fits.

Taking the Next Steps in Consumer-Friendliness

While the value of the AR tool is clear, added steps can be taken to improve usability. “There is a caveat to this method of measurement,” says Mango. “You need a flat color-contrasted background for accuracy,” which helps the computer vision model detect the foot’s edges.

Mango also discovered that the tool had difficulty differentiating a foot from the rest of a leg. The solution was simple: by wearing ankle socks he could segment the two with consistent accuracy. These limitations, along with the requirement of a universal reference object, present opportunities to make the tool more user-friendly for commercial use.

Despite its current limitations, the tool shows how brands might identify common issues on both the business and consumer side of the equation, then implementing technology in new creative, new ways to solve them. In the immediate term, these unique applications can provide the retailer with brand equity through emerging technology’s novelty. More importantly, though, they can deliver a better online or offline shopping experience to customers.

Here, our friends at SoDA (or The Digital Society) sit down with MediaMonks Founder and COO Wesley ter Haar, who is also on the SoDA Board of Directors.

In this year’s Global Digital Outlook Study with Forrester, SoDA expanded their inquiry in the area of innovation labs to uncover some interesting findings.

SoDA: Not only are agencies continuing to launch internal labs and incubators but, more importantly, they are making direct investments into these initiatives. Do you think agencies are finally getting serious about innovation and realizing they need to invest in R&D rather than just hope to do cool work as part of regular client initiatives? How does MediaMonks approach innovation and do you invest in it outside of directly funded client projects?

Wesley ter Haar: I’ve never been a great fan of the Lab moniker. It’s a weird way to silo innovation as a play-thing instead of making it a core part of the day-to-digital-day we all live in. 

…The US is our largest market and the most ambitious, APAC leads the way in user behavior, LATAM has some of the most creative visual talent I’ve seen and I’m always amazed by the creative ideas that bubble up from smaller European markets. It goes to show that constraints are never a reason to deliver mediocre work.

This article was originally published in The SoDA Report – with key findings from Forrester.

Wesley ter Haar: That X used to be 24 to 36 months, and has probably been whittled down to 9 months with the constant change that besets consumer behavior and client adoption. At MediaMonks we hire or acquire against an internal innovation roadmap based on where we see the confluence of people, products and platforms are headed. For us, that has meant the acquisitions of a VR-first production company and a connected commerce company, the launch of a digital first content company and a hiring spree to bolster our AR capabilities. So, yes, innovation is critical to the health of our business, but I don’t believe a ‘Lab’ is the way to make it central to who we are and what we do.

SoDA: MediaMonks is hired by client-side marketers (and agencies) to deliver cutting edge work. This year we found that Chatbots/Conversational Interfaces, AI/Machine Learning and Programmatic Advertising topped the rankings for anticipated impact and planned investment in emerging technology. Agency leaders and marketers were generally aligned on this front with one major exception… Virtual and Augmented Reality. Marketers are planning to make significant investments in VR/AR while agency leaders are lukewarm on the short-term marketing impact. Why do you think there’s such a big gap between marketers and agencies on this front?

Wesley ter Haar: I think this gap mostly represents the excitement for the future state of “The R’s” (augmented reality, virtual reality and mixed reality) relative to the technical maturity of current platforms and production processes.

From the SoDA perspective, this shows a mature agency landscape with many agency leaders trying to think from a client perspective, focus on value/budgets and look at reasonable metrics. As agency leaders, we have to make sure we educate clients on the now while planning for the future so we don’t miss the boat when the scale and spread of these technologies starts impacting brands, business and bottom lines.

SoDA: For many years, SoDA has tracked what marketers value most in their agency relationships and, on the flip side, what agencies think their clients value most in their partnership with them. “Expertise in Emerging Tech/Trends” and “Process/Project Management” are consistently rated in the Top 5 by both agency leaders and client-side marketers. How does MediaMonks balance the importance of project management rigor with the desire for clients to explore (and quickly deliver on) the latest technologies? Is there a healthy tension between these two factors?

Wesley ter Haar: There will always be tension between doing difficult things for the first time and delivering difficult things for the first time, on time. Our role is to explain risk, mitigate against it as best we can, and make the “fall down and get back up” process of research, innovation and iteration one that is transparent to clients.

…But, the level of comfort on the client side is going to rely on the quality of the process and the project management rigor around it. In the same way that ad agencies are not artists, digital agencies shouldn’t hide behind labs and a “Crazy Scientist” vibe when it comes to new technology and trends. It’s all about the practical application for clients and their (potential) customers.

SoDA: This year we asked agency leaders to identify strategic factors they saw as most critical to their ongoing growth and evolution. Not surprisingly, “Attracting and retaining top talent” and “Developing new services / capabilities” topped the list. Interestingly, very few looked at “Expanding to new markets/geographies” as an important part of their strategic plans. MediaMonks appears to be quite the opposite with offices now in Amsterdam, London, Stockholm, New York, LA, Dubai, Sao Paulo, Buenos Aires and Singapore. How do you approach geographic expansion and why has it been so central to your growth strategy? What challenges have you wrestled with in managing the business across such a broad geographic footprint? What do you see as the most exciting new markets?

Wesley ter Haar: To start with the reasons, it gives us the opportunity to recruit and retain talent at a much larger scale, and in turn helps us cater to the ambition many of our Monks harbor when it comes to working in other countries and cultures. 

We run all offices as a single P&L which sounds like an admin choice, but is a critical cultural component. We are one company operating across 9 countries and 10 offices, with teams and talent working across time zones. Budgeting, resourcing and planning needs to be seamless to make that work, and that’s been the operational focus from Day 1.

VR and AR are the future, or so they say. With headlines like “2016 Will Be the Year That Sets the Stage for Virtual Reality” and “How VR Is Starting To Become Our Reality In 2017” taking over the hyper-saturated blogosphere, it might seem like VR and AR are the only technologies worth investing in.

But, as a virtual and augmented reality creative technologist, I’m constantly telling clients and colleagues to question this sentiment.

I work on some of the world’s most forward-thinking VR and AR projects every day and wholeheartedly believe in the power of these technologies to alter, integrate, or create new experiences and memories. At the same time, it’s also my job to think critically about technology and what new tools are best suited to meet client objectives.

VR struggles with two things: sharing and distribution. Since digital advertising lives or dies by social, the question many brands face is, how do we share cutting-edge VR and AR experiences? The struggle is that these experiences are not inherently shareable. VR hijacks your perception of the world by creating an illusion for you and your eyes only. Unless the VR experience is broadened through another channel, such as a teaser video on Youtube for example, if you want to share it with your friends, they have to be as big as a VR geek as you.

So if you’re a digital strategist or brand manager stretching yourself to explore how you can engage with these tools, I encourage you to stop, take a breather, and first read this post. It might just be that the best tool for the job is another technology entirely.

The Next Big Thing — Social VR Integration

Currently brands are only interested in creating their own VR experience, instead of exploring opportunities surrounding the VR hype. Brands creating their own one-off VR or AR experiences in hopes of creating a new channel for brand awareness is like creating an entire branded social network or community. This was very popular in the early days of social media advertising, but now you advertise within these social networks instead of trying to replace them. The same will happen with AR and VR.

For example, while Facebook is working on their experimental VR social platform Spaces, there already exists social platforms like the Rec Room and Altspace where users can virtually join others and talk, play games, and create things. If you’re hell-bent on creating VR and AR content for your brand, this is the trend to watch. The social iterations of VR will have an infrastructure designed for you to tell brand stories. This will be much cheaper and easier than creating your own application, and you can see the beginnings of this in Facebook’s AR studio. And similar to the way much of our current work is focussed on creating content for existing platforms, we’ll be delivering VR and AR in this way.

Experiences Unique to VR & AR

It’s undeniable that VR and AR can create unique experiences and express creative ideas that are not possible with any other technology. So if you’re dead set on creating a VR or AR experience, then make sure that you play to the medium’s strengths.

Take for instance this mixed reality case: ASM: Into the Wild, the world’s largest mixed reality experience to date. Using AR markers to place tiny virtual objects or characters on your table has been around for ages, but this is different. Using Google’s Tango technology, a museum was augmented into a living rainforest. Walls were transformed into trees and corridors into forest paths and guests were given tablets which they could walk around and interact with endangered animals, something they can’t do in real life. The magic of AR is not just bringing virtual animals to your dining room table, but to your entire environment.

Or perhaps my favorite example, Google Earth VR. Imagine having the whole planet at your fingertips, one moment deftly flying through the skyscrapers of midtown Manhattan, and the next sitting peacefully at the top of Mount Everest. Taking data from satellite imagery and 3D photogrammetry, the environments in Google Earth VR are majestic, and are some of the most presence inducing in all of VR. There’s nothing more mind blowing than virtually standing outside of your apartment looking up at your window, knowing that you’re actually inside, decked out in VR gear. Furthermore, the experience really gets you excited about the VR’s future. If this is what it looks and feels like now, what will it look like 10 years from now?

The Future

But what about other VR experiences that could well and truly be useful for real life? In the automotive industry you could allow potential customers to drive through impossible test drives, while changing features and testing during the same experience.

Or, think about how VR can provide an amazing tool for training new professionals in technical fields.

What if you could train to be a wind turbine technician by running through a variety of possible scenarios before you ever step foot in one? Or, imagine learning a language. How useful would it be to simulate the feeling of language immersion by placing someone learning french in a Parisian cafe, and they can only navigate the experience by correctly pronouncing various phrases?

Or museums! Imagine going to the MOMA and seeing Jackson Pollock via an AR tablet ferverishly throwing paint at one of his canvases!

The possibilities are endless, but that doesn’t mean that every possibility is right for your brand. So before you spend a whole lot of your (or your client’s) money, ask yourself this. Why do you want to create a VR or AR experience? Do these technologies really provide your brand something better than other technologies? Or is your idea just a gimmick? If you’re looking for reach or engagement, then maybe wait a while before reaching for a VR headset. As the VR and AR markets mature, the channels for telling your brand stories will mature with them.

This article originally appeared on Shots on July 5, 2017.

In February 2017, together with World Wildlife Fund, ArtScience Museum and Google Zoo, MediaMonks launched a large-scale mixed reality experience “Into The Wild.”

All backed by the effort to help people in Singapore experience the devastating effects of deforestation and learn more about some of the world’s most endangered species and their habitats.

It was the world’s first Tango-enabled smartphone Lenovo Phab 2 Pro, and guided visitors through personalised digital adventures, which started with augmented reality (AR) on the ground floor of the exhibition space, before transitioning to full virtual reality (VR).

The end of the experience shifts back to AR, where users go up to the fourth floor for an experience that includes planting a virtual tree.

Transforming over 1,000 square meters of the Singapore ArtScience Museum into a virtual, interactive rainforest, making it the largest AR experience in the world, and second ever AR museum experience developed using Google Tango.

And it wasn’t easy. From a technical perspective, we faced the massive challenge of how to accurately and smoothly map a virtual rainforest onto a physical and dynamic museum space, making sure the walls aligned with trees, corridors with the forest’s paths, and that we worked our way around the museum’s existing exhibitions and staging.

With this, we can complete the alignment. And since Tango reports the device’s coordinates back in ecef coordinates, we can easily calculate the corresponding unity coordinates. Effectively we update the Unity camera with every Tango update we receive using this transformation.

What’s more, for every virtual tree planted, a real tree was planted in Rimbang Baling, one of the last pristine rainforests in Sumatra where the endangered Sumatran tiger lives. 5000 new trees were pledged in the project’s first month.

I hope by sharing this we can inspire the imagination of current and aspiring developers to build even more exciting AR/VR experiences that map to the real world. Go forth and conquer!

This article originally appeared on TechWorld on July 18, 2017.

So How Did We Do It?

To start with, if you’re augmenting the real world with virtual objects, it’s important that the device rendering your view (such as a smartphone, monitor, CAVE or head mounted device) is exactly aware of where it is in the real world.

For this, a device needs to know its position and orientation in a three-dimensional space.

In the case of Tango, where the augmentation happens on a camera feed, the position and orientation of the rendering device needs to be in real world coordinates. Only if the position and orientation of a Tango device is reported accurately, and fast enough, proper augmented reality is possible.

The fact that Google Tango does this for you is very cool because it allows developers to augment real world locations within their own virtual world which is different from Snapchat-like AR which, for example, augments bunny ears to your head.

With real world bound augmentations, you can potentially create shared AR experiences that revolve around and involve landmarks.

In this case, it allowed us to transform the ArtScience Museum into a lush virtual rainforest and from the user’s perspective, exploring the rainforest becomes as natural as exploring the museum itself because every corridor or obstacle in the virtual world matches a corridor or obstacle in the real one.

Google Tango Coordinates

We used Unity3D to create our virtual world. To begin, we assured our Unity developers that they wouldn’t have to worry about alignment and were free to design the virtual world using whichever position or orientation they liked, as long as it was true to scale.

Developers familiar with Geographic Information Systems (GIS) know there are a lot of coordinate systems out there called “datums”. Historically, a lot of institutes developed their own, but since the introduction of GPS, the US developed WGS84 which is the most often used for commercial devices.

The great thing about this coordinate system is that it is Cartesian, it calculates in meters, and uses the centre of the earth as its point of origin. This is important because, in a properly mapped environment, Google Tango can give you its exact position and orientation on the globe and gives you these in WGS84.

Google Tango calls these coordinates ecef coordinates, so, we’ll call it ecef also.

Determining the Correct Approach

The next step is to ensure our Unity world overlaps with the real world so we can achieve augmented reality. Two approaches to solve this come to mind.

1. Transform (move+rotate) the Unity world to sit on top of the ecef coordinates of the museum.
2. Transform (move+rotate) the ecef Tango device coordinates into Unity world coordinates.

The approaches are 80 percent the same, as in both cases you have to calculate the transformation from virtual (Unity) to real (ecef). The difference, however, lies in whether you transport the virtual world onto the real one (approach one), or whether you transport the real camera onto the virtual world (approach two).

To determine which approach is best, we had to see what these coordinates look like in a real use case. Here are some examples of how Unity coordinates look:

Object A: [10.000, 63.250, -11.990]

Object B: [-92.231, 33.253, -62.123]

By contrast, below are two examples of how ecef coordinates look:

Hilversum MediaMonks HQ 2nd floor near the elevator: [3899095,5399920414; 353426,87901774078; 5018270.6428830456]

Singapore ArtScienceMuseum in front of cashier shop: [-1527424,0031555446; 6190898,8392925877; 142221,77658961274]

Obviously, the ecef coordinates are quite large numbers. In fact, it’s clear that single-precision floating points (or floats) are going to have a lot of trouble with these.

Without going too much into detail about floats, it’s important to note that performing arithmetic with numbers around 10–6 with numbers around 106 means that you significantly lose accuracy.

In addition, there’s also no way of getting around the fact that a lot of 3D programming is done around 10–3 to 103 (think of transformation, model, view, or projection matrices).

To understand this further, I recommend watching this video as it demonstrates this point perfectly. It shows a fighter jet taking off from around the origin [0; 0; 0] with a camera following it and, as its own position gets larger and larger (as well as the camera’s position), the floating point calculations become less and less accurate.

Imagine then what the error would be if the coordinates of your camera are like the ecef coordinates shown above? You would be combining fine scaled rotation values with very large position values. The error in the result will be enormous.

Add to this the fact that Unity is hard coded to work with floats (rather than double-precision floating points, or doubles), and the fact is that we can’t afford any large errors in AR. It’s therefore clear that approach one is unfeasible. This is because the camera needs to stay relatively close to the origin to avoid precision errors.

So, we proceed with approach two which is to transform the ecef Tango device coordinates into Unity world coordinates.

Find the Transformation

Transformations between coordinate systems in 3D graphics usually entail finding translation (positional), orientation and scaling values.

Each of these three concepts act in 3D space, so they must describe their positioning, rotation, and scaling for each of the three axis (x, y and z). This gives us nine values to find.

The nine unknowns are a hint of how many equations you need to find these nine unknowns. This is important when determining how many real world coordinates are needed to anchor our virtual world to.

Our initial idea was to create a transformation that would deal with all three concepts (translation, rotation, and scale). However, due to difficulties, and the fact we were able to design our virtual world true to scale, we decided to drop accounting for scale and focus on translation and rotation only.

This meant that effectively we now only need to find six unknowns.

Calculate the Transformation Matrix

At this point in solving this challenge, we’re down to finding a transformation matrix that only accounts for location and rotation. Luckily, this problem has been solved a million times already by Computer Science students.

If you simply search Google, you’ll find countless examples of how to transport a rigid body from one coordinate system to another. This is one example that will get you there 90 percent of the way.

Finding a transformation matrix revolves around minimising the sum of squares error between two sets of data points. The following method is tailored for this problem since it deals with rotation and translation separately.

Conceptually, we approach this by picking a point in the real world, and we say that that point corresponds to another point in the virtual world.

Basically, the worlds are anchored to each other on that point. However, as you can imagine, choosing a single point as an anchor still allows the worlds to pivot around the anchor, in which case they will be misaligned most of the time.

Therefore, to place the virtual world squarely on top of the real world perfectly, you need at least a few anchors. Depending on the number of unknowns you’re trying to find, you need an equal or more amount of equations.

Equations can be derived from known pairs (in this case ecef and Unity coordinates). In this case, a total of 3 pairs (or anchors) is enough to allow us to find a full 3D transformation matrix.

The idea is that you choose N amount of points (at least three) in the real world, and find their ecef coordinates. Then, you go into the virtual world and place a point on their corresponding virtual locations (so N in total). For the museum project, we used 10 easy to find landmarks at the base of each pillar inside the museum.

For this, we used a third-party library called Math.Net that allows us to do linear algebra with doubles. You only have to run this code once at the start of the program.

The result is that now have 10 ecef coordinates and 10 Unity space coordinates shaped in a circle, resulting in 10 pairs of coordinates. The next step then is to apply the steps discussed in this article and find a transformation matrix that allows us to transform a point from ecef space to Unity space.

We ran into a few problems while implementing this. For example, Unity is a left-handed coordinate system, while ecef is right-handed. And the article we referenced above also used row major ordering, while the library used column major ordering.

This makes filling, transposing, and multiplication ordering of matrices different. We eventually overcame all of these problems through careful reasoning, and not trying to take too many steps at the same time.

Apply the Transformation

Following the previous step, we have a transformation matrix we could call ecefTunity, (or unityTecef depending on how you calculated the matrix). So, transforming a point in ecef space to unity space becomes as trivial as: