This week, Fast Company launched its annual Innovation Festival, featuring Fast Track sessions that take attendees behind the scenes and into the homes of some of the most innovative companies in New York City. Billed as “Fast Company’s unique take on the field trip,” Fast Track sessions engage brands and creatives through hands-on talks and experiences hosted by participating companies, including MediaMonks.

Our New York office opened the doors to the dojo, inviting brands into our home to discuss all things extended reality. In a panel session devoted to augmented reality and its application to music and entertainment, our Monks dove deep into the design and development process of Pharos AR—a mobile AR experience made in collaboration with Childish Gambino, Wieden+Kennedy, Unity and Google. Taking users on a cosmic journey set against an exclusive track from Childish Gambino, the app is notable for being the first multiplayer music video.

With a panel including Snider-Held (Creative Technologist, MediaMonks), Thomas Prevot (Senior Producer, MediaMonks) and Tony Parisi (Head of VR/AR Brand Solutions, Unity), the session served as a casual fireside chat. The conversation kicked off by establishing the state of VR and AR, often characterized by the conflicting feelings that VR is dead and that the clear use case for AR hasn’t yet been found. But both technologies are well established, each excelling in achieving different goals within different environments.

Showcasing our Batman experience as a strong example of the immersive powers of VR, Snider-Held noted that “These experiences are still very installation-based,” and that AR’s distribution through mobile offers the potential for greater reach with a simpler experience. In explaining the process of developing Pharos AR in particular, the group explored key considerations for challenges to consider when developing an extended reality experience.

AR Can Feel Real Without Being Photoreal

Constraint prompts creativity—an adage that applies just as well to AR as any other medium for art making. Because mobile AR experiences are designed for use across a variety of devices, they must be relatively lightweight to provide a smooth experience to the widest share of users. Failure to keep technical constraints at top of mind can instead result in a lagging, stuttering experience that breaks immersion.

While this is true for any digital experience, it’s especially true for AR, a medium which Parisi says aims to “intelligently interact with the real world.” This expectation to seamlessly integrate with the surrounding environment can make stuttering graphics stick out like a sore thumb. “You want to keep the frame rate above 30 frames per second,” says Snider-Held, “because the user will compare the motion on the screen with the action happening around them in reality.”

The trio took this challenge as an opportunity to discuss the highly stylized look achieved with Pharos AR. While a photorealistic graphics might be impractical for a mobile device to realistically render in real time, a stylized look presents the opportunity to differentiate your experience through a strategic choice in aesthetic; for Pharos AR, the team took visual inspiration from Childish Gambino’s laser-punctuated stage shows, ensuring the app’s look and feel naturally integrated with the rest of the artist’s visual brand.

“Stylistically, we were trying to remain within the constraints of mobile processing in a visually appealing way,” said Snider-Held. An example of this is the use of particle effects, in which sparkles of light coalesce into a ghostly image of Childish Gambino as he dances to the music, animated via motion capture. “This is the best example on why you don’t need to do photorealism,” Parisi said. “We were able to capture the essence of Donald, because it’s his dance.”

Carefully Plan the Narrative Environment

Extended reality experiences are interactive by nature, meaning they rely on a different approach than how you would plot out and plan more linear experiences. There’s a careful balancing act between giving users the reigns to explore on their own versus stringing them along a narrative thread. MediaMonks Founder and COO Wesley ter Haar notes that “Narrative UI is key for onboarding and guiding the user in an AR experience,” making it incredibly important that you plan out users’ interactions and use environmental cues to shape a narrative.

While Pharos AR begins and ends through open-ended user interaction, it still follows a clear narrative through the virtual performance of Childish Gambino’s single, “Alogrhythm.” In exploring the primary path through the experience, the team began planning it in storyboard form, much like you would make for a film. “This process not only serves as visual research, but also in briefing the animation team and envisioning how actions will play out in the user’s environment,” said Snider-Held.

That makes sense—but what sets storyboarding for AR or VR different than other forms of digital storytelling? Because users could play with the app in their homes, the team had to plan for other variables how large the virtual scene should be, and what actions would be possible for multiple users to make within a smaller environment—like the cramped living room of a New York apartment, to offer an example that the Fast Track attendees could relate to. The challenge demonstrates how important it is to map out the virtual scene for different scenarios and users.

An interesting insight uncovered in the panel was that the team didn’t just rely on visual methods of planning like maps and storyboards. Due to the nature of the background music building up as users explore the space around them, the team also developed a musical timeline that maps up how different interactions trigger the layering of the music. The step showcases how sensorial, environmental cues can shape the action within an immersive, extended reality experience.

Whether developing for VR or AR, extended reality experiences require developers to rethink the creative approach beyond the standard linear story. From considerations in setting, technical constraints and variations in number of users across platforms, extended reality development relies on a comprehensive understanding of the building blocks that make up a total user experience. Snider-Held capped off the session with an ambition on what MediaMonks aims to achieve with brands through such experiences: “We strive to further the use of the technology from impossible to probable, and experiment in how to further that, too.”

Earlier this summer, we released  Pharos AR, a mobile app that takes Childish Gambino fans on a virtual, hallucinogenic journey through the cosmos starting from wherever they stand in the real world. Fans and press alike were impressed by the artist’s bold foray into virtual space.

VentureBeat was excited by the long-term value of the app, for example: “Between the cool visual effects and Glover’s music—which the app will apparently update with new songs over time—there’s certainly enough here to merit a download for fans.” Variety, meanwhile, applauded how well it integrated within the larger Childish Gambino universe: “The whole thing is very spacey, and stays true to Childish Gamibo’s other ‘Pharos’ projects” that have expanded across performances.”

As we see interest build in augmented and mixed reality over the interim, we’re looking back on the project’s development and how multiple parties and team members came together to employ best practices in UI and design, helping everyday users ease into a mysterious–and perhaps technologically overwhelming–new interface without limiting space for play and exploration.

Made in collaboration with Google Zoo, Unity and MediaMonks, it’s the world’s first shared augmented musical experience, allowing multiple users to enjoy a unique, artistic experience together. “This app is a breakthrough for AR,” says Thomas Prevot, Sr. Producer at MediaMonks. “It serves as another outlet for Childish Gambino’s creativity, letting him update his fans with future song releases over the cloud,” says Prevot, explaining how the app fits within the larger Childish Gambino brand.

Pharos AR also showcases the power of the ARCore platform and how it can enable immersive, social storytelling experiences. In particular, it shows off the capabilities of Cloud Anchors, which lets multiple users interact in a shared virtual space–which also makes it fairly unique among AR experiences. The tech’s newness can be intimidating, though: how can apps cultivate an interactive, exploratory experience for those new to AR?

Integrate Brand Familiarity & Digital Ecosystems

Childish Gambino fans will recognize Pharos AR’s light-particle silhouette from a Pixel 3 TV ad, and scenes projected within an enigmatic pair of monoliths at this year’s Coachella festival. These elements are purposefully and artfully executed across touchpoints, and this integration with a pre-existing digital ecosystem helps to make the process of designing for emerging tech more intuitive.

From the Pharos Festival to a TV spot to Coachella and beyond, you’ll find the same trademark, psychedelic motifs true to the nature of the artist. This goes to show the potential of AR as a powerful channel to not only engage audiences in an immersive way, but to bridge together the Childish Gambino gospel as well.

Get Intuitive

With AR still being fairly new to some, developers must think carefully about onboarding new users. “Experiences like Pharos AR provide an exciting opportunity for us to help make AR and VR more accessible to wider audiences,” explains Justina Sung, UX Designer at MediaMonks. “If people habitually use their phones a certain way, how do we break out of that to teach them new behaviors for emerging tech like AR?”

Inspired by Childish Gambino’s spirit of ambiguity and minimalism, the app integrates a sensory communication method, instead of verbal directions, to onboard. “Most app tutorials task the user with swiping through instructional cards, but we made it feel like the title sequence of a movie,” says Alex Otto, Associate Creative Director at MediaMonks. When users first open the app, for example, a pink laser beam dynamically snakes through space and into icons that illustrate next steps. “It’s designed to give off a mysterious feeling that builds up layers of suspense,” says Otto.

Direct Users Through Environmental Storytelling

Because AR rewards users through exploration, designing for it requires a careful balance between providing freedom versus direction. “It’s important that the design gives fans room to explore on their own to find meaning, using subtle environmental cues that nudge users in the right direction,” says Sung.

In Pharos AR, the scene gradually dims as users discover hidden glyphs on the cave walls. This nudges them to shift their focus to the altar, which lights up as users discover more paintings—a bit like a progress bar. Other cues include haptic feedback and the gradual, sonic buildup to Childish Gambino’s song “Algorythm,” which released through the app.

AR is a powerful platform for brands to tell their stories and engage directly with key audiences. In the case of Pharos AR, the technology provides users a chance to embark on a wild, immersive journey that encapsulates Childish Gambino’s message of enlightenment. As the first interactive, shared musical experience in AR, the experience truly pushes the limits on what can be achieved—inspiring brands and artists alike to consider what’s possible.

Eager to experiment with easily approachable AR?