Earlier this year, our research and development team MediaMonks Labs partnered with FLUX, our fashion and luxury team, to explore the virtualization of fashion. The report explored digital’s impact on fashion design, production and consumer experiences—one of those being the  opportunity to try on digital garments using AR.

 Now, the Labs team has completed a prototype that allows people to do just that. Based on full-body tracking, the prototype features an original digital garment designed by Brandi LaCertosa, a Creative at MediaMonks with a background in fashion design. But the immersive experience does more than let people virtually “wear” a garment; it also offers a glimpse into the ways that digital technology can help consumers engage with the meaning and inspiration behind a design through storytelling and interactivity.

Trying on a New Technology

 If you’ve ever played around with a face filter on Instagram or Snapchat, then you already have an inkling of how the try-on prototype works—the main difference is that the prototype tracks the whole body, rather than simply tracking the face. So just like how a face filter might let you try on cosmetics, full-body tracking lets you view an entire virtual outfit on your own body.

When the team first began experimenting with the prototype, only 2D tracking was available by Snapchat. Since then, the platform has released 3D body tracking, which recognizes the position and rotation of joints for a more convincing experience.

There are some limits to the technology. The camera’s view must frame the entire body, which is good to get an overall look at an outfit, but can make it tricky to capture the finer details (like buttons on a blouse) that require bringing the device in closer range to see.  The technology also doesn’t allow for sizing adjustments, meaning a single virtual garment won’t fit all body types. Grading (fashion-speak for making larger or smaller sizes) a virtual garment follows the same process as a physical one.

The team anticipates demand for 3D content will continue to grow in the fashion industry. Realizing this, they built their approach around the way fashion houses produce physical collections. “We used the industry standard tools for design, modeling and output to see how these tools work together, and what we need to learn for future projects using this production pipeline,” says Geert Eichhorn, Innovation Director.

Designing in a New Dimension

The process of designing an outfit and translating it into digital was a unique collaboration between cutting-edge technology and traditional fashion design. “I wanted this garment to be a heritage piece, something that I would design regardless of it being digital,” says LaCertosa, noting that a fashion brand wouldn’t take the digital aspect into consideration when designing—instead, executing digitally would be our challenge to solve. The team worked with a Marvelous Designer, which allowed the team to work with the same kind of digital patterns that brands are already using in their current design process.

The digital production process emulated the way that a garment comes together physically, with the Labs team translating LaCertosa’s designs into patterns that would join together in a 3D shape. “We weren’t physically together, so I couldn’t make patterns for them,” says LaCertosa. She provided the team with references—“We use very specific terminology in fashion,” she adds—and connected with them over calls to suggest any changes that needed to be made. “It’s the same process you’d have by physically meeting in a factory,” says LaCertosa. “We followed that same production flow, but virtually—and it was quite smooth.”

This process gave the team the chance to test the new value chain mapped out in their previous report—a fashion cycle transformed by new technologies. “It speaks to how we do things. We have so much expertise across our teams, and it’s about trusting each other and knowing what someone doesn’t have and needs,” says Eichhorn. “There’s an understanding that all these things feed into one another, from design to production to the end-consumer, so it’s natural for the Monks to work together in an integrated way.”

Translating Inspiration into Tactile Experiences

The technology prototyped by Labs is more than just a tool for trying on clothes digitally—it also opens up sophisticated forms of storytelling. The inspiration behind LaCertosa’s design hearkens back to the Greek island of Chios, her family’s homeland. Among the island’s most famous stories is that of the Ottoman invasion of Anavatos, a fortress-like village high up in the mountains. As the Ottomans stormed the village, the women made a drastic sacrifice to avoid falling into a life of slavery: they jumped from the cliffs to their deaths and were regarded as heroes by the locals.

Elements of this story come to life in the details of LaCertosa’s design. Its silhouette takes inspiration from traditional Greek garb worn in the War of Independence, while ruffles climb up the shoulder to evoke ascension (“Anavatos” translates to “ascendable” or “climbable”). Worry beads made from resin produced by the mastic tree, which is most prevalent on the island, become shank buttons adorning the garment.

While it’s not unique for a piece of clothing to tell a tale, such stories are seldom shared with consumers. “Designers love telling their stories and sharing how they pull inspiration,” says LaCertosa. “Now, you have more opportunities to let people get to know the garment and learn more. Even small details incorporated through sound and animation could have a great effect.”

On that note, Guajardo continues to experiment with the prototype by adding different visual effects that make the try-on experience more unique, immersive and emotionally engaging. “I’m fascinated that you can mix different techniques that we use with lenses,” he says. “I’m using particles and segmentation to test different atmospheric effects.”

So, while some may use full-body tracking and 3D technology to show how a garment looks on their body, others might wield virtualization to tell feature-rich, emotionally driven narratives. From production to the consumer experience, digital’s unique ability to convey the inspiration behind a design and immerse people within the world of the brand continues to grow—and the team is keen to see where that takes the industry next.

There’s more where that came from.

At its Snap Partner Summit just weeks ago, Snapchat announced its Snap Games platform. A bit like Facebook’s Instant Games for Messenger, Snap Games are simple, HTML5-based experiences that users can instantly activate via chat. What makes the platform unique, though, is that its games seamlessly integrate into the chat platform, allowing users to continue talking to friends via text or audio as they play. The move signals the changing way that brands and users are approaching games: not just as a diversion, but as a means of connecting digitally with one another through shared experience.

“Friendship is more than just the things you chat about,” Head of Snap Games Will Wu explained to an audience at the summit. “Friendship is also about experiences you have together.” He went on to highlight the need for offering an environment for friends to get together and do more than just socialize—an idea that’s taken wind as people have come to view the popular game Fortnite as more than just a game but rather a hot, new social environment.

Today’s Gamer Isn’t Who You May Think They Are

The still-new platform isn’t yet open for everyone to develop, but brands and developers that are inspired by the sense of play can still provide gaming experiences in the platform. Take Snappables, for example, which are Snap-produced selfie lenses infused with game mechanics to make them more fun. While the Snappables moniker is reserved for lenses developed by Snap themselves, anyone can likewise make experiences of their own that are game-like.

First, why support game-based AR lenses? The gaming population has changed surprisingly with the growing popularity of mobile gaming. While many assume the typical gamer is a boy under the age of 18, running the numbers challenges that notion: according to a report by the Entertainment Software Association, 45% of US gamers are women, and the average age for female gamers is 36. The average age of gaming males in 32.

Second, the social mobile environment is an easy fit for supporting modern gaming behaviors. According to the same ESA report, 42% of gamers play together with friends, while 36% play on a smartphone—equal to the percentage of gamers who play on dedicated game consoles.

Consumers Seek Value-Added Interactions

But for MediaMonks Senior Project Manager Tommy Lacoste, infusing an AR lens with game mechanics often just makes sense for providing a compelling interactive element without making the experience too complex. “People typically want to emulate flashy, cool AR experiences that they’ve seen elsewhere, but the experience must demonstrate brand value or fit the platform,” says Lacoste.

For example, our #LongestGoal Snapchat lens (made in collaboration with 180 LA for UNICEF to celebrate the World Cup) challenged users to scream “Goal!” for as long as possible. The longer they yelled, the higher their score—turning a key spectator behavior into an opportunity for friendly competition. The interaction perfectly fit the brand goal—in this case, highlighting a World Cup campaign.

As a Snapchat partner, MediaMonks helps brands zero in on ideas for captivating interactive experiences by offering several options to build on a brand’s idea. “We try to enhance ideas by taking them to the next level,” says Lacoste. But this can be tricky, as AR lenses for Snapchat and Facebook Camera alike are constrained to specific size limits. To squeeze the most value and functionality out of your experience, working with a creative and production partner who knows how to balance platform opportunities and limitations is key.

Designing an AR experience is easier than you think.

Selfie Lenses Go Beyond Face Masks

Gamified lenses that emulate Snap’s own Snappables series of games require quite a bit of ingenuity and creativity. “You don’t usually see much of this sort of thing with AR lenses, because it’s not necessarily the way they were originally intended,” says Lacoste. But adding in game mechanics doesn’t make a Snap lens heavier; instead, it requires technical understanding of what’s possible given advances in computer vision.

“With the Snapchat AR Developer Platform, we can detect where your head or parts of the body are,” says Lacoste. “From there, we can detect and measure interactions with virtual objects. So if we wanted the user to bounce a virtual ball on their head or nose, we can count that and keep score.” With these game-like interactions, selfie lenses go beyond being simple facial masks and encourages interactivity—and a reason to keep playing to get a high score.

Our recent Snapchat lens for ice-cream brand Magnum does this by challenging users to try to eat every Magnum bar floating around them before their screen fills with ice-cream, a bit like bobbing for apples. “Testing is crucial with these experiences,” says Lacoste. “First, you need to prototype and iterate the technical side of things. Second, you also must ensure the experience is fun…or funny for the user.”

That’s a good point: chomping at virtual ice-cream bars that dangle around you may be fun, but it’s also fun for others to watch. While a user may feel driven to share their high score, playing with the lens results in user-generated content that’s sure to get a laugh from friends. And that brings us back to our starting point: today’s consumer seeks out experiences they can enjoy among friends. Gamified AR selfie lenses offer an excellent opportunity for brands to provide simplified yet engaging moments to deliver on this demand.

Dive deep into the possibilities of mixed reality.