In many places around the world, learning disabilities and impairments often go undiagnosed due to a lack of resources and access to facilities that test for them. But when missed, learning disabilities or color blindness can oftentimes discourage children from participating and excelling in class–sometimes eroding self-confidence in the process, which in turn can lead to far-reaching difficulties later in life.

But technology can help bridge the divide in communities where traditional testing isn’t scalable or accessible to students—so when children’s multivitamin brand Pharmaton Kiddi wanted to impact kids’ lives through technology, MediaMonks’ innovation team sharpened their pencils and put their digital skills to the test. The result is an assessment designed to determine whether students may have one of the learning impairments mentioned above—all while they’re having fun by playing their way through an engaging, interactive digital picture book.

Called Kiddi World, the app takes users aged 9-12 on a journey through the charming World of Words, which is populated with living school supplies and rendered in a charming, painterly aesthetic. Our hero is a sharp, courageous pencil named Lapi in pursuit of Gomba, an irritable eraser whose rampage results in the erasure of parts of the world and words becoming jumbled together. Through a series of visual and language-based activities, kids must erase the prickly eraser’s influence on the world and set things right. The app shows what edtech can achieve through a hands-on approach that brings together data, UX expertise and best-in-class creative storytelling.

Reducing Testing Anxiety with Under-the-Hood Innovation

“Kiddi World fills an empty space where you can pre-diagnose the child,” says Geert Eichhorn, Innovation Director at MediaMonks. He cautions that it isn’t meant to replace a medical professional’s diagnosis of any impairment. “It functions as an indicator based on professional tests,” he says, prompting parents’ or teachers’ attention if a child is likely to have an impairment.

Through AI, the test can measure students’ legibility with precise accuracy—for example, how much a student’s writing extends outside the bounds of a line. “It’s a matter of determining a percentage of error that allows us to apply some simple rules,” says Luis Guajardo, Creative Technologist at MediaMonks, “to tell teachers to look further into it and check in with the child.”

And that’s a good point; while children can play through the test by themselves (guided by text and voiceover instructions that lead them through timed exercises), the app doesn’t signal a reduction in the teacher’s role. Instead, it gives teachers a tool they can use to help them understand how individual students process information differently. They can apply this knowledge to how they educate their students—like making a customized lesson plan or supplementary programs—to aid in these students’ learning.

With so much happening under the hood, children are free to enjoy the assessment without feeling the pressure and anxiety that comes with being tested. Instead, the experience is designed and built to emphasize a child’s unique journey exploring the world with Lapi.

One of the more interesting ways the app achieves this is through handwriting recognition, which aids in identifying dysgraphia automatically as students write. Many touch devices, like Apple’s iPad, natively offer handwriting recognition to translate users’ handwriting into print text. This feature is great for extracting meaning from handwriting, like if a user is jotting down notes. To detect atypical writing behavior from typical ones, MediaMonks took handwriting recognition a step further.

“We used an external tool that could measure stroke speed and density, helping us measure if a line was made fast or slow,” says Eichhorn. These variables are key for making a precise assessment for whether a student might have dysgraphia. “We employed a set of rules that identify aspects of dysgraphia that are particularly notorious and can be examined by the app,” says Guajardo.

In addition to native handwriting recognition, the team also had to disable spell check native to iOS apps. If a student’s spelling mistakes are corrected as they go, it would impact their score—an obvious, but easy to overlook, barrier to accuracy. “Instead, we use an API that determines if what a student has written is a real word, or close to a real word,” says Eichhorn.

Building Immersion Through Design

The tech under the hood is remarkable, but the designs on the surface of the experience are what bring Kiddi World to life. For every action, students write or draw something on the screen, adding to the sense of engagement as they make their mark—literally—on the story’s world. This is why the user’s avatar is a pencil: Lapi reflects the everyday tools that students use in class.

Narrative is ingrained into each of the challenges that kids must overcome in their journey through the World of Words. First, they’re tested on color blindness: Gomba has erased a path through the land, so readers have to draw it back by tracing a line that cuts through a pattern commonly used in tests to determine the type and intensity of color blindness that one might have. Users complete this challenge by drawing a series of segments of the path, which then come together like comic panels to create a full scene.

If a student is color blind and therefore can’t see one or more paths, they can simply skip that segment. This way, students aren’t penalized for something they can’t control, which is critical to how the assessment is designed and progresses. No matter how students respond or interact in any of the exercises, results aren’t telegraphed to them nor do they affect the narrative. Instead, results are saved in a report that’s delivered to the teacher upon each test’s completion.

Eichhorn notes that the narrative is constructed in a way that gives students a real sense of progress. “In the first chapter, students are working with Lapi to respond to what Gomba has already done,” he says. “In the middle, they’re about the same level as him, and in Chapter 3, they’re ahead of him and trying to thwart his plans.”

The flow (and lack of scoring interrupting the narrative) ensures students feel confident and empowered regardless of difficulties they face in the assessments—of which there are a variety. “A lot of testing is required for accuracy,” says Eichhorn. “We found that we needed maybe six times as much information than we originally imagined to come to a reliable and accurate result.”

Still, the team was able to deliver the necessary assortment of tests seamlessly integrated into an engaging narrative—a good test for partners who understand both tech, user experience and visual design. From responding to a direct regional need and offering a familiar and engaging narrative experience, Kiddi World shows that innovation doesn’t have to come at the cost of usability or accessibility. In fact, it can help us facilitate better accessibility to users who need it.

Tools like this can have a profound impact on education—not just by aiding students who might have one of the tested learning disabilities or color blindness, but by initiating more personalized educational plans. By providing measured feedback on how students process information, edtech like Kiddi World enable closer, one-to-one connections between students and teachers, helping them identify the best individualized approach to enhance a student’s learning.

This week, Fast Company launched its annual Innovation Festival, featuring Fast Track sessions that take attendees behind the scenes and into the homes of some of the most innovative companies in New York City. Billed as “Fast Company’s unique take on the field trip,” Fast Track sessions engage brands and creatives through hands-on talks and experiences hosted by participating companies, including MediaMonks.

Our New York office opened the doors to the dojo, inviting brands into our home to discuss all things extended reality. In a panel session devoted to augmented reality and its application to music and entertainment, our Monks dove deep into the design and development process of Pharos AR—a mobile AR experience made in collaboration with Childish Gambino, Wieden+Kennedy, Unity and Google. Taking users on a cosmic journey set against an exclusive track from Childish Gambino, the app is notable for being the first multiplayer music video.

With a panel including Snider-Held (Creative Technologist, MediaMonks), Thomas Prevot (Senior Producer, MediaMonks) and Tony Parisi (Head of VR/AR Brand Solutions, Unity), the session served as a casual fireside chat. The conversation kicked off by establishing the state of VR and AR, often characterized by the conflicting feelings that VR is dead and that the clear use case for AR hasn’t yet been found. But both technologies are well established, each excelling in achieving different goals within different environments.

Showcasing our Batman experience as a strong example of the immersive powers of VR, Snider-Held noted that “These experiences are still very installation-based,” and that AR’s distribution through mobile offers the potential for greater reach with a simpler experience. In explaining the process of developing Pharos AR in particular, the group explored key considerations for challenges to consider when developing an extended reality experience.

AR Can Feel Real Without Being Photoreal

Constraint prompts creativity—an adage that applies just as well to AR as any other medium for art making. Because mobile AR experiences are designed for use across a variety of devices, they must be relatively lightweight to provide a smooth experience to the widest share of users. Failure to keep technical constraints at top of mind can instead result in a lagging, stuttering experience that breaks immersion.

While this is true for any digital experience, it’s especially true for AR, a medium which Parisi says aims to “intelligently interact with the real world.” This expectation to seamlessly integrate with the surrounding environment can make stuttering graphics stick out like a sore thumb. “You want to keep the frame rate above 30 frames per second,” says Snider-Held, “because the user will compare the motion on the screen with the action happening around them in reality.”

The trio took this challenge as an opportunity to discuss the highly stylized look achieved with Pharos AR. While a photorealistic graphics might be impractical for a mobile device to realistically render in real time, a stylized look presents the opportunity to differentiate your experience through a strategic choice in aesthetic; for Pharos AR, the team took visual inspiration from Childish Gambino’s laser-punctuated stage shows, ensuring the app’s look and feel naturally integrated with the rest of the artist’s visual brand.

“Stylistically, we were trying to remain within the constraints of mobile processing in a visually appealing way,” said Snider-Held. An example of this is the use of particle effects, in which sparkles of light coalesce into a ghostly image of Childish Gambino as he dances to the music, animated via motion capture. “This is the best example on why you don’t need to do photorealism,” Parisi said. “We were able to capture the essence of Donald, because it’s his dance.”

Carefully Plan the Narrative Environment

Extended reality experiences are interactive by nature, meaning they rely on a different approach than how you would plot out and plan more linear experiences. There’s a careful balancing act between giving users the reigns to explore on their own versus stringing them along a narrative thread. MediaMonks Founder and COO Wesley ter Haar notes that “Narrative UI is key for onboarding and guiding the user in an AR experience,” making it incredibly important that you plan out users’ interactions and use environmental cues to shape a narrative.

While Pharos AR begins and ends through open-ended user interaction, it still follows a clear narrative through the virtual performance of Childish Gambino’s single, “Alogrhythm.” In exploring the primary path through the experience, the team began planning it in storyboard form, much like you would make for a film. “This process not only serves as visual research, but also in briefing the animation team and envisioning how actions will play out in the user’s environment,” said Snider-Held.

That makes sense—but what sets storyboarding for AR or VR different than other forms of digital storytelling? Because users could play with the app in their homes, the team had to plan for other variables how large the virtual scene should be, and what actions would be possible for multiple users to make within a smaller environment—like the cramped living room of a New York apartment, to offer an example that the Fast Track attendees could relate to. The challenge demonstrates how important it is to map out the virtual scene for different scenarios and users.

An interesting insight uncovered in the panel was that the team didn’t just rely on visual methods of planning like maps and storyboards. Due to the nature of the background music building up as users explore the space around them, the team also developed a musical timeline that maps up how different interactions trigger the layering of the music. The step showcases how sensorial, environmental cues can shape the action within an immersive, extended reality experience.

Whether developing for VR or AR, extended reality experiences require developers to rethink the creative approach beyond the standard linear story. From considerations in setting, technical constraints and variations in number of users across platforms, extended reality development relies on a comprehensive understanding of the building blocks that make up a total user experience. Snider-Held capped off the session with an ambition on what MediaMonks aims to achieve with brands through such experiences: “We strive to further the use of the technology from impossible to probable, and experiment in how to further that, too.”