Interest has grown significantly in voice platforms over the years, and while they have proved life-changing for the visually impaired or those with limited mobility, for many of us the technology’s primary convenience is in saving us the effort of reaching for a phone. Yet we anticipate a future in which voice platforms can provide more natural experiences to users beyond calling up quick bits of information. This ambition has prompted us to look for new ways to provide added value to conversations, making smart use of the tools readily available by organizations leading the charge in consumer-facing voice assistant platforms.

The primary challenge in unlocking truly human-like exchanges with virtual assistants is that their dialogue models are best fit for transactional exchanges: you say something, the assistant responds with a prompt for another response, and so on. But we’ve found that brands that are keen on taking advantage of the platform are looking for a more than a rigid experience. “There are plenty of requests from clients about assistants, who are under the impression that the user can say whatever,” says Sander van der Vegte, Head of MediaMonks Labs. “What you expect from a human assistant is to speak open-ended and get a response, so it’s natural to assume a digital assistant will react similarly.” But this conversation structure goes against the grain for how these platforms typically work, which means we must find new approaches that better accommodate the experiences that brands seek to provide their users.

Giving Digital Assistants the Human Touch

One way to make conversations with voice assistants more human-like is to empower them with a distinctly human trait: emotional intelligence. MediaMonks Labs is experimenting with this by developing a Google Assistant action that serves as a marriage counselor that uses sentiment analysis to draw out the intent and meaning behind user statements.

“If the assistant moves in this direction, you’ll get a far more valuable user experience,” says van der Vegte. One example of how emotional intelligence can better support the user outside of a counseling context would be if the user asks for directions somewhere in a way that indicates they’re stressed. Realizing that a stressed user who’s in a hurry probably doesn’t want to spend time wrangling with route options, the action could make the choice to provide the fastest route.

As assistants become better equipped to listen and respond with emotional intelligence, their capabilities will expand to provide better and more engaging user experiences. In a best-case scenario, an assistant might identify user sentiment and use that knowledge to recommend a relevant service, like prompting a tired-sounding user to take a rest. Such an advancement would allow brands to forge a deeper connection to users by providing the right service at the right place in time. While Westworld-level AI is still far off in the distance, we’ll continue chatting and tinkering away at teaching our own bots the fine art of conversation—and we can’t wait to see what they’ll say next.

To better understand what this looks like, consider how two humans effectively resolve a conflict. Rather than accuse someone of acting a certain way, for example, it’s preferable to use “I messages” about how others’ actions make you feel, so the other party doesn’t feel attacked. So whether you begin a statement with “you” (accusatory) or “I” (garnering empathy) can have a profound impact on how others invested in a conflict will respond. Likewise, our marriage counseling action analyzes the vocabulary and inflection in two users’ statements to dole out relationship advice to them. Responses are focused not just on what they say but how they say it.

“We can learn to speak more effectively to an AI, just like how AI learns to speak to us,” says Joe Mango, Creative Technologist at MediaMonks. According to him, users have been conditioned to speak to bots in, well, robotic ways through their experience with them. “When we had someone from our team test the action by simply speaking to it, he wasn’t sure what to say at first.”

Speaking a New Language

The action takes a large departure from the standard conversational setup with a voice bot. Rather than have a back-and-forth chat with a single user, the action listens attentively as two users speak to one another. Allowing Google Assistant to pull off such a feat gets at the heart of why so few actions provide such rich conversational experiences: the inherent limitations of the natural language processing platforms that power them. For example, the Google Assistant breaks conversation down into a “you say this, I say that”-style structure that limits the amount of time it opens the microphone to listen to a user response.

Conventional wisdom surrounding conversational design shies away from “wide-focus” questions, encouraging developers to be as pointed and specific as possible so users can answer in just a word or two. But we think breaking out of this structure is not only feasible, but capable of providing the next big step in richer, more genuine interactions between people and brands. “It speaks a lot to the mission of what we do at Labs,” said Mango. “We always want to push the limitation of the frameworks out there to provide for new experiences with added value.”

What does such an interaction look like? When a couple tested the marriage counselor action, one user mentioned his relationship with his brothers: some of them were close, but the user felt that he was becoming distant from one of them. In response, the assistant chimed in to remind the user that it was good that he had a series of close relationships to confide in. Its ability to provide a healthy perspective in response to a one-off comment—a comment not even about the user’s romantic relationship, but still relevant to his emotional well-being—was surprising.

Next Stop: More Proactive, Responsive Assistants

While the action is effective, “It’s just the first step down an ongoing path to support more dynamic sentence structures and deeper, richer conversation,” says Mango. While the focus right now is on inflection and vocabulary, future iterations of the action could draw on users’ tone of voice to glean their sentiment even more accurately. From there, findings from this experiment aid in providing other voice apps a level of emotional intelligence that helps organizations engage with their audience in even more human-like ways.

Here, our friends at SoDA (or The Digital Society) sit down with MediaMonks Founder and COO Wesley ter Haar, who is also on the SoDA Board of Directors.

In this year’s Global Digital Outlook Study with Forrester, SoDA expanded their inquiry in the area of innovation labs to uncover some interesting findings.

SoDA: Not only are agencies continuing to launch internal labs and incubators but, more importantly, they are making direct investments into these initiatives. Do you think agencies are finally getting serious about innovation and realizing they need to invest in R&D rather than just hope to do cool work as part of regular client initiatives? How does MediaMonks approach innovation and do you invest in it outside of directly funded client projects?

Wesley ter Haar: I’ve never been a great fan of the Lab moniker. It’s a weird way to silo innovation as a play-thing instead of making it a core part of the day-to-digital-day we all live in. 

…The US is our largest market and the most ambitious, APAC leads the way in user behavior, LATAM has some of the most creative visual talent I’ve seen and I’m always amazed by the creative ideas that bubble up from smaller European markets. It goes to show that constraints are never a reason to deliver mediocre work.

This article was originally published in The SoDA Report – with key findings from Forrester.

Wesley ter Haar: That X used to be 24 to 36 months, and has probably been whittled down to 9 months with the constant change that besets consumer behavior and client adoption. At MediaMonks we hire or acquire against an internal innovation roadmap based on where we see the confluence of people, products and platforms are headed. For us, that has meant the acquisitions of a VR-first production company and a connected commerce company, the launch of a digital first content company and a hiring spree to bolster our AR capabilities. So, yes, innovation is critical to the health of our business, but I don’t believe a ‘Lab’ is the way to make it central to who we are and what we do.

SoDA: MediaMonks is hired by client-side marketers (and agencies) to deliver cutting edge work. This year we found that Chatbots/Conversational Interfaces, AI/Machine Learning and Programmatic Advertising topped the rankings for anticipated impact and planned investment in emerging technology. Agency leaders and marketers were generally aligned on this front with one major exception… Virtual and Augmented Reality. Marketers are planning to make significant investments in VR/AR while agency leaders are lukewarm on the short-term marketing impact. Why do you think there’s such a big gap between marketers and agencies on this front?

Wesley ter Haar: I think this gap mostly represents the excitement for the future state of “The R’s” (augmented reality, virtual reality and mixed reality) relative to the technical maturity of current platforms and production processes.

From the SoDA perspective, this shows a mature agency landscape with many agency leaders trying to think from a client perspective, focus on value/budgets and look at reasonable metrics. As agency leaders, we have to make sure we educate clients on the now while planning for the future so we don’t miss the boat when the scale and spread of these technologies starts impacting brands, business and bottom lines.

SoDA: For many years, SoDA has tracked what marketers value most in their agency relationships and, on the flip side, what agencies think their clients value most in their partnership with them. “Expertise in Emerging Tech/Trends” and “Process/Project Management” are consistently rated in the Top 5 by both agency leaders and client-side marketers. How does MediaMonks balance the importance of project management rigor with the desire for clients to explore (and quickly deliver on) the latest technologies? Is there a healthy tension between these two factors?

Wesley ter Haar: There will always be tension between doing difficult things for the first time and delivering difficult things for the first time, on time. Our role is to explain risk, mitigate against it as best we can, and make the “fall down and get back up” process of research, innovation and iteration one that is transparent to clients.

…But, the level of comfort on the client side is going to rely on the quality of the process and the project management rigor around it. In the same way that ad agencies are not artists, digital agencies shouldn’t hide behind labs and a “Crazy Scientist” vibe when it comes to new technology and trends. It’s all about the practical application for clients and their (potential) customers.

SoDA: This year we asked agency leaders to identify strategic factors they saw as most critical to their ongoing growth and evolution. Not surprisingly, “Attracting and retaining top talent” and “Developing new services / capabilities” topped the list. Interestingly, very few looked at “Expanding to new markets/geographies” as an important part of their strategic plans. MediaMonks appears to be quite the opposite with offices now in Amsterdam, London, Stockholm, New York, LA, Dubai, Sao Paulo, Buenos Aires and Singapore. How do you approach geographic expansion and why has it been so central to your growth strategy? What challenges have you wrestled with in managing the business across such a broad geographic footprint? What do you see as the most exciting new markets?

Wesley ter Haar: To start with the reasons, it gives us the opportunity to recruit and retain talent at a much larger scale, and in turn helps us cater to the ambition many of our Monks harbor when it comes to working in other countries and cultures. 

We run all offices as a single P&L which sounds like an admin choice, but is a critical cultural component. We are one company operating across 9 countries and 10 offices, with teams and talent working across time zones. Budgeting, resourcing and planning needs to be seamless to make that work, and that’s been the operational focus from Day 1.