Meena: Google's Attempt At Making A True Conversational AI
Sun, April 11, 2021

Meena: Google's Attempt At Making A True Conversational AI

AI assistants like Alexa, Siri, and Google Assistant are not exactly conversational / Photo Credit: TZIDO SUN (via Shutterstock)


Khari Johnson of Venture Beat, a news channel on transformative tech, reported that conversational AI refers to natural language models for AI to enable them to understand human words, speak to people, or fulfill tasks or computation with natural language. However, today’s AI assistants like Alexa, Siri, and Google Assistant are not exactly conversational. They can answer factual questions or crack jokes but having a conversation is still a human endeavor.

Google’s Meena, a neural network with 2.6 billion parameters, can handle multiturn dialogue. Google claimed it’s better than other AI agents built for conversation. Meena has even cracked an off-the-cuff joke. Google also debuted Sensibleness and Specificity Average (SSA), a metric developed by Google researchers to gauge the ability of a conversational agent to maintain responses in conversation that is sensible and specific. In SSA, humans scored around 86% while Meena scored 79%. Pandora Bots’ AI agent Mitsuku had a score of 56% while Microsoft’s Mandarin-speaking Xiaolce scored 31%.

Meena is trained on 40 billion words, utilizing a seq2seq model and a variation of the popular Transformer architecture, which was first launched in 2017. SSA evaluates dialogue “based on static performance with a fixed set of prompts or interactive performance” to enable a free-flowing conversation. Each one was required to last at least 14 turns and no more than 28 turns. Results are arrived based on the percentage of turns considered sensible and specific. Generic responses are penalized by SSA. The said standard that Google proposes is different from other metrics used by AI assistants in assessing a truly conversational AI.

Now in its third year, the Alexa Prize is a challenge for student developer teams to build an AI capable of holding a conversation for up to 20 minutes. The latest round of finalists will be announced this May. Overall, AI assistants that can hold a conversation may forge closer bonds with humans, providing them with emotional support or cure their loneliness, noted former Alexa Prize head and present Google Research director Ashwin Ram in 2017.