Interaction Design


Nowadays, the conversational user interface (CUI) is prevalent and powerful. Hearing-impaired people do not have access to this technology. However, Voca serves as a temporary App to provide accessibility for those people.


Product Designer, Prototyper


2017 Sep. - 2017 Dec.


Led the design and prototyping of accessibility of voice user interface from start to finish, including the early research, user flow, interaction design and prototyping.

Translation between voice message & text bridges the accessibility


Hearing-impaired users can type in the command first, then Voca would translate the text into voice message & speak towards the device. Then the device would compute & respond with a voice message. Voca would pick it up & translate it back to text for the users to see.


Painpoints of current virtual assistants


I researched how deaf people could use Alexa, Google Assistant, Cortana and Siri to communicate with the virtual assistants. Here are some insights:

Image Image

Design of error recovery leads to good experience

Current Flow

We used Echo as an example of a voice-controlled device, and experimented different kinds of errors. Then we concluded with the following flow:

Image Image

Designing for errors

  • A good communication model to balance good experience & 3 different agents

  • A practical message translating model which provides feedback & error recovery

  • A clean visual hierarchy to unify & stratify the same component with different information

  • An efficient volume increasing model to help the user replay the command if confronting an error

  • An engaging error recovery flow to guide & educate users how to recover from errors

A good communication model design balances good experience & 3 different agents

Design Thinking 1
Click here to see the design thinking Image

From the test result, we finalized with our 4th design: Voca acts as neutral textbox, VA (Virtual Assistant) acts as respondent. We chose it because it can deliver a clear communication model to users, and it can also increase exposure of Voca for branding.

Visual affordance & immediate feedback while waiting for response

Design Thinking 2

After observing the communication process of Echo & recorded message, we refined the interaction process of Echo. Then we iterated on our design by checking the timing of light feedbacks & responses.

Usability of smart-recommended commands & visual hierarchy

Design Thinking 3

After researching the current design solutions & the average number of smart-recommended commands, we realized that there were several problems with design 1.0, such as the same visual design for different information hierarchies, and that the vertical design limits the number of smart-recommended commands.

Usability of replaying the command with increasing volume

Design Thinking 4
Click here to see the design thinking Image

After testing 2 different replay volume increment solutions, it was found out that it took 3 times to let the device hear the voice message in incremental volume design. However, it only took 1 time for the device to hear in maximal volume design. So we finalized the maximal volume design.

Educating & engaging users in error recovery tips

Design Thinking 5

To engage & educate users, we should concise the tips in 1 page & ensure users follow the instructions. So we used the checkbox design & consolidated the volume increasing & distance shortening tips in one page.

The position of the function "change virtual assistant"

Design Thinking 6
Click here to see the design thinking Image

In our research, users can switch the virtual assistant in the same device(Cortana in Amazon Echo). Also, it is possible that users would talk to different virtual assistants in different scenarios, so it is necessary to implant this function. In the testing, I put the changing va function into 3 different positions: hamburger menu, VA's legend & virtual assistant icon. The result showed that most testers chose the hamburger menu.

Voca is focused on onboarding, communication model & error recovery


In general, Voca is emphasized on onboarding: getting users aware of the system and user-friendly features, communication model: elevating the brand value and users' understanding of the mechanism, and error recovery: helping users recover from the error-prone scenarios.

A response with recommended commands

Invision & After Effects Prototype

When the VA sends back the voice message, Voca would translate the voice message into text with recommended commands for further interacting with the VA.

A tip button while Voca is waiting for the response

While waiting for the VA's response, user can click the tip button to check if the device is listening or replay the command.

A tutorial of error recovery tips for the non-responsive error

The tutorial tips would guide users to check the volume is at maximum & the distance is close enough for the replay of the command to be effective.

Favorited commands offer good interaction experience

The user can open the command panel & click the favorited command. Voca would speak the favorited command to the device.

Smart-recommended commands offer quick command typing experience

When the user is typing in a command, Voca would smart-recommend related commands for users to choose.

"Default virtual assistant" setting offers good flexible communication experience

When users need to interact with different virtual assistants (devices), they can change the default virtual assistant and trigger word.

Translating mutually from test message to voice message


We developed the workable prototype to test the core function of Voca: translating texts into voice messages, and translating voice messages back to texts. Here is the Github link.

We tested the workable prototype by typing in different category of commands, and recording all the texts & translated texts from voice messages. Then we compared the texts with translated texts from voice messages to calculate the accuracy. The following is the chart of the accuracy of Voca's voice message translation:


What I learnt in a hard way

Reflection & Next step
  • For design decision making, the majority is not always right. The testing result, however, is.

  • For the fundamental model or design, it would be always greater to test it in an earlier stage than in a later stage. The adjusting cost is larger if it is later.

  • Design thinking can be cooperated, the design implementation (wireframe, ui, prototype), however, will compromise if delegating to different designers.

  • Better said than done. For communication, a hasty sketch is sometimes better than a beautiful speech.

The future implementation & goal

  • Deploy the communication model

  • Deploy the message transferring model

  • Increase the accuracy of voice message transferring

  • Deploy the error recovery modules

  • Deploy the system setting module

  • Deploy the command favorite & recommendation modules

Wireframe, UI & Design Specification