Overview
Five participants were tested on the ‘Airbnb Conversate’ prototype, using a set a of metrics developed through the Natural Conversation Framework (NCF). A brief description of the metrics will first be highlighted followed by a discussion of the most important insights from the test.
measuring a conversatioN
Measuring the effectiveness and efficiency of a conversation between a user and virtual agent is a very new field. As a result, It was difficult to identify a set of standardised practices to test the prototype. A set of metrics were taken from the NCF, which was discussed in detail in the ‘Research’ section of this project. The following metrics were tested on the final prototype.
Number of Sequences: This is a measure of how many base sequences or adjacency pairs were initiated throughout a conversation. This is used to test the level of engagement of a conversation, but also as a foundational metric that serve the other two metrics.
Sequence Completion Rate: This is the percentage of the base initiated sequences that were completed by the agent or user. This is essentially the rate of successful sequences throughout the whole conversation. This figure can highlight the level of mutual understanding both parties had throughout the conversation.
Interactional Efficiency: This is a measure of how much extra work both parties needed to do in the conversation to understand the conversation and progress. It is calculated by counting all the expansions in order for either the agent or participant to clearly understand what was being asked or said.
Additionally, a SUS and product reaction card was completed by the participants, to understand the overall user satisfaction of the prototype.
Testing Method
The prototype was tested on five participants. All participants were asked to complete one task within one scenario using ‘Airbnb Conversate’ . The task was as follows:
‘You are going on a trip to London with your friend from Tuesday the 14th of May until Monday the 20th of May. Have a conversation with ‘Airbnb Conversate’ to discover what cooking classes are available near Kings Cross on Saturday the 18th of May. If there is a cooking class that is particularly interesting to you, you can try and book this experience for you and your friend.’
The task of booking a cooking class was decided upon based on user interviews conducted in a prior stage of the project. When interviewees were asked what they enjoyed doing most while on vacations, ‘trying the local food’ was the most common answer. As a result, finding a local cooking experience was a natural task for the scenario at hand.
Test Results
There were two significant insights gathered from the test. First being that all participants purchased via ‘Airbnb Conversate’, and secondly, user satisfaction scored very high with Airbnb ‘Conversate’.
Purchasing with Airbnb Conversate
100% of participants made a purchase via voice. This was a very exciting and positive step forward for this prototype. This outcome highlights two significant points. The first, that voice interaction can provide sufficient information to make purchases and secondly, the hypothesis made in the research section, where users are more likely to make voices purchases from brands they have made previous purchases with, is proven.
Airbnb Conversate provided the right information that allowed users to gain enough context to make a booking. This type of information included universal needs such as knowing what time the class started, the cost, and availability. The prototype was also able to provide participants information that was unique to their needs. This included sending images of the cooking class to the participant, relaying reviews of the class and approximating distance from their accommodation to the class. A voice application that has a diverse set of information accessible, to cover a user’s potential unique inquiries, is imperative to creating a positive voice experience.
4/5 participants opted to use bank details that were saved to an Airbnb account when asked about payment details. This level of gained trust highlights an excellent advantage for companies such as Airbnb, who do not need to win trust via voice, but rather transport their already established trust developed on the website or mobile device..
High User Satisfaction
Airbnb Conversate scored 85 in the SUS test. The most common words used to describe the prototype in the product reaction card test were ‘easy to use’, ‘effortless’, and ‘efficient’. These results show strong potential for the adoption of a complimentary product like ‘Airbnb Conversate’, especially for the scenario that the prototype was tested under, where a user has a holiday already booked and wants to learn about things to do.
Word Cloud generated from users product reaction card results.
Test Conclusions
The test results show strong positive evidence to pursue ‘Airbnb Conversates’ development as a complementary voice application to the already existing ‘Experiences’ product. The combination of all participants purchasing a cooking class via voice, the ability for the prototype to answer any unique user questions, and the very positive user satisfaction results highlight the potential to further develop this prototype.