FINDINGS FROM THE USER SURVEY TO UNDERSTAND THE EFFICACY OF VIRTUAL INTERPRETER FOR INDIAN SIGN LANGUAGE
Executive summary
Friends For inclusion and International institute of Information Technology – Bangalore (IIIT-B) has been jointly developing a virtual sign language interpreter that can convert speech and textual inputs into Indian Sign Language (ISL).
An online user survey was conducted to study the efficacy of the virtual sign language interpreter that converts speech and textual inputs into ISL. The survey was on field for a month, from 20-November-2020 to 20-December-2020.
The survey revealed that the solution scored extremely high on three out of the four quantitative dimensions, including the dimension of Understandability (96.67% of words/ sentences were understood well by the user groups), Clarity (100% of users found the ISL words clear), and on familiarity (90% of words were correctly guessed by the users). However, the insights revealed that the solution needs to work on ISL grammar sentences, with users providing detail feedback on the grammar. Further, on reviewing the qualitative responses, the team has identified several focus areas to work as they continue to develop the solution in near future.
The survey provided encouraging results and indicated that users can understand the output of the virtual interpreter. However, the study also unearths a few areas like improving the ISL translation model etc. to focus on before the final product could be rolled out to the users.
Introduction
Friends For inclusion and International institute of Information Technology – Bangalore (IIIT-B) has been jointly developing a virtual sign language interpreter that can convert speech and textual inputs into Indian Sign Language (ISL). The team has created an advance proof-of-concept (POC), which was presented to deaf users and ISL professionals in a focus group discussion in June 2020, which was followed by a dipstick survey in August-September 2020. The team receive significant and important feedback from the users. Several of the suggestions were incorporated into our next version of the prototype.
An online user survey was conducted to study the efficacy of the virtual sign language interpreter that converts speech and textual inputs into ISL. The survey was on field for a month, from 20-November-2020 to 20-December-2020. The following report discusses the findings of the survey.
Survey design
Having addressed several of the issues and feedback that we received from the community, the team decided to conduct a user survey to study the understandability of the solution built so far in November 2020. Due to constrains imposed by Covid-19, we decided to use the Google survey, as we did for the dipstick survey.
Question categories
The form has a set of 30 questions comprising of words and sentences, which were divided in three categories., namely:
1) Category 1 : which comprised a set of 10 identified words that is signed by the virtual interpreter. Words in this set were unmasked that is the users were informed what is the given word and were asked if they were able to understand the given word and if they had any observation for the given word.
2) Category 2 : which comprised of a set of 10 blind words. In this set, users were asked to guess the given ISL word and indicate if it was easy to understand.
3) Category 3: which comprised of set of 10 short phrases/ sentences. The users were told what the given phrase/ sentence is and were requested to indicate if they were able to understand and if the ISL grammar was correct.
Field work
In this round of user survey, the team targeted to get feedback from at least 20 qualified users. This included deaf ISL users and ISL professionals. The survey was on field for a month, from 20-November-2020 to 20-December-2020. As on 21 December 2020, we received a total of 31 responses, out of which 26 were qualified respondents and 5 were from those who did not understand ISL, which were subsequently removed from the final analyses.
Objective and dimensions
Through this survey, we wanted to understand how users are responding to interpretation provided by a computer-generated avatar. The survey was designed to gather insights at four dimensions, that is:
• Understandability, that is the users are understanding well the words or sentences presented by the avatar.
• Clarity, that is the users find it clear to understand the words presented by the avatar.
• Familiarity, that is if the users can identify the words presented by the avatar.
• Grammar, that is the users understand the translated sentences well and find it grammatically correct / understandable.
Assumptions and constrains:
Among the deaf users, there exists variations in acceptance and understanding of ISL from one state to another. This is often observed in the form of different signs being used in different region for a particular word. At the same time, like other languages, in ISL too there are social influences which adds to the heterogeneity. As a result, some ISL words are more universal and the others are not. there are users who prefer Indian Sign Language and there are others, especially in southern part of the country, who prefer American Sign Language and there are those who prefer regional variation of ISL. As a result, there were bound to be biasness and variation in the responses.
To simplify the analyses, the team agreed to classify the words / sentences in the following buckets:
• Grade 1 – universal: These are words / sentences which received more than 66% of positive response.
• Grade 2 – Substantial: Those words/ sentences that received more than 33% but less than 66% of positive response.
• Grade 3- Selective: those words or sentences that received less than 33% positive response.
Survey findings and insights
Based on quantitative analyses
Understandability
• For this dimension, all words / sentences presented in the survey was taken in account for analyses.
• The solution scored extremely high on the dimension of Understandability, with 96.67% of the users positively responding and indicated that they understood the given words/ sentences by the digital avatar.
• What is exciting is that about 70% of these words had universally understood and 26% was substantially understood. This is a strong indicator of liking towards the solution.
• Only about 3.3% of the words/ sentences presented had a selective appeal. On analysing this further we found that this was because a lot of users were not familiar with a given sign or the sentences given in the question were complex sentences.
Clarity
• Clarity of words was evaluated for the ISL words presented category 1 and category 2 questions. We had a total of 19 valid data instances.
• 100% of the users positively responded to this dimension and found the words presented by the avatar clear and understandable. This included 94% of words which were rated as “universal”, indicating that the given words received positive responses from most of the users. This, yet again, is a strong indicator of liking towards the solution.
Familiarity
• This was evaluated based on the ISL words presented in category 2 of the questionnaire, where the users were able to guess the given ISL word. (Please note we accepted the responses as correct if the users provided a word/ phrase that is closest to the meaning of the given word).
• The solution scored well on the familiarity dimension, with 90% of the words being correctly guessed by most users.
• Only one word had a mix response. The given word was “Agreeing”. We observed that about half of the respondents identify the word correctly, however the remaining referred to “goal/ aim”. This is because the ISL signs for the two words are remarkably similar. The team has made a note of the same and will be evaluating methods to avoid confusion in future.
Grammar:
• This dimension was evaluated based on the sentences presented in category 3 of the questionnaire, where the users were presented a set of 10 short sentences.
• The solution must do maximum catching up on this dimension. About 60% of sentences received highest response. Several users found the sentences grammatically incorrect and provided detail feedback on the correct order of sentences. It was observed that the translation model worked well for simple sentences but struggled when it came to longer or complex sentences.
based on qualitative analyses
The following observations were made based on the qualitative responses:
• Overall, the users feel that the prospective solution will be beneficial for them.
• The users are not happy with the translation model and have indicated that the order of words while converting from an English sentence into ISL sentence, is often incorrect. thereby diluting the meaning of the sentence.
• The users are responding well to the two newly created male and female avatars, named as Sanket and Mudra. At the same time, users provided several areas of improvement for the characters including changing the colours of Mudra’s clothes, re-validating signs, especially in those cases where there is more than one sign for a given word.
Voice of the users
The following captures some of the interesting quotes provided by the users in the survey:
“Thank you for your effort in making this app. It would be great help to Deaf World soon”.
“In India, Indian deaf people using different gestures ISL. Because different States different ISL using and mixed ISL also. Need to prepare & fix one ISL language for Indian deaf people. This app can help to understand ISL in India everywhere”.
“Very well made, more words and vocabulary needed”.
“Body movements and facial expressions are missing in few; hence it is confusing. Some words do not appear properly than actually how it should be”.
“Just incorrect Grammar order. All good word by word”.
Interpreter is particularly good in explaining”.
“The signs are clearly understood word by word. It is just only sentence which is grammatically incorrect in order. All good”.
Key take-aways for future:
• Augment the solution’s ISL word-bank to add more words from the ISLRTC dictionary.
• Focus on motion capture and automated techniques to scale the word-bank.
• Need to test the solution with live audience
• Need to validate the solution with a larger user-base and dataset.
• Need to work on the ISL translation model and factor more use-case scenarios for complex sentences and sentiments.
• Need to integrate various modules and build a coherent front-end solution.
• Improve the character set-up and relook at the color schemes, lighting, and camera angles.
Acknowledgements
We would like to thank all the users who participated in this survey. The names are not listed for privacy purpose. In addition, we would like to acknowledge Anuj Jain Secretary – National Association of the Deaf, Neeraj Agarwal – ISL Expert, Sarika naik – EVP & CMO Capgemini India, Kasturi Sridhar – Independent Sign Language Interpreter for evangelize this survey with the users and encouraging them to take the survey.