A DEMO OF VIRTUAL INDIAN SIGN LANGUAGE INTERPRETER

The solution that is being developed in collaboration with Friends for Inclusion & IIIT Bangalore (sponsored by the Mphasis F1 Foundation) can enable seamless communication between people who are hearing impaired and those who can speak and hear. The project is called Virtual ISL Interpreter.

Our solution uses a blend of cognitive computing, natural language processing and 3D animation to deliver a real-world experience as if someone is actually standing in front of you and talking in sign language. Friendly avatars, both male and female, are used for sign language as the app processes the audio data that it receives in real-time.

Expressions play a very important part when a person is speaking in sign language. Most of the mobile apps that aim to facilitate communication between the deaf community and the world at large lack this important feature. In our app, we have worked hard at making our 3D animation use facial expressions to communicate feelings and contextual meanings.

The project was started in September 2018 and since then, the team has been able to build a prototype solution using the most advanced cognitive computing that converts speech into Indian sign language.

Why Virtual ISL Interpreter?

• There are over 63 million deaf and hearing-impaired people in India who can benefit from this solution. Due to communication barrier, they are unable to talk to each other as well as people who can hear and speak. Making matters worse is the fact that India has only a few hundred trained and certified interpreters for Indian Sign Language. As a result, millions who need this much needed service, don’t have access to it.  

What are the use-case scenarios for this solution?

Listed below are the various ways our solution can be productively used:

• Teaching tool for the hearing-impaired children and students.
• Personal communication tool, for example: it can enable a seamless communication at the workplace between deaf and non-deaf employees.
• Access to recreational activities like binge-watching a movie on your favorite streaming application.
• Collaboration tool add-on for services like WebEx, Zoom and Slack.
• Interpretation tool for digital content broadcasting through TV, streaming services, internet and chatbots.

 

What makes Virtual ISL Interpreter standout?

As mentioned above, facial expressions play a crucial role when people are using the sign language. This vital aspect is missing in other similar technology initiatives.

The solution is made in India for Indian users. It is made for the Indian sign language. Hence, our word bank is based on the Indian Sign Language Research and Training Centre dictionary.

So that the users can relate and find it easy to use the app, we have used 3D avatars that look Indian. Check out this blog posts where we have introduced our avatars.

The solution will be easy-to-use. People in all age groups – as long as they can use a mobile phone – will be able to benefit from the app.

The cross-functional team

The solution has been developed under the leadership of Prof. Dinesh Babu & Mr. Ankit Rajiv Jindal. The team comprised of people with and without disabilities, and included deaf individuals, sign-language interpreters, 3D animation professionals,  and Professors, PHD and Masters students of computer vision and cognitive computing from IIIT-B. Know more about the star team.

Early feedback

• Very promising initiative. It is the only technology solution that really understands the real needs of deaf people. I felt that the team appreciates that expressions are equally important as gestures in sign-language.
• Great to know that team is prioritizing Indian Sign Language and is taking a lot of efforts to Indianize the solution by creating Indian looking avatars and following ISLRTC dictionary.

Contribute to our open source project

In an effort to expedite the development of this project and so that other bright minds can build on  our work and find innovative solutions that help the community to overcome barriers, we are making our work available to the open-source community.

Check out our Get Hub page