Live automated captioning has become a growing presence in the lives of many of us who are deaf or hard-of-hearing. From captioning live presentations, to providing transcripts of online meetings, to on-the-go captioning with mobile devices, these AI (Artificial Intelligence) tools continue to improve in accuracy and speed. In the past few years, they have rapidly become a versatile addition to our toolkit of strategies for improved accessibility in academic and other settings.
Thibault Duchemin and Skinner Cheng are the co-founders of Ava, a live automated captioning app with various transcription features, including translation and text-to-speech. The assistive technology is designed to enhance communication in different scenarios such as professional, academic and social situations. As a CODA (child of deaf adult) and a deaf individual, respectively, Thibault and Skinner have close ties to the deaf and hard-of-hearing community. We caught up with both of them to learn more about the origins of Ava and their experiences in creating this professional captioning tool. Below, Thibault first shares how Ava got its start, and then Skinner answers our questions about his journey from Taiwan to working with Ava.
Thibault: Backstory & how Ava began…
I grew up the only hearing person in a Deaf family, i.e. a CODA (great movie). My sister wanted to be a lawyer, but with the cost of interpreters to help with closing statements, client meetings, etc. there would be financial issues and communication barriers. I understood the challenge, and wanted to help!
While I was at Berkeley, I started working on “smart gloves” to translate sign language, which stirred a lot of excitement and showed there was a clear need for a communication tool.
After creating a prototype, plus hundreds of hours and bike rides across the Bay Area to meet and talk/sign/write with our potential users, we got some hard-earned learnings during customer development. We kept the mission – to bridge communication gaps between hearing and Deaf worlds – but we pivoted the product.
The pivot was to move to a mobile application that transcribes group conversations using speech-recognition technologies. The goal was 24/7 autonomy that would allow Deaf/Hard-of-Hearing users to understand and participate in group situations, effortlessly. Together, we designed a product that was easy to access in social, academic and professional conversational situations.
In early days, Skinner would go to an isolated space to check his phone at events. Today, in small groups, he uses the app [Ava] to communicate with others, transcending the silence blockade. At lunch and during meetings, we all use Ava to connect with each other.
What was just my personal story now became a team story as we slowly dissolved the communication barriers between each other.
Every day, these simple moments justify the thousands of hours we work to develop and improve this tool. We have a 45-person team (and counting), currently located between San Francisco and Paris. Skinner is a brilliant developer, who is Deaf and an inspiration to us all.
Skinner interview:
1. Tell us about your background? For example, tell us about your hearing loss, your schooling, and/or your family/culture
I was born in Taiwan. I was not born Deaf, but lost my hearing when I was 2-3 years old. My mother told me it was caused by an injection of medicine that I received at a clinic, which contained material that harmed my hearing.
I never received a standard education for Deafness in Taiwan—it is more offered and accepted here in the USA. I was educated by my mother to read lips and also speak. I never learned Sign Language because my mother wanted me to learn how to communicate like hearing people. I attended Deaf school for half of 1st grade, but then my mother enrolled me in a hearing school, so there were no disability accommodations I could rely on.
During primary school, my mother taught me math and other subjects. I started working with tutors one-on-one in high school until I graduated. In college, there were no captions, so I taught myself for the most part.
I usually communicate with my family, friends and colleagues by speaking, but sometimes we communicate through writing, if it’s too complex or difficult to say something clearly. Writing on paper was later substituted with a smartphone, which we often use today.
2. What has been your professional trajectory?
In college and at the university in Taiwan, I studied Computer Science. After graduation, my work was all about coding. I never worked at a large corporate company. I spent several years with a startup, which was later acquired by a medium-sized company.
My job position was always Software Engineer. I took a Senior engineering position 2-3 years before I left Taiwan to move to San Francisco. I studied and received my second Computer Science degree at the University of San Francisco. After graduation, I began working with Ava as Co-Founder and CTO [chief technology officer], and currently spend most of the time working as an Engineer within the company.
3. How did you meet and how did you come together to create Ava?
Many people ask this question—and it’s a fun story, indeed. When I graduated from the University of San Francisco, I was looking for a job, which could sponsor me to stay in the US and in San Francisco, specifically. I saw a post on a bulletin board at the school, which was from Thibault Duchemin and Ava’s COO, Pieter Doevendans. I don’t remember the exact words, but it said they were offering some accommodations (with a machine, or an assistive tool), which would allow Deaf and hard-of-hearing people better means to communicate during job interviews. So, I contacted them. And that was the first time Thibault and I met—in some cafe in downtown San Francisco.
Thibault didn’t know at the time that I was TOTALLY DEAF and he probably overestimated my lip reading ability. So after Thibault spoke for almost 30 minutes, I had to interrupt, and tell him that method of communicating wasn’t working. Eventually, we used my laptop to type and communicate. At that point, I understood they didn’t have the tool built, but wanted to do some user testing.
The second time we met, I met with Thibault and Pieter in another cafe in Millbrae, where we still used the laptop to communicate. I remember onlookers in the cafe were curious and inquired what we were doing. Afterwards, Thibault asked me to come to the University of California, Berkeley, where he described what they wanted to do. He asked me if I could help develop the MVP (minimum viable product) for the Android smartphone—and I did. We moved to San Mateo, launched the startup named Transcense in Boost, which is a startup incubator. After we raised adequate funding, we moved to Oakland. We got an established designer to help us design the company icon, and there you go… Ava was born!
4. Tell us what Ava is; how does it work?
Ava is an assistive tool and technical solution that Deaf and hard-of-hearing people can use—on their smartphones and personal computers—which transcribes what hearing people speak, in one-on-one or group situations. In brief, it’s a communication bridge between Deaf and hard-of-hearing people and hearing people.
5. Does Ava work with multiple spoken languages? How did you choose what languages to invest in?
Yes, Ava does work with multiple spoken languages. However, Ava cannot automatically detect the exact spoken language and switch between the languages just yet. The user has to choose the spoken language, then Ava will accurately transcribe.
We also provide translation, so if people in a conversation speak different languages, they can choose the language they speak, and Ava can translate different spoken languages into the language they can read.
Since our teams are located in the US and France, we focused our primary support on the English and French languages. However, it transcribes many languages and we continue adding more based on user request, the market, or direction from our Sales team.
6. Who is using Ava? What situations are they using it in? The Mind Hears readers are primarily working in academic settings; in what manner is Ava being used in academia?
Deaf and hard-of-hearing people are Ava’s primary users. The DHH community uses Ava when they need to communicate easily with hearing people—when they want to know what someone is saying, and speak if they are unable to voice as clearly as they would like.
In academia, Ava can be used to transcribe speech from instructors or students for anyone who is Deaf or hard-of-hearing, and also for hearing students to have transcriptions of lectures. From personal experience, having captions to understand what a professor or teacher is saying makes the lesson a lot more interesting. Accessible accommodations make a huge difference with understanding and learning.
Ava also provides Scribe service, which merges AI with a human scribe to catch nuances and improve the accuracy of real-time transcription. We also offer CART services—a service I benefited from a lot when I studied at the University of San Francisco. Our mission is to make Deaf and hard-of-hearing peoples’ lives easier and happier in any situation, including academia—that’s why Ava was created.
7. Has knowledge exchange between academia and industry played a role in the development of Ava? Are there any intersections between industry and academia that have been important in your entrepreneurial journey?
Yes, absolutely. A concrete example is that we have a team in France, and they focus on AI and speech, and most of them are researchers in academia. Most of us in the US are engineers and part of industry, and we help convert the team’s ideas into real tools, which help people. The perfect intersection between academia and industry is to make the dream come true.
8. What limitations do you see for Ava? What are the greatest challenges in getting it to work as the tool you envision it being? Where do you see technology like Ava going in the future?
I feel most comfortable using an assistive tool, whether it’s an app or device that I can use autonomously—rather than asking other people in a conversation to set up the application with a smartphone, tablet or laptop. Also, accuracy of speech is always the challenge and something we are constantly improving. Everyone within Ava is aware that the limitations lie in certain situations where there may be background noise or other reasons why accuracy is not 100%. Other companies that offer captions experience the same issues. These are the challenges we want to solve, which will be the greatest of achievements, if we can succeed.
In the future, I envision more of a Utopia, where I can wear glasses, and the caption will show up like a bubble caption attached to the speaker in the screen of my glasses. With such a device, I won’t need to ask another person to set up the device and application. And I’ll know who is speaking, no matter how many speakers there are, so the conversation is clear for me and I can keep pace with a discussion.
Also, I haven’t figured out a way for others to better understand my speaking—I can speak, but it’s unclear. I know a little sign language, but I’m used to speaking, so it would be great if there was a way to make unclear speech more clear.
9. Have you faced any challenges related to hearing-loss on your entrepreneurial journey?
Yes, there have been a lot of challenges. Communication is the pain and hardship. And the side effect of not being able to communicate as one would like is isolation, which makes me feel alone and excluded. I think that’s also why I don’t prefer to work in large companies, because I am sure the situation would be worse. On the other hand, I have been lucky as colleagues I had when I lived and worked in Taiwan accommodated my situation. And here, since I am working with Ava, which aims to help Deaf and hard-of-hearing people, the challenges due to my situation are turning less and less.
10. Do you have any advice for people with hearing loss who might be starting out on their own entrepreneurial journeys?
I was lucky, and my journey is a bit different. However, I think, regardless of any special circumstance or situation, in any stage of the entrepreneurial journey, you should not be alone. It is great if you have people who can empathize, accommodate and understand you. Surrounding yourself with people who are also passionate about the mission and understand why you want to start your entrepreneurial journey will be beneficial.
11. You are both from different countries, and through your work on Ava have likely interacted with deaf/hard of hearing people from many parts of the world. Do you have any insight to share on perceptions of deafness in different parts of the world?
I know disability accommodations are very different in different countries. Luckily, it’s the trend that technologies, including AI, are growing everywhere and have been applied in some modern countries. But for countries that don’t have advanced technology such as the Internet, I think we should try our best to support them by providing more resources.
I moved to the US because accessibility accommodations for Deaf people were better here than in Taiwan. It’s gotten better over the past four decades, but there is still a long way to go.
**In case you are wondering, TMH received no financial compensation from Ava for publishing this interview. We were just really interested to hear their story!
4 Comments