A Big Step Forward in Conversing with the Human Spirit, CIO News, ET CIO
Have you wondered how spam is filtered, search engines show most relevant pages, word app keeps correcting grammar, voice assistants talk to us while composing a message on WhatsApp, you get suggestions on the next words? How do evaluators quickly detect cases of plagiarism in assignments submitted online? Behind all of this is a major application of Artificial Intelligence (AI) called Natural Language Processing or NLP.
We humans have always been fascinated by others who can talk like us, whether it’s a parrot or a voice assistant like Alexa. Now parrots just repeat what they hear without understanding the true meaning of what they are saying. Yet our voice assistants are smarter than that; they can perform meaningful tasks for us, control our devices, and even have an interactive conversation with us.
The main area that gives voice assistants this power is natural language processing or (NLP). Simply put, NLP allows devices to process natural languages in written or spoken form and interact with us. In the case of voice commands, there is another step of converting speech to text when we speak to the device and text to speech when the device speaks to us.
How do these devices understand natural languages such as English, Hindi or French? Well, computers and algorithms don’t really understand words. Text is converted into numbers by computer programs because computers only understand numbers. In the case of NLP, there is an intermediate process called converting human speech into calculable properties or features called feature vectors like intent, timing, and feeling. This is necessary because language, unlike computer commands, has many nuances.
This all happens using some specialized machine learning and deep learning algorithms. These applications are then trained for specific tasks using a large amount of data. When we use such apps on our phones to dictate a message, we not only use the NLP app but also provide more data which is used by the program to improve its performance. In other words, these apps keep learning, which is why we notice an improvement in their performance as we use the app more and more.
We get alerts on the prevailing sentiment on social media platforms like Twitter especially when an incident has taken place like India losing a cricket match in the T20 world cup. How is it done? By reading a tweet, we learn whether the text displays positive or negative or neutral sentiment. NLP-based apps are trained to analyze tweets by examining words/phrases after removing articles, punctuation marks, etc. A dictionary of words/phrases is prepared based on previous tweets, marking them as negative, positive or neutral. New tweets are reviewed with appropriate hashtags like @T20World Cup for our example, and using NLP rated whether people are happy, sad, or neutral with the outcome.
Companies use tools based on NLP. One such application is to identify a topic in a large document and summarize the report. Using a well-trained algorithm to summarize these reports into key points saves a lot of time. Tools for writing summary reports are used by law firms and government departments where they have to wade through a large number of pages.
Governments seek input from citizens on draft policies. A large number of responses makes it difficult to manually analyze thousands of responses. NLP tools are ideally placed to do this for government agencies.
Chatbots, which are NLP-based tools, have suddenly become very popular on websites. They are trained in FAQs and can respond to calls efficiently, which eases the work of helpdesk staff. They also solve purely informational queries, which many people find difficult to locate on websites. Many departments have started using chatbots. The next step could be a multilingual Chatbot on government portals to allow people from different states to interact and get information/help by speaking to it in their native language or using text.
Voice assistants are the best known application of NLP. It’s a mix of several emerging technologies: a human will say, “Hey, helper, turn on the light.” Using voice recognition, spoken words are converted into text; the text is converted into numbers that the system understands; the numbers go through a set of trained NLP algorithms that give a message about the action. The action message is transmitted to the light bulb using the Internet of Things or IoT, and finally the light bulb is turned on. Of course, all of this will happen in a split second.
Another historical use case of NLP is language translation. Many such apps are available to help users translate from one language to another. For a country like ours, which has 22 official languages, a trained NLP translator can help us convert documents, websites, public notices into various regional languages. In fact, it can even help us speak in regional languages.
Although a lot of progress has been made, at the moment NLP cannot give us very high precision on the tasks. Even 75% accuracy is fairly well considered. One of the reasons is that languages have complex attributes such as sarcasm, irony, idioms and many more. For example, a person wrote “I want to jump off the bridge”. The app responded with a list of six nearby bridges. A frustrated passenger tweeted: “@XXX Thank you for sending my bags to Hyderabad and flying me to Kolkata at the same time. Brilliant service. #XXX.” The airline’s chatbot replied, “Glad to hear that. #KeepFlying XXX. Also, the NLP algorithm is very domain-specific, so an algorithm trained to analyze legal documents won’t do. to analyze medical records.
Finally, NLP algorithms give better results for languages like English, for which there is a large amount of data available. If we need similar performance on regional languages, we will have to put a lot of effort into collecting and cleaning the data and training these applications.
The authors are Prakash Kumar, CEO, Wadhwani Institute of Technology and Policy, and Apoorv Vishnoi is Senior Manager, Technology Training, Wadhwani Institute of Technology and Policy