An Automatic Conversion of Punjabi Text to Indian Sign Language

An escalating focus worldwide on the development of persons with disabilities results in the disability-inclusive 2030 UN agenda for sustainable development. This paper discusses the development of Punjabi text to Indian Sign Language (ISL) conversion system aiming for the better communication and education of hearing-impaired people. The developed system accepts Punjabi text as input and deliver output as 3D animation. The currently created corpus involves more than 100 commonly used Punjabi words under various categories. This system has been tested on all the prepared words. Results of the developed system are very encouraging and the work is still going on with full enthusiasm. Since there is no universal or written form of sign language available, so generating synthetic sign animation is the best solution. In near future it can be implemented as a tutoring system to promote sign language education among common people of Punjab.


Introduction
Human life without communication is hard to live.Different ways are used to communicate and share their ideas between two people (known as speaker and listener).To communicate between people speech and gesture are the most common ways.Speech communication is performed in audible/vocal sound and understood through hearing power.On the other hand, gesture communication various body parts specially hands movements are used to share ideas between people [1]- [3].Sign Language is gesture language that is received and understood through the power of vision [4].Hand-shape, Location, movements are basic parameters of sign gesture [5].Hearing impaired people have many difficulties in their life due to lack of knowledge and communication gap [6], [7].On the other hand, normal people are not aware of sign language.
To create a bridge between hearing impaired people and normal people, we need a translator, which will help to translate their communication to each other.As per geographical area variety of sign language and spoken languages are used for communication [8] [9].The target people in this research are mainly Punjabi language understanding community because of the facts that Punjabi language is first official language in Punjab (India), third in Canada, fourth most spoken language in United Kingdom, tenth in the world and eleventh in India.Deaf community of India used Indian Sign Language as their primary language of communication.But there is not much teaching material available for hearing-impaired people.Parents of hearingimpaired children are not aware about Indian sign language.In public dealing offices, there are urgent requirement of ISL interpreters.Rapidly increasing the needs of users, make the development of online and offline applications are the most promising fields in computer processing.Variety of applications have been developed to serve diverse range of users [10].The focus is on creating an automatic machine translation system for Punjabi text to Indian sign language.
Spoken languages use vocal medium and are 2D (two Dimensional) whereas Sign languages use visual mode of communication and are 3D (three dimensional).In sign language space and time are the important parameters, therefore require specific technique for processing.While signer perform gesture, signer uses his/her whole body but most commonly anywhere from above the head to waist height on the body and space around signer during communication.Hands of a signer are the major component for manual and facial expressions used as non-manual components in sign language.In time domain, speech signals produced sequentially e.g. the word "sound" can be phonetically separated into "s-o-u-n-d".While performing gesture in sign language, movements are processed in time domain.In spoken language, a phoneme is a smallest unit of sound that distinguishes words in particular language.For example, the word pen, ten, hen and men distinguish by p, t, h, and m.In sign language gestures are distinguished by features such as hand shape, orientation, location, movements.

Research Motivation
The inspiration for this piece of work is the enhancement of accessibility to informative announcements for deaf and hard of hearing people.The essential motto in the United Nations -"Persons with disabilities as beneficiaries and agents of change in society and development" is progressively captivating concrete forms.In the UN Convention, the duality of disability both as a human rights issue and a development issue, has been recognized which addresses the 2030 agenda for sustainable development as disability-inclusive. Furthermore, the Figure . 1 shows the total percentage of disabled people in various categories as per census record conducted by government of India in 2011 and hearingimpaired people has the second highest rate of disability.The primary motivation for this research work is a web-based communication and learning tool belongs to the improvement of Indian Sign Language skill.
• According to the 2011 census there are 2.68 crore hearing impaired people, the number of deaf schools are very low [11], [12].• In addition to analysis, most of the work of sign language applications do not contain valuable information and they did not support our target input language [13].• One more concern among deaf students, they do no develop enough to understand other written language as other users developed so unemployment rate of hearing impaired people is very high in developing countries [14], [15].
Keeping this motivation up and considering the census of India 2011, where the percentage of disability in hearing is around 19%.This paper discusses the developed automatic machine translation system for hearing impaired people.

.2 Need of the Study
Sign languages lack political recognition along the globe and are poorly resourced in comparison to their spoken language counterparts [17].Sign language is an important tool to bridge the gap between the people who can't hear and those who can hear.Sign language is not only used by hearing impaired person, it is also used by the parent(s) of deaf child, child of deaf person, teacher of the deaf student and so many other areas of communication with deaf [18].Hearing impaired person can do everything which a normal person can do, except hearing.The target citizens in this research are mainly concentrated on Punjabi language understanding people.The reason is quite clear from the following facts representing the status of Punjabi language in the whole world [19], [20]: • Indo-Aryan Language Punjabi is spoken by more than 150 million speakers in the world.• First official language in Punjab (State of India) • Second official language in Delhi and Haryana (States of India) • Third most spoken language in Canada • Fourth most spoken language in United Kingdom • Tenth most broadly spoken language in the world • Eleventh most popular language in the India So, the need of the hour is to develop such a system that can assist these people.It is really necessary for these people to have such a system that will lend a hand to them for communication as well as upliftment to achieve education.Consequently, they can streamline their lives by better understanding and good command over the basic concepts.currently hearing-impaired people are unable to understand the important announcements.Sign Language interpreter can be used in several applications to provide valuable facilities for specially-abled people.Few of the application areas include: Public dealing offices, travelling departments, Social life communication, Education departments, Sports activities, Religious and cultural activities, Media and Telecommunication, etc.In social perspective, the scope as well as its relevance is not restricted.Rather this research work can be extended to greater coverage.The aim is to bring these people under one roof as others to provide them equal opportunity to communicate, educate, rise, shine and live.

Social Relevance of the study
Our research work presents a potential technology solution for enhancing the communication process for hearingimpaired people on web platforms through automatic machine translator.

Related Work
Over the past few decades when spoken languages were not properly standardized, people used sign to convey messages with very limited meaning.Early days medieval monks had taken vows of silence to communicate in a simple form of sign language.In 1620, Juan Pablo, a Spanish priest published the first method for educating the deaf children from modern text.First school for the deaf in Paris found by Abbe Charles in 1771 and created standard sign language [21].
Elliott et al., developed a ViSiCAST (Virtual Signing, Animation, Capture, Storage and Transmission) system for sign language which is capable of translate spoken language or text to gesture oriented notation.Major advantage of the research is, it provides more flexible and dynamic sign gesture animation description system.The drawback of the system is that to edit or add new sign animation, hardware (sensors) setup is required [22].Cox et al., translated United Kingdom Post Office Clerk's speech into British Sign Language (BSL) using sign avatar "TESSA" (Text and Signing Sup-port Assistant) to analyze the conversation transactions between PO clerk and deaf customer.Facial expressions of avatar make more attractive and natural 3D sign gesture for deaf people.Improvement in speech recognitions where phrases need not be speaking as the same way they recorded in database word by word [23], [24].Hanke described in detail about HamNoSys (Hamburg notation system) used to transcribe spoken/written language to sign language in the form of 3D animation.[25].
Nasr proposed E-Learning paradigm for Deaf/HOH.Main objective of the proposed paradigm as virtual classroom is to provide equal rights in acquiring knowledge in class room [6].Raghavan et al., provided a new architecture for signing gesture using 3D avatar for hearing impaired people.Main objective was to develop dynamic method for convert English text input to gesture animation [26].
Kaur and Kumar presented a web-based system to generate HamNoSys notation corresponding input word.There is no automatic animation generation system for Sign language representation and need to combine various tools used in this research [27].Kaur and Kumar have implemented a Short Message Service (SMS) system for deaf people and tested with 250 daily used sentences.The system comprises three major components: 1) sign visual interface for deaf person 2) sign language to English text sequence generator 3) English text SMS to speech translation system [28].Shahriar have proposed a two-way smartphone application for deaf people living in Bangladesh.This system has capable to translate Bangla speech into Bangla sign language and Bangla text to Bangla speech.To translate speech into sign language, CMUSphinx toolkit had been used as a speech recognition system.Whereas for text to speech, the text is sent over Google translator server and server responds as an audio stream of that word [29].Sugandhi have developed an 3D avatar animation based multi-lingual text to Indian sign language translator system.The system has feature to accept input in English and Hindi text.The input text is converted to HamNoSys and translated into Signing gesture markup language (SiGML).SiGML is eXtensible Markup Language (XML) and it is easy to transfer on the web platform [30].Dewani have implemented an e-learning system which is a resource for learning and translating English text into Pakistani sign language (PSL).The input of data has been acceptable only through text entries and compared with the database entries.If word is available in database then PSL word displayed otherwise PSL of every single character is displayed [10].Kaur and Singh presented a system to translate and generate ISL animation corresponding to 100 Punjabi words.Input to this system is purely text based and output of the system is in 3D avatar.It needs to add nonmanual features of gesture and provide system as automatic translation tool for text to 3D sign animation [31].
Verma and Kaur developed a system to generate Sign language gesture animation corresponding to Punjab text as input.Research work in ISL is very limited and lack of research in this field is very difficult because there is no proper grammar of ISL.Non-manual components are not used to express gesture using other parts of body rather than hands [32].Nair, Nimitha and Idicula presented Malayalam text to ISL machine translation system using HamNoSys.System's accuracy could be increased by adding facial expressions, lip movement, which is not present in the current system.Implemented system can be used as learning sign language tool for people of Kerala [33].Goyal and Goyal developed an Indian Sign Language dictionary using synthetic animation using HamNoSys notation system.Works for English words only and required complete automation machine translation system for end user [34].
Taner Arsan and Oguz Ulgen designed a system that convert sign language to voice and voice to sign language.They implemented two approaches: first is motion capture system using Microsoft Kinect Sensor XBOX 360 that recognize the human gesture and translate it into audible speech.Second approach is to recognize audible speech of natural user and display sign language in the form of image or video.CMU Sphinx decoder is implemented to recognize spoken words using java programming language [35].ATLASLang machine translation system for Arabic text to Arabic Sign Language (ArSL) is developed for deaf people living in Arabs [8].ATLASLang system perform morphosyntactic analysis for the input text and generate 3D animation of human avatar.

Notation Systems for SL
SL is visual language which is understood only though the power of vision.Researchers are doing great work for deaf community.Notation systems are sequence of characters or symbols to represent sign language.To make notation system for SL is strongly required to advance the study of its structure.An analysis of the sign language notation systems that exist will be carried out in this section.Some of the notation systems for sign language that are currently being used in various ways are discussed below.
Stokoe Notation: William C. Stokoe, proposed a notation system for American Sign Language (ASL) in 1965 [36], [37].It is the first phonetically based notation introduced for sign languages; most other sign language notations are based on this notation.Stokoe describes that there are three elements in ASL which distinguish each sign. 1) Signs like CANDY and APPLE are different due to shape of hand.2) SUMMER and DRY are different in place of articulation.3) On the bases of movement signs such as TRAIN and CHAIR can be distinguished.So Stokoe proposed three parameters: Hand shape, Location, and Movement for Stokoe notation system.
• Designator (dez): determine the hand shape • Tabula (tab): define the location of hand • Signation (sig): describe the movement or action of sign SignWriting Notation: Valerie Sutton developed a different notation system named as Sutton SignWriting system in 1974 [2], [7], [36]- [39].SignWriting system is based on previously developed notation system to represent elements of dance choreography.Any sign language in the world could be represented using SignWriting notation system because it depicts pictorial view of Hand shape, facial expressions, palm orientation, hand position and movement.A SignWriting Markup Language (SWML) developed to describe SignWriting notation in XML format.For writing Japanese Sign Language (JSL) with SignWriting, JSPad tool is developed which translate SignWriting into SWML file [40].
HamNoSys Notation: Thomas Hank developed Hamburg Notation System (HamNoSys) for Sign Languages which is an alphabetic system describing signs on phonetic level [25], [41], [42].Initially this system system was handwritten, but later on special Unicode font has been developed.HamNoSys is capable to accept manual as well as non-manual features extracted from sign gesture.HamNoSys support manual as well as non-manual features of sign language.It consists of handshape, location, orientation, movement, and non-manual components.XML based Signing Gesture Markup Language (SiGML) script is introduced for HamNoSys notation by the University of East Angila which is used for generate synthetic 3D human avatar animation [38], [43].
Gloss Notation: One more notation system is used for write sign language is Gloss notation.Gloss is the most commonly used notation system, but it has its own limitations.Gloss notation stem root words from spoken language sentence.It uses only Capital letters for representation so when writing gloss notation then number of words in gloss notation and they do not equal to original sentence.Indian Sign language use single or both hands to represent sign gesture.Most commonly right hand is used as dominant hand and left hand is used as non-dominant hand.So, our proposed automatic machine translation system implemented using HamNoSys notation system.

Methodology Used
Machine translation is one of the oldest fields in computing.Automatic machine translation of sign languages is complex task because there are basically two steps: the translation component, and recognition of sign language or synthesis gesture that is depending on the direction of translation process.Synthesis or recognition can be done within the same language but between two different medium such as speech and text or text to speech.while we are translating a language to another language such as English language to Hindi language, needs mapping between two languages.Sign language machine translation include both the components Natural Language Processing (NLP) and a synthesis or recognition component.In our research work, we synthesis the gesture using 3D avatar to translate text into sign language which render animation according to input text.Our translation system as illustrated in figure 2, uses database query to search given input text from sign corpus and transfer corresponding HamNoSys sign notation to markup language converter module in SiGML format.SiGML formatted file is lightweight to transfer over internet and easy to render using 3D avatar in webpage.
Input Module: Most of the previously developed machine translations systems neither automatic nor platform independent.Previously developed system mostly used different tools for different operations such as ESign Editor for generate ISL's hamnosys notation and JA SiGML URL App for displaying output in animation format.Both of the tools required JRE (Java Runtime Environment) for every single user on Window, MAC, and Linux OS.To operate on mobile platforms, developed systems does not work directly.They need to develop build mobile applications for every unique mobile platform.So that to make automatic machine translation ISL system platform independent was also our research goal.Frontend of our proposed system is designed using ASP.net web programming language.System have two options for input 1) typing text in Punjabi language, 2) directly click on words.Translation Module: Translation module of the sign language requires mapping between input word and the format suitable to drive a graphical avatar animation.Our ISL corpus which contain information about the hand shapes, position, orientation, and movements of the gesture.In this module two major operations are performed: 1) for fetching HamNoSys notation from MySql database for input word, 2) convert HamNoSys to SiGML markup format.
Output: Our purposed system provide output in both format 1) natural video recording of sign language experts, 2) synthetic animation of 3D human avatar using Visicast's project.

Figure. 2. Punjabi text to automatic machine translation system for ISL (Punjabi)
The following algorithm explain the basic steps included in proposed methodology for automatic machine translation of Punjabi text to Indian sign language.Whereas Wi is the input parameter which takes the input as simple text in Punjabi Unicode from input text box or from GUI by clicked on button.In the next input parameter Wi forward to Web server (Ws) where ISL corpus is stored and then Ws return response code (Rc) along with response data (Rd).If Ws is working fine and input request is in proper format as required then Rc should be return as 2xx most commonly 200 (successful request).Ws perform database search query for the input parameter Wi and store results into Qr.If the Qr is not null means input word found in ISL corpus then HamNoSys (Hm) data will be extracted from Qr otherwise Rd set to be null that means Ws is working fine but input word not found in ISL corpus.In successful Qr, HamNoSys script (Hm) translation process convert Hm to SiGML (Sg) markup language.Sg is main script which is required by the 3D avatar.Sg is written in predefined XML tags which generate synthetic animation.If Rc is other than 2xx then Ws is not working properly or input request is not in proper format as required by Ws.Further, when it is converted into SiGML, the system translates them into two different tags.In the final stage, a 3D avatar that was developed by (ViSiCAST) uses this SiGML file as input and render animation as per SiGML tags.

Implementation Details
In this section, the implementation details of the proposed work are given.Automatic machine translation system is developed using ASP.netWeb server programming language and MySQL as database server to store corpus detail.HamNoSys Sign language notation script is used to translate Punjabi words in ISL.For gesture animation representation 3D avatar is used which is developed by "Virtual Humans Research for Sign Language Animation" and available at vh.cmp.uea.ac.uk.There are 10 or few more avatars such "anna, siggi, max, lunna, etc." and in developed system "marc" avatar is used which is male mask model and clearly visible.3D avatar can translate SiGML xml format so SiGML translation is also required to convert HamNoSys notation system to SiGML format.The screenshots of the developed system have been taken, which are shown below in a sequential fashion ranging from Figure.

Experimental Results
In the first experiment we analyse the performance of time taken by the process of Punjabi text to ISL translation.Table 1 display the results of performed three different experiments: E1) stream ISL video from webserver corresponding to input word, E2) fetch HamNoSys notation corresponding to input word then perform translation process to generate SiGML markup file, E3) fetch already saved SiGML markup file corresponding to input word.
• Non-manual components exist completely.

Cons:
• Streaming speed should vary according to bandwidth • Same as E2 Cons: • Same as E2 Next experiment is performed using randomly selected ten words.Table 2 display the experiment results of translation ten unique isolated words.In this table Input column display isolated words in Punjabi Language along with English language.Next column shows the output of given word in synthetic 3D animation.Description column describes, how the animation is performing.Output of every signed word is compared with ISL dictionary book as well as compared with recorded videos of ISL.

Input (Punjabi)
Output Description  Both "fist" hands, facing each other at chest level, shake from side to side in a symmetric fashion

Discussion
Nowadays, eHealth is very popular research area and technology is very helpful to diagnose and improve health issues [44].There is an expert system to analyse the health of patient for arthritis disease that is easy to use for both medical specialist and non-specialist.Developed Expert system used fuzzy logic model to provide highly valuable results [45].Another expert system using Mamdani Fuzzy Inference helps to diagnosis heart disease [46].Social media sites and applications also helps to diagnose health issue and serves as alternative to extract important knowledge from posts.Author analysed the emotions from the facebook post and provide the results related to depression [47].Same as extract data from twitter and detection of Real-time hay fever is performed by using character embeddings and neural attention [48].Digital technology is doing good work by providing sustainable healthcare systems.As per our research work we are proposing technological aid to people who are unable to hear partially or completely.The currently created corpus involves more than 100 commonly used Punjabi words for communication under the various categories.Table 1 display the randomly selected 10 words with their output in Sign language.The conversion of Punjabi word to ISL has been discussed with ISL experts and teachers of School for Deaf, Saifdipur, Patiala, Punjab, registered under the Society for Welfare of the Handicapped.The generated sign animation output was shown to the users and according to the quality and understandability of signs, they were classified as valid or invalid.Results of the developed system are very encouraging and the work is still going on to scale up for more words with full enthusiasm.Technically we analyse that Visicast's 3D avatar require SiGML script for animation so that SiGML files can be stored in ISL corpus rather than conversion of HamNoSys to SiGML on every request.It will minimize the response time and utilize the resources.
SignWriting notation was initially introduced for communication purpose rather than linguistic purposes where as HamNoSys notation is developed for research purpose and provide linguistic description of sign.Some advantage of HamNoSys notation system are that it is written linearly (left to right as English language), it has its own Unicode font, formal structure which is easier to store in database and parse during machine translation.Advantage of Gloss notation is that mapping between the source language and target language is easier because transcribed sign words will already be in words from target language.

Conclusion and Future Work
Since over 4 million people go through hearing problems in India it is exceedingly significant to build up the automatic system for the conversion of Punjabi words to Indian Sign Language using synthetic animations.Machine translation for spoken language to sign language require intermediate notation.After analysis of recent work, we examined Stokoe, SignWriting, HamNoSys, and Gloss notation are most commonly used.HamNoSys is capable of writing signs used in every sign language.This notation has feature to provide written form of sign language such as English, Punjabi or other written languagesHand shape, orientation, location and movements are basically used in gesture.This paper presents an early work on developing the system for generating ISL signs for input Punjabi words.This user-friendly system could be used by especially the people who know Punjabi language for learning sign language.It also integrates an interactive choice to include signs of more words into the sign database.The usability and overall performance of the system can be increased by adding large collection of signs as well as words used by the people.Accuracy of the system could be increased by adding non-manual component of each sign.
Significance of the area of the study is that research will be useful where announcements are delivered to people in audible form or textual form of Punjabi language and EAI Endorsed Transactions on Scalable Information Systems 10 2020 -12 2020 | Volume 7 | Issue 28 | e9

For
example, Punjabi word "ਪੰ ਜ" is fetched from corpus in the form of HamNoSys ( ).In this example five icons are used to write sign notation.First icon represents a hand-shape parameter as open hand.Second and third icons represent the hand orientation, in which up arrow is used to set direction of hand and ellipse icon represent the palm orientation parameter.Fourth icon represents above shoulder's location for placing hand and last icon is for movement parameter.After that, these sign notation parameters are translated into SiGML format.SiGML format uses XML tag standards to generate "sigml" file.For example, first icon (handshape) is converted into "<hamfinger2345/><hamthumboutmod/>".It uses two tags because in above mentioned example open hand shape is a combination of two different character encodes ( of Punjabi Text to Indian Sign Language EAI Endorsed Transactions on Scalable Information Systems 10 2020 -12 2020 | Volume 7 | Issue 28 | e9 3(a) to 3(d).

Sign Language translation algorithm 1.
Wi as input word in simple text 2. http request to web-server with input parameter Wi 3. wait for response code Rc and response data Rd from web-server Ws a. if Ws success then i. search query result Qr for Wi in ISL corpus table ii.if Qr is not NULL 1. Add HamNoSys script from Qr to Hm 2. Translate Hm into SiGML script and store in Sg 3. Rd set from Sg iii.Else 1. Rd set as NULL // not found iv.Rc return 2xx // for successful web request b. else if Ws client error then i. Rc return 4xx // request contain bad syntax c. else if Ws server error then i. Rc return 5xx // web server not able to process request 4. If Rc is 2xx then and Rd is not NULL then a. Rd set as input to Visicast's 3D Avatar b.Start init() of avatar c.Animation performed successfully 5. Else a. Display error message: Sign or webserver not found

Table 1 .
Experiment results to analyse performance

Table 2 .
Punjabi words, Indian Sign Language and their descriptionAn Automatic Conversion of Punjabi Text to Indian Sign Language EAI Endorsed Transactions on Scalable Information Systems 10 2020 -12 2020 | Volume 7 | Issue 28 | e9