Abstract
Sign language translation systems often overlook the emotional signals that are necessary for a complete understanding of gestural communication. In this paper, we propose a multimodal approach to understanding sign language that combines manual gesture recognition with facial expression-based emotion recognition to produce more expressive translations. We are developing a two-threaded Long short-term memory (LSTM) architecture that processes sequences of hand and body postures for signs and sequences of facial landmarks for emotions, combining them using a strobing mechanism to condition the interpretation of the sign depending on the emotion detected. The model is trained on a specially collected dataset of gestures in Kazakh sign language (KSL) and facial expressions. Integrated with a large language model for text generation, our system translates signs into natural language sentences that reflect the speaker’s emotional tone. On a small test set, the model achieved almost 70.6% accuracy in gesture recognition and 77.4% accuracy in emotion classification, and the generated output data demonstrated corresponding tone changes depending on the signer’s facial expression. These results illustrate the possibility of emotional translation into sign language and highlight the importance of non-scientific signals for a full understanding of the intentions of the signer.