SignPlug - ISL plugin for the web

Masthan Sharif Shaik1, Mohammed Raza Ur Rahman1, Aditya Singh Rathore1, Seshadri Mazumder1, C.V. Jawahar1, Vinay P. Namboodiri2
1CVIT, IIIT Hyderabad, India 2University of Bath, United Kingdom
ICVGIP 2026
SignPlug at a glance: the input sentence 'He will come home' is converted to the glosses HE, HOME, COME, whose stored keyposes are retrieved from the gloss keypose dictionary and played on a real-time, CPU-only SMPL-X avatar.
An input sentence is turned into ISL glosses, each gloss's corrected keyposes are retrieved from the dictionary, and a CPU-only SMPL-X avatar signs the sentence in real time.
The SignPlug avatar signing the ISL sentence 'He wants to eat now'.
"He wants to eat now," synthesized end to end.
The SignPlug widget embedded in a web page, signing 'He wants to go to the office'.
The SignPlug widget embedded in a web page.

Abstract

We present SignPlug, a lightweight, real-time framework for synthesizing continuous Indian Sign Language (ISL) avatar motion from input sentences directly on the web. The system is designed to bridge the communication gap for Deaf and hard-of-hearing users. Digital accessibility research has grown substantially, yet native browser support for sign language rendering remains largely absent. Existing signing avatar pipelines are either motion capture systems too costly to scale, GPU-hungry neural motion generators, or notation-driven systems such as HamNoSys and SiGML that demand substantial linguistic expertise and yield mechanical, unnatural movement. None of these reconstruct motion directly from real signers at dictionary scale.

We address this gap by reconstructing SMPL-X keyposes directly from monocular ISLRTC ISL dictionary videos and refining them with a purpose-built sparse correction annotation tool, Gizmo. This yields the Widget Dictionary, a reusable gloss pose dictionary with corrected body, hand, and finger articulation. At inference time, glosses predicted from an input sentence are retrieved from this dictionary, retargeted onto a canonical SMPL-X avatar, and composed via spherical linear interpolation (SLERP), avoiding both frame-by-frame animation and GPU-based motion generation entirely. SignPlug therefore enables low-latency, browser-deployable ISL sentence synthesis grounded in captured human motion rather than symbolic notation.

Videos

Widget Dictionary

How the Widget Dictionary is built: monocular ISLRTC dictionary videos, automatic SMPL-X estimation, sparse keypose selection, and manual correction of body, hand, and finger articulation.

Website Demo

SignPlug runs entirely in the browser, with no GPU and no server-side model: a user selects a sentence, and the avatar animation is synthesized and played back with playback and scrubbing controls.

SignPlug Pipeline

Offline, each ISLRTC dictionary video is processed by SMPLer-X to recover an initial SMPL-X motion sequence, a sparse set of linguistically salient keyposes is selected, finger and wrist errors in these keyposes are corrected by hand, and the result is stored as a reusable dictionary entry. Online, an input sentence is converted to an ISL gloss sequence with an LLM, the corrected keyposes for each gloss are retrieved, retargeted onto a single canonical SMPL-X avatar, and joined with per-bone SLERP transitions for real-time playback.

Offline stage: ISL dictionary video, SMPL-X pose detection, sparse keyframe selection, manual pose correction, corrected gloss pose dictionary.
(a) Offline dictionary construction, done once per gloss.
Online stage: sentence, gloss sequence, dictionary lookup, pose transfer onto the canonical SMPL-X avatar, interpolation between keyframes, animation.
(b) Online sentence synthesis, performed for every sentence in the browser.

Widget Dictionary

The Widget Dictionary is a 3D gloss pose dictionary for ISL built from real signers. Each isolated, gloss-aligned sign is stored as manually verified SMPL-X keyposes with corrected body, hand, and finger articulation, retargeted onto a canonical avatar so that signs are independent of the original signer. New glosses can be added at a one-time, linear cost per sign, making the dictionary scalable to the full ISLRTC vocabulary.

Sparsemanually corrected SMPL-X keyposes
Canonicalsigner-independent avatar
25 FPSnative playback rate
Body + HandsSMPL-X articulation
Dataset construction: monocular ISL dictionary videos, automatic SMPL-X estimation, sparse human correction, reusable gloss pose dictionary.
From monocular dictionary videos to a reusable 3D gloss pose dictionary.

Sparse Pose Correction with Gizmo

Monocular SMPL-X estimates often get finger curl and wrist orientation wrong, and in sign language such errors change the meaning of a sign. Gizmo shows the reference video next to the reconstructed avatar. The annotator selects a bone, adjusts its rotation about each local axis, and propagates corrected finger poses to other frames, so only a few keyposes per gloss need to be fixed.

Correction of the gloss 'office': reference video, automatic SMPL-X pose with a finger curl error, and the corrected pose.
Gloss "office": (a) reference video, (b) automatic SMPL-X pose with a finger curl error, (c) corrected pose.
The Gizmo annotation interface, with the reference sign video, the SMPL-X pose viewer, bone selection, per-axis rotation controls, camera presets and frame saving.
The Gizmo annotation interface.

Results

Study with Deaf ISL Signers

11 Deaf and hard-of-hearing ISL signers watched 34 avatar sentences.

88.8%sentences identified exactly (332/374, four-option task)
2.67 / 5understandability
2.61 / 5hand shape
3.0 / 5web demo usability

Participants also rated naturalness (2.48) and body posture (2.50). They stressed the importance of facial expressions and noted that some signs change form in context (e.g., GO in "he is going home"), which motivates our future work on context-dependent sign forms and non-manual markers.

Does Manual Correction Help?

Each avatar gloss video is embedded with Uni-Sign and ranked against all real signer videos by cosine similarity. A retrieval is correct at Top-k if the real video of the same gloss is among the k highest-ranked.

Avatar videosTop-1 (%)Top-3 (%)Top-5 (%)Top-10 (%)
Raw SMPL-X (no correction)10.1720.1727.3338.00
Pose corrected (Gizmo)35.1753.5062.0072.00

Real-Time Synthesis in the Browser

One-time preparation latency before playback and steady-state frame rate, over 10 runs per sentence length (cold start excluded).

Sentence lengthLatency (s)FPS
2 glosses0.12925.02
5 glosses0.29625.00
10 glosses0.54225.00
20 glosses1.41925.00
Right wrist rotation across the OFFICE to GO gloss boundary, showing smooth curves through the SLERP transition span.
Right-wrist rotation across the OFFICE→GO boundary: SLERP transitions keep motion smooth between glosses.

Conclusion

SignPlug shows that sentence-level ISL avatar animation can run on consumer hardware, entirely client side, without runtime neural inference, and plays back at the native 25 FPS of the source recordings. The current system covers manual articulators (body pose and handshape). Non-manual markers such as facial expressions, and context-dependent sign forms, are the next steps. We hope this pipeline helps the community build more expressive and widely deployable tools for ISL accessibility on the web.