A depth camera can tell Bharatanatyam dances apart by reading poses and movement together

How a dance is broken into poses and steps

Bharatanatyam is built from small, repeatable units called Adavus, and each unit has a strict rhythm of stillness and travel. A dancer holds a key posture, moves through a connecting motion, settles into another posture, then travels again. The recognition pipeline starts by turning that performance into the same kind of list. After cleaning out bad or incomplete recordings, the dataset contains 13 Adavu types and 51 variations, drawn from a larger set of 15 types and 58 variations. It also uses 182 key-posture classes, which are the named stationary shapes that can appear in the dance.

Each video is first reduced to grayscale frames, and a depth-based mask removes everything except the dancer. An annotation file then marks which frame ranges belong to postures and which belong to motions. In practice, a long video is sliced into intervals. One interval holds a recognizable shape, while another captures limbs or weight in change. The order matters because an Adavu is defined by that ordered list of stills and transitions.

A posture segment usually contains several nearly identical frames, so any one frame can stand for the same held pose. That creates a small choice at every stop in the dance. If a segment has ten usable frames, the sequence builder may pick different representatives without changing the underlying Adavu. The number of possible alternating sequences is limited by the shortest posture segment in that Adavu, because each segment must contribute at least one representative frame. The first ready posture before the dance begins is dropped from the list, so the sequence starts with the actual movement pattern rather than a neutral starting position.

Motions are treated as separate categories too, but only when there is enough recorded evidence for them. Classes with fewer than 12 samples are set aside to avoid training on too little data. With that rule in place, 54 motion classes remain usable out of 334 total. This threshold matters because dance motions vary in length and many are easy to confuse. By keeping only better-supported motion classes, the system builds a cleaner inventory of steps before it tries to read them.

Reading the stationary postures from single frames

A Bharatanatyam dancer can hold a pose for several frames, and the system treats any one of those still moments as enough to name the posture. Before classification, each video frame is turned into grayscale, while depth information from the Kinect removes everything behind the performer so that only the body remains in view. The network then looks at this cleaned image directly, without needing a sequence of frames or a separate skeleton tracker.

The classifier is deliberately small compared with large general purpose image models. It has two convolution layers, first with 32 channels and then with 64, followed by three fully connected layers. Layer normalization, max pooling, dropout, and Leaky ReLU activations help it learn stable features while reducing the chance that some neurons stop responding. Training stops after 15 epochs because accuracy began to overfit beyond that point. The final softmax layer produces a score for each of 182 posture classes, so every frame is judged against the full set of stationary key postures rather than a handful of examples.

On the test set, the network got 8,895 out of 8,943 frames correct, an accuracy of 99.46 percent. That means almost every single representative frame was assigned to the right posture. Fifteen classes were recognized perfectly, and most others scored above 95 percent. The failures are mostly visual twins. Two standing poses differ only by whether a foot is flat or raised onto the toes. One of those pairs reached about 57 percent accuracy because six test frames were swapped with its lookalike. A small right leg stretch can resemble a bent knee posture, and when the Kinect misses a hand raise, that pose can drift toward another class.

Tracking moving segments when costumes hide the body

Once a stretch of video has been marked as movement, the task is to decide which named Bharatanatyam motion it contains. This part of the problem is harder than naming a frozen pose because a dancer’s body keeps changing shape while the skirt, sleeves and jewelry can cover large parts of the limbs. The recognition process therefore does not try to follow every joint through time. Instead, it builds one summary image for each moving segment.

The summary, called a motion history image, starts as a blank picture and fills in as the dancer moves. For each new frame, the outline of the dancer is found and compared with the outline from the previous frame. The difference between those two outlines is added to the accumulating image. After the whole segment has been processed, areas of change show where the body moved, while empty regions mark places that stayed still.

Before classification, this large image is reduced from 480 by 640 pixels to 120 by 160 pixels. The smaller picture is then turned into a long list of numbers that describes the direction and strength of edges in different parts of the image. That list has 9576 values, and a support vector machine uses it to choose the most likely motion label.

Because many dance movements appear only rarely in the recordings, the pipeline keeps only those with at least twelve examples. That rule leaves 54 usable motion classes out of 334 total. With that smaller set, the classifier reached about 84 percent accuracy overall. The mistakes were spread thinly across the classes, because twenty-two motion classes were recognized perfectly, twenty had a single error, and twelve had two or three errors.

When the requirement was raised to at least twenty-five examples per class, only twelve motion classes remained, but accuracy rose to 95.58 percent. That jump shows that the main limit is not a weak classifier; it is the small amount of training material available for many movements.

The most telling errors come from the parts of the body that a costume can hide or blur. A rise and fall movement may be confused with a weight shift plus foot tap, especially when the legs are covered by fabric. Mirror-image versions of similar movements can also be swapped when the camera sees only one side of the body. A half-turn around the torso is not always captured well in the summary image, so it can look like a simple foot tap. Very small heel-toe shifts are almost invisible unless the hands move slightly.

Matching the full sequence to name an adavu

A Bharatanatyam adavu is more than a collection of pretty poses. It is an ordered pattern in which still key postures and the movements between them alternate. To name the adavu, the system does not ask whether one frame looks like a known pose or one clip looks like a known motion. It builds a full label sequence from the video and compares that sequence with stored patterns for candidate adavus.

Each predicted posture and motion becomes an item in this list. The comparison uses edit distance, a measure of how many substitutions, insertions, or deletions would be needed to turn one sequence into another. If the predicted list is close to a stored pattern, that adavu wins. In tests on 1,645 sequences, this whole dance matching reached 98% accuracy. That result matters because individual motion labels were less reliable, around 84%, yet the longer sequence still pointed to the right choreographic unit most of the time.

The system also has to handle a small ambiguity inside each posture segment. A stationary pose may last for several frames, and any one frame can stand in for that pose. As a result, a single video can produce more than one plausible alternating list. The number of candidates is limited by the shortest posture segment in the adavu, so the search stays manageable. The initial ready posture before the dance begins is left out, because it belongs to preparation rather than the named pattern.

This sequence based approach explains why adding motions changes the outcome. Earlier systems that used only postures scored 77.51% on the same test set. Their failures were concentrated in one family of adavus whose posture order was identical, often a repeated single pose. Postures alone could not separate those variations. When the two families from an earlier comparison were tested separately, accuracy became 100%, which shows that the remaining errors came from genuinely indistinguishable posture lists rather than from weak image reading.

A dance becomes a readable sequence

The strongest result is that Bharatanatyam can be recognised as an ordered performance, rather than a collection of separate images. A camera captures the dancer, still frames are read for posture, moving segments are compressed into motion histories, and the resulting labels are compared with known Adavu patterns. When the full order matches closely enough, the dance receives its name. Whole-dance recognition reaches 98 percent across a large part of the standard repertoire, which is impressive because individual motions are harder to classify than frozen postures.

The broader promise is preservation and teaching. A library of recorded Adavus can become a searchable archive, where students compare their own movement with canonical sequences and teachers can point to specific deviations. The method also suggests that other classical dances might be handled in the same way once enough labelled examples exist. That would turn visual performance into data that can be studied, corrected, and passed on without relying only on memory.

The obstacles are equally clear. Many motions appear too rarely for reliable training, costumes hide parts of the body, and some postures differ by a foot position or hand angle that sensors miss. The current setup also depends on controlled recording conditions, and timing performance has not yet been reported in detail. More data, better occlusion handling, and faster inference would be needed before such tools move from laboratory to classroom.

If a dance can be read as a sequence, what else might become legible to machines?