Turning Guitar Recordings Into Playable Tabs That Capture Bends and Slides

The physical ambiguity of guitar

A guitar does not map pitch to one fixed place on the neck. It has six strings, and each string can be played open or stopped at a fret position. In the notation used for tablature, those positions run from an open string up to the 24th fret. Because of that layout, one pitch can often be produced in several different ways. The same note may sit on a low string or on a high string farther up. To a listener, the pitches can match. To a player, they are not the same gesture. Even a correct pitch sequence can hide several possible performances. The physical layout makes the ambiguity unavoidable.

This is why guitar transcription is harder than simply writing down notes. A score that lists only pitch and rhythm can be musically plausible while still being physically wrong. It may suggest a hand position that feels awkward or impossible in context. A guitarist chooses fingerings based on nearby notes, technique, string color, and the shape of the passage. Simple rules about playability are not enough to recover what the performer actually did. It matters whenever a recording is turned into playable notation. The ambiguity is therefore not just a labeling issue. It is part of how the instrument works.

Detecting notes in noisy recordings

Detecting notes in noisy recordings is the first hard step in turning guitar audio into a score. A raw recording contains pitch, timing, and loudness all mixed together. The system must separate each note from that mixture. It also has to work when the sound comes from a phone mic, a room with echo, or an electric pickup with distortion.

The note-detection stage uses a high-resolution neural network called a CRNN. It listens to a compressed picture of the sound, not the raw waveform. That picture is made at 16 kilohertz and shows 229 frequency bands over time. For every tiny slice of audio, the model guesses four things: whether a note starts, whether it ends, how active each pitch is, and how hard it was played. This lets it place notes with fine timing instead of snapping them to a coarse grid.

Training on clean studio audio alone would leave the model weak in real rooms. So the note detector was trained on four guitar datasets with different instruments and recording setups. This mix includes classical guitar, electric guitar, and direct-injection signals. The goal is to teach it that a guitar can sound very different while still playing the same notes.

During training, the system also learned to ignore common noise. Half of each batch was deliberately dirtied. A low-frequency rumble was removed with a high-pass filter set at 80 hertz. Then white noise, pink noise, or electrical hum was added. The signal-to-noise ratio varied between 25 and 45 decibels. This range covers background hiss that is annoying but not overwhelming.

Unlike piano-style cleaning, this method did not smear attacks with reverb or strip harmonics with aggressive equalization. Guitar notes often begin with a sharp pick strike. If that strike is blurred, the model can miss the note’s start time.

The system also throws away predicted notes shorter than 30 milliseconds. A real guitar note rarely lasts less than 50 milliseconds, even in fast passages of 15 notes per second. This rule cuts false blips caused by noise.

To test real-world robustness, two new noisy test sets were built from existing guitar recordings. Each clean clip was mixed with room echo taken from recorded spaces. The echoes came from places such as nature spots, sanctuaries, and venues. Some clips imitated phone or laptop microphones.

A note was counted as correct only if its start time and pitch were close enough to the true note. The allowed error was 50 milliseconds for timing and 50 cents for pitch. A cent is one hundredth of a semitone, so this tolerance is small.

On clean recordings, the detector was close to the best previous system on one benchmark but not clearly ahead everywhere. On other benchmarks it pulled far ahead. Its average score across four test sets was 81.35 percent. The next-best average was 74.68 percent.

The noisy results show why this matters. On a clean electric-guitar set, the detector scored 79.0 percent while the previous best scored 68.9 percent. On the noisy version of that same set, it still reached 76.8 percent. That drop is much smaller than the collapse seen in older systems.

This first stage sets the ceiling for everything after it. If a note is missed here, later stages cannot recover it. The noise-robust design therefore matters beyond accuracy charts. It makes transcription possible when recordings are imperfect.

Recognizing the gestures between notes

Guitar playing is not just a sequence of pitches. A slide, bend, or harmonic changes how a note feels and sounds. The classifier therefore treats these gestures as labels attached to notes. Each detected note receives one of nine technique classes. These are bend, hammer-on/pull-off, harmonics, kick drum, palm muting, picking/no technique, slide, snare drum, and vibrato.

For each note, the model listens to the audio from its start to its end. It turns that sound into a short list of musical measurements. Every 23 milliseconds, it records 40 timbre features, 40 loudness bands, and 12 pitch-color values. Together they form a 92-number description of the note. These descriptions are stretched or cut to about three seconds.

A small neural network then reads this time series. It uses convolutional layers to spot local sound patterns. A bidirectional memory layer follows the pattern over time. This matters because a bend and vibrato can look similar in one frame. Their difference is how the pitch moves as time passes.

The classifier was trained on five public guitar technique collections. They were merged into one set with a common nine-class label scheme. Class weights helped balance rare gestures against common ones.

On the held-out test set, it reached 97.4% weighted accuracy. Its macro F1 score was 95.9%. Every class had recall above 95%. Percussive gestures were easiest, with kick drum and snare drum at 99.5% recall. Vibrato was hardest, with an F1 score of 89.4%. It is often confused with bend because both involve moving pitch.

Older feedforward models treated each note as a fixed snapshot. That approach missed the timing cues that separate gestures. The classifier reached 95.9% macro F1 with only about 160,000 parameters. A comparable dense network needed 2.08 million parameters and scored 71.6%. Another convolutional baseline reached only 62.7% macro F1.

This means the system can mark expressive gestures with high reliability. It does not just list notes; it captures how they are played.

Assigning strings and frets from sound

Once the notes are known, each one still needs a place on the guitar. That place is a string and fret pair. Tablature uses that pair to tell a guitarist what to press. AudioFret solves this as a translation task. It takes symbolic pitch events from the earlier stage. It also listens to the raw audio around each note. For every note, it cuts out 200 milliseconds of mel-spectrogram near the onset. A small convolutional network turns that slice into an audio embedding. Those embeddings are placed before the MIDI token sequence in a T5-style encoder-decoder. The decoder then proposes string and fret tokens for the whole passage.

The model uses timbre to break ties. The same pitch can sound different on another string. String thickness, tension, and overtones change the spectrum. A symbolic-only model cannot hear that difference. AudioFret can. Its backbone was enlarged to 256 dimensions, six layers, eight attention heads, and about 15 million parameters. It was pre-trained on synthesized tablature data. Capo positions from 0 to 7 were added as conditioning tokens. Four tuning variants were also used: standard, half-step down, full-step down, and drop D. The model was then fine-tuned on real guitar recordings. The audio encoder was first trained as a string classifier before joint training.

At inference time, the system uses beam search with width 4. It enforces simple rules at each step. Every note must be followed by a valid duration token. A chord may not use the same string twice. Any chord needing more than five frets of span is masked out. That limit matches a realistic hand reach. With correct notes supplied as input, AudioFret reached an average Tab F1 of 71.8 percent across four zero-shot benchmarks. Pitch accuracy was 100 percent. A note counted as correct only when onset, pitch, and string all matched within tight tolerances.

Assembling a playable tablature score

The last step takes the pieces from earlier stages and joins them into one score. It receives note events, technique labels, string-and-fret choices, and a tempo estimate. These items are stored in an intermediate annotation file before being drawn as tablature. The tempo comes from BeatNet, a separate beat-tracking model.

First, timing is cleaned up. Onsets that fall within 30 milliseconds of one another are grouped together. This removes tiny timing wobbles left by human playing or recording noise. Each note start and length is then snapped to a sixteenth-note grid. The grid uses the estimated tempo as its clock. Notes that land on the same beat position become chords. If two notes collide on the same string, the longer one survives.

Next, the score is written to MusicXML, a standard file format for musical notation. Single-note techniques are attached directly to their note. These include bends, harmonics, vibrato, palm muting, drum hits, and plain picking. Two-note gestures need extra logic. Hammer-ons, pull-offs, and slides link one note to the next note on the same string. The system searches within a one-beat window for that partner note. If the user knows the capo position, tuning, or tempo, those values can be supplied. Otherwise, standard tuning and no capo are assumed.

The assembled score is then tested from raw audio to final tab. Across four zero-shot benchmarks, the full pipeline reaches a 54.08 percent tablature F1 score. That number counts a note as correct only when its start time, pitch, and string all match. A higher oracle test gives the string-and-fret model perfect notes first. It then averages 71.83 percent. The gap is about 17.75 percentage points. This loss shows how early timing or pitch mistakes can travel through the chain.

What the guitar still hides

A four-stage route now turns raw guitar sound into playable tablature. It does more than list notes. It marks bends, slides, harmonics, and percussive touches. It also chooses a string and fret for each note. That last step matters because one pitch can live in several places on the neck. The system listens to tone color as well as pitch. A thick low-string note sounds different from a thin high-string note. Using that clue helps place fingers where a player would likely put them. When all stages run together, a little more than half of the notes land correctly.

The result is useful beyond clean studio audio. Noisy recordings are handled better than before. Phone clips, room echo, and background hum no longer break the whole chain. That opens a door for everyday music learning. A student could upload a video and receive a rough but playable score. An archivist could preserve styles that live mostly in recordings. A composer could mine old tapes for ideas without transcribing by hand.

The obstacles are still real. The first stage misses unpitched sounds such as drum hits. If those sounds are not heard, later technique labels can fail. Each note also gets only one technique label. A bend with vibrato is a common gesture, but it cannot be fully described in the current output. Rhythm grouping uses fixed timing rules, so chords can sometimes split or merge incorrectly. Errors from early stages travel forward and cost accuracy.

The bigger picture is that guitar transcription has moved closer to performance notation. It no longer treats the instrument as a simple pitch machine. The next challenge is richer expression: layered techniques, live correction, and scores that adapt to different players. If a system can hear how a note was played, what will it take for it to understand why?