Kyutai Releases MuScriptor: An Open-Weight Decoder-Solely Transformer for Multi-Instrument Music Transcription to MIDI

Automated Music Transcription (AMT) converts an audio recording into symbolic notes, often MIDI. Single-instrument transcription already works fairly nicely. Nonetheless, transcribing a full multi-instrument combine stays tough. Kyutai and Mirelo staff now launch MuScriptor to shut that hole. It’s an open-weight mannequin educated on actual, multi-instrument recordings throughout many genres.

This text explains how MuScriptor works, what the benchmarks present, and tips on how to run it.

What’s MuScriptor?

At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a brief audio section. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In impact, transcription turns into a language-modeling job, following the MT3 tokenization scheme.

The discharge ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and giant (1.4B). The inference code makes use of the MIT license. The weights use CC BY-NC 4.0, so business use is restricted.

How the Three-Stage Pipeline Works

MuScriptor’s important thought is knowledge, not structure. Accordingly, coaching strikes by means of three phases, and every builds on the final.

1. Pre-training makes use of D_Synth, roughly 1.45M MIDI information. An on-the-fly pipeline synthesizes them throughout coaching. Augmentations embody pitch shifting, tempo adjustments, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
2. Tremendous-tuning makes use of D_Actual, an inner set of 170,000 recordings. Collectively they whole greater than 11,000 hours with aligned observe annotations. Most alignments come from audio-symbolic synchronization utilizing interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a most time-dilation issue.
3. Reinforcement studying post-training makes use of D_RL, 300 manually verified tracks. The staff applies a GRPO-like technique combining REINFORCE with group-relative benefit normalization. The reward sums three F-scores: onset, body, and offset. In consequence, the mannequin learns to favor cleaner transcriptions.

Efficiency

For analysis, the analysis staff use D_Check, 372 held-out tracks with correct annotations. They report instrument-agnostic metrics from the mir_eval library. Amongst them, Multi F1 is strictest, because it additionally requires the proper instrument.

The desk beneath traces every coaching stage towards the YourMT3+ baseline, utilizing the big (~1.3B) mannequin.

Clearly, each stage improves outcomes, and actual knowledge issues most. Artificial-only coaching reaches aggressive body F1 however weak onset and multi scores. Including D_Actual then lifts all metrics by roughly 20 factors. Lastly, RL post-training reduces false negatives and sharpens onset timing.

Cross-dataset assessments level the identical method. For instance, body F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset keep decrease on exhausting kinds like chorals.

Getting Began

Set up takes one command, and inference streams observe occasions instantly.

# pip set up muscriptor   (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel

# Downloads the default "medium" variant (additionally accepts "small" / "giant")
mannequin = TranscriptionModel.load_model()

# Stream observe occasions; optionally situation on identified devices
for occasion in mannequin.transcribe("audio.wav", devices=["acoustic_piano", "drums"]):
    print(occasion)   # NoteStartEvent / NoteEndEvent / ProgressEvent

# Or write a MIDI file instantly
Path("out.mid").write_bytes(mannequin.transcribe_to_midi("audio.wav"))

For the launched fashions, hold cfg_coef at 1, since they’re already RL post-trained. Moreover, uvx muscriptor serve launches a browser net UI with a dwell piano roll.

Use Circumstances with Examples

As a result of the output is normal MIDI, many workflows open up:

• Producers can extract a MIDI bassline from a mixture, then re-voice it in a DAW.
• Musicologists can convert historic recordings into editable scores for evaluation.
• MIR researchers can feed transcriptions into chord or key recognition methods.
• Educators can construct observe instruments exhibiting a dwell piano roll throughout playback.
• Builders can transcribe solely drums by passing instrument conditioning.

Strengths and Weaknesses

Strengths:

• Skilled on 170k actual recordings spanning classical to heavy steel.
• Open weights plus MIT-licensed inference code, in three measurement variants.
• Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D_Check.
• Instrument conditioning customizes output and stabilizes cross-segment predictions.
• A streaming API emits observe occasions and MIDI, alongside a browser net UI.

Weaknesses:

• Weights are CC BY-NC 4.0, so business deployment is restricted.
• The tokenizer drops velocity and can’t symbolize overlapping same-pitch, same-instrument notes.
• Onset and offset accuracy keep decrease on chorals and comparable kinds.
• The big mannequin desires a GPU for sensible pace.
• The 5-second section measurement limits long-range context and inference pace.

Take a look at the Paper, GitHub Repo and Mannequin Weights. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you possibly can be a part of us on telegram as nicely.

Must accomplice with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and many others.? Join with us