Turning audio to MIDI

Have you ever listened to a song you loved and wished you could get the notes to play it yourself? Our new audio-to-MIDI model lets you do exactly that. Give it a song, and it gives you back the notes, separated into a track for each instrument: voice, drums, bass, keys, and more.

Today, together with Kyutai, we are open-sourcing this model and the research behind it. We've also trained an improved version, on even more data, which you can test with our free Audio-to-MIDI product in Mirelo Studio. It transcribes your full mix from any WAV or MP3. From there, you can get a full music sheet to learn an instrument part, or export any track as MIDI and take it into your DAW to create something new. Go check it out!

• Product overview: Audio-to-MIDI
• Try it free in Mirelo Studio
• Model weights on Hugging Face

• GitHub repo

• ArXiv paper

What you get

Our model is built for the full mix use case. Upload a recording and it gives you:

• Automatic instrument detection. The model finds each part in the mix and labels it (voice, drums, electric bass, synth lead, and so on).
• A separate MIDI track per instrument. Every detected part comes back as its own track on a piano roll, where you can review the transcription and adjust any notes you want to change before you export.
• Musical context. As shown in our Audio-to-MIDI tool, applications built on the model can go further and detect chords, key, and tempo, giving you the full harmonic picture.
• One-click export. Solo, mute, or download any track as MIDI and drop it into your DAW.

Why we're excited

Audio-to-MIDI tools already exist, and some are good. But almost all handle one sound at a time: a single vocal, a solo piano, one guitar take. Real music is denser, layering vocals, drums, bass, and more all at once, often buried under effects. That density is where existing models break down, and why turning a full song into usable MIDI has stayed a hard problem. That is the gap our model closes: it takes the full mix and transcribes every instrument at once.

Music transcription as language modeling

In its most basic form, music transcription involves taking the waveform of a music audio recording and extracting the start and end times of all the notes played within. In the multi-instrument case, we are also interested in knowing the exact instrument that played a particular note. The result of such a transcription is usually visualized using a piano roll representation, where the x-axis corresponds to time, the y-axis corresponds to pitch, and colored boxes indicate the notes played by different instruments.

Our model, similar to prior work [MT3], represents such a piano roll as a 1-dimensional sequence of discrete tokens representing pitch, timing, and instrument type. This representation is similar to the popular MIDI standard used for encoding note sequences in digital instruments.

The model predicts this token sequence using a straightforward, decoder-only transformer backbone, thus performing music transcription as language modeling. The model accepts a mel-spectrogram representing a 5-second audio chunk as prefix conditioning and autoregressively generates a stream of transcription tokens until an EOS token is obtained.

Synthetic pre-training, RL post-training

Music transcription as language modeling has already been explored in prior work [MT3, YMT3]. So why then has effective multi-instrument transcription remained so elusive?

A primary bottleneck in building multi-instrument AMT systems is the scarcity of music audio with aligned note annotations. Such data can be obtained relatively easily for specific instruments like piano [MAESTRO], but is much harder to create in multi-instrument scenarios. Synthetic music audio, which can be generated at scale, is a promising alternative. However, although previous work does utilize synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings.

For our model, we utilize synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning, leading to a combination of three training datasets:

• DSynth: We collect a dataset of roughly 1.45 million MIDI files across different genres (with a focus on pop and Western classical music). In order to utilize this data for transcription, we develop an on-the-fly synthesis pipeline using varying instruments, keys, tempos and tunings.
• DReal: To complement our synthetic dataset DSynth and to investigate the impact of different data types, we also utilize an internal dataset of 170 000 real music audio recordings (totalling over 11 000 hours) with aligned note annotations.
• DRL: Finally, we curate 300 tracks from DReal and apply a GRPO-like algorithm to our model to align it with this small dataset of high-quality transcriptions.

We pre-train the model on DSynth, fine-tune on DReal and post-train on DRL. We observe that each training stage improves results over the previous one. In particular, while the model trained exclusively on synthetic data is already competitive with the prior state-of-the-art AMT baseline [YMT3] in terms of frame F1 score, we note that fine-tuning on DReal is essential for performance and improves all metrics by roughly 20 percentage points. In addition, the reinforcement learning phase on the high-quality dataset DRL further improves results, leading to our best overall model.

In the paper accompanying the release we provide further analyses on the effect of synthetic pre-training as well as outline our optional instrument conditioning mechanism, which allows for customizing transcription outputs according to specific instrumentations.

Open weights for researchers, an app for musicians

For all details, check out our paper on arXiv.

To aid future research, the inference code is on GitHub and the model weights are on Hugging Face.

And the improved, even more accurate version is live now — see the Audio-to-MIDI product page or try it free in Mirelo Studio.

To cite this work, please use the following reference:

@inproceedings{muscriptor2026,
title     = {MuScriptor: An Open Model for Multi-Instrument Music Transcription},
author    = {Rouard, Simon and Krause, Michael and Roebel, Axel and
Simon-Gabriel, Carl-Johann and D{\'e}fossez, Alexandre},
year      = {2026},
note      = {Mirelo AI, Kyutai, IRCAM}
}

References

[MT3]      J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel, "MT3: multi-task multitrack music transcription," in ICLR, 2022.
[YMT3]     S. Chang, E. Benetos, H. Kirchhoff, and S. Dixon, "YourMT3+: multi-instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation" in MLSP, 2024.
[MAESTRO]  C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, "Enabling factorized piano music modeling and generation with the MAESTRO dataset," in ICLR, 2019.