MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) has developed a deep-learning system that can watch (yes, watch - read on) a video of a musical performance and isolate the sounds of specific instruments.
The ‘PixelPlayer' system was trained on more than 60 hours of videos, and is now capable of identifying instruments ‘at pixel level' after viewing a video just one time. It can then extract the sounds associated with those instruments and play them back, without any other noise.
At the moment PixelPlayer can identify sounds from more than 20 instruments, which could be increased with more training data.
The system uses three neural networks: one to analyse the visuals of the video; one to analyse the audio; and a third ‘synthesiser', which associates specific pixels with specific soundwaves to separate the different sounds.
So far we have only seen videos of two instruments being played at once, so we do wonder how the system would handle a larger-scale performance like an orchestra.
Lead author Hang Zhao said that the system may have trouble handling subtle differences between subclasses of instruments (such as an alto sax versus a tenor).
The system can adjust the volume of individual instruments after the fact, which has implications for cleaning up old recordings, or previewing how certain instruments sound in a new composition.
A similar system could even be used on robots, to better understand environmental sounds made by animals or vehicles.
In the past, attempts to isolate sounds have focused on audio alone, which MIT says ‘often requires extensive human labeling'; but PixelPlayer's reliance on vision makes labelling unnecessary.
The system locates the image regions that produce sounds (implying that concert recordings would be a much more difficult proposition; we have asked MIT about this), then separates the input sounds into a set of components that represent the sound from each pixel.
The downside to this ‘self-supervised' deep learning is that MIT doesn't exactly understand how PixelPlayer learns which instruments make which sounds.
Microsoft receives a 30 per cent cut of all purchases on the Xbox digital store
Credit card thieves used Apple ID accounts to buy and sell virtual currency for Clash of Clans and Clash Royale and Marvel Contest of Champions
$5.1bn fine further evidence that the EU is anti-US, claims Trump
New cable will connect Virginia to France