Hi Pd-people!

I'm currently working with a VR-games studio that possibly wants to create gameplay that would depend on the player doing this in sync to background music. I'm currently looking into Pd to see if this is something I could do within it.

Things that are on my wishlist;

1. Being able to analyze a audio file from a table to estimate tempo

2. Be able to differentiate between things such as a kickdrum and a lead guitar by using the likes of FFT or very steep bandpassing

3. Take this information and store it in a seperate array that has a size that matches a phasor cycle with the lenght of the audio file. (Does this request makes sense?)

I know that this inquiry is an immense one and I don't expect to be able to solve all of these things, if any. But if anyone feels like they could give any tips to point me in a direction that could possibly produce something close to my wishlist I would really love you.

Thank you for your time, loving these forums!

@Jocketor It's the musicians Holy Grail. Once all the instruments are mixed together the waveform is hard (or impossible..... which depends on the complexity) to decipher. Of course our brains do it easily, and so we think it is possible.
There are a few wav >> midi programs that make a good stab at this. But the results rarely have any resemblance to the original audio.

Is it possible that you could make your own tables / click tracks / instrument midi files for the music being used, or are the users going to use any music that comes to hand?
David.

@whale-av Thanks for replying!

Agreed, it is not a very simple thing to set up in Pd. However, since BPM-calculation of an audio file isn't terribly complex to get going in more traditional syntax (We've looked at rough prototypes of it in C#) i'm honestly surprised that I can't find more examples of it in Pd.

Things that I can get immediately upon loading an audio file to an array is basic info such as sample size and length of file in samples. Using these it should be fairly simple to get a length of the file in MS, which in turn should allow me to at least guesstimate a BPM. From there, being able to use FFT, bandpassing or maybe even fiddle~ to distinguish between different forms of instruments playing would just be a huge bonus. (Again, even FFT seems to be easier to set up in C# than what I've had success with in Pd.) For the sake of argument you could imagine that this would be used for a very basic rhythm game such as the likes of Guitar Hero.

If all we wanted to do was to beatmatch to pre-defined music tracks then we'd be set - i'd just use a form of a click track derived from the actual track like you suggested, write that to an array that spans the length of the song and then read back from it during playtime. Being able for users to load the own music would give such an increased value though.

We've been inspired by the VR game Audioshield, in which the BPM tracking on user-defined music works "fairly well".

Audioshield:

Again, thanks for engaging! Hopefully this could at least lead to some cool discussions

@Jocketor For music with a straight beat you could try using [bonk~], together with an object like [beat] from the maxlib library to estimate the tempo (not sure how it would do with more subtle rhythms though).

As for identifying the different sounds, there is an external library that has a lot of objects for timbre analysis http://williambrent.conflations.com/pages/research.html#timbreID
I'm not sure if it'd do what you want but it's worth looking into.

It is indeed a very difficult thing to achieve and I expect that whatever solution/workaround is hardly going to be very reliable with very different sources.

In case anyone is interested in progress - I'm still trying to set up a FFT-chain that i'm happy with. Different articles online all seem to have their own steps with regards to things like normalizing the signal, and I feel like i'm not all after the re-synthesis part of ifft/rifft as i'm only looking to analyze audio from an array, not do any processing on it.

While I am doing FFT on a wavefile loaded into an array and visualizing the FFT:d signal in a new array, i'm still at a loss on how to extract the data that I want. Ideally i could somehow group the indexes, say per 100 blocks, so around 5 groups if i'm using a blocksize of 512, and look at how they are outputting using bonk~ or something of the sort. Right now i'm using either tabdump or tabread from the FFT-array, coupled with unpack to get individual indexes, but since there's 512 of them it's just not practical. Overall i'm having trouble finding any examples online on where FFT is used to get control-value information rather than as a part of the audio signal.

I'm still up for more discussion if anyone of you out there are intruiged!

@Jocketor spectral envelope pdf

This has some interesting information that you could find useful, as well as screenshots of pd patches to find the spectral envelope of each block of an audio signal.

@weightless Thanks! I've read that article, and while it's the most promising publication for helping me figure out what I want, the author uses a few abstractions called poltorec~ and rectopol~ that they seemingly wrote themselves. To be honest I couldn't make sense of it even though the article mentions a workaround. I might just have to spend a few hours dissecting that article again, because the things in there seem very close to what i'm pining for here.

Again, huge thanks!

@Jocketor Ah, I misread them for [cartopol~] and [poltocart~]. I'm not sure what the differences are or how to go about recreating them. The article has the author's email so you could try asking him for the patches.

Also in the timbreID library I mentioned earlier there is a [cepstrum~] object which does the same, but perhaps you are looking at doing with no externals.

@Jocketor I almost always bow to katja for fft......... and here is a great read and a patch at the bottom of the page.......
http://katjaas.nl/helmholtz/helmholtz.html might well be useful.
David.

@whale-av Katjaa is indeed a Pd mastermind. I had not read that article before, so thanks for that! Great stuff.

I've actually made some progress in my endeavours. After loading a wavfile to an array, running it through the FFT analysis and writing the results to a new array i'm using tabdump together with unpack to get out my individual values containing the amplitude for seperate indexes in any given moment. What needs to be done is to store these values in a sequential order to create somewhat of a timeline, either by writing them down to a new array or by using list, i'm not sure yet. I'm actually planning to create a new thread here discussing the idea of writing to an array while continuously moving forward through the indexes in a set timefreame, to create a timeline.

Ergo; I've abandoned the idea of accurately deciding on a BPM by using FFT to analyse a wavfile. I'm sure it can be done, and i've some progress on the subject, but it's very hard to do reliably. For my purposes I also find that I can do just aswell with using FFT to determine when "events" happen in different ranges of a spectrum, and as long as I can store these moments in time to an array/list I will be able to create what I set out to do.

Always open for more discussion on the subject, so big thanks for everyone who got involved!