Hello,

This post is about a plan to develop a software library with realtime analysis routines, to be used in Pd and possibly other realtime dsp frameworks. The idea was loosely discussed with Jwif in another thread.

My concept of RAP (realtime analysis project) would be based on the following: an audio stream has features which can be found from the waveform itself, features which can be used to reorganize that waveform in a sensible and musically pleasing way. A realtime cut and paste concept, applied to an incoming audio stream, rather than a file from disk. I apply this concept already in Instant Decomposer and Slice//Jockey. Jwif does so in his RjDj apps.

Analysis procedures for realtime cut and paste must be very precise, robust, and low-latency. Existing signal analysis classes in Pd (bonk~, fiddle~, sigmund~) are of good quality, but they can not give very precise locations of where things happen in a signal.

For this reason, I had earlier developed classes slicerec~ and sliceplay~, which operate with sample-precise attack cuepoints to indexes in a signal buffer. With this construction, incoming audio is spliced so fast and precise that rhythms can be created on the fly without sloppiness in the timing. slicerec~'s attack detector is however not so sensitive for all signal types as bonk~.

The concept of realtime splicing could be expanded beyond the obvious cut-on-attack approach, if period tracking is taken into account. If period length is known, periods can be cut, copied and pasted as well. This is a finer grain of signal reorganization, but still based on information of the signal itself. Like complete notes, periods can be stretched or shrunk to change their spectral information. You could use this for pitch shifting, realtime stretching or formant manipulation. Together, transient- and pitch detection give sufficient information to do very weird things with a signal, without completely loosing the original character.

The main info's needed from a signal are: envelope, attack point, transient length, and in case of periodicity the period length.

The RAP library would be a set of analysis routines written in C, which can be used in different combinations to create RAP-based Pd classes. The C routines should be of high quality and efficiency. For pitch tracking, procedures in [helmholtz~] are probably the best option (http://www.katjaas.nl/helmholtz/helmholtz.html). For attack detection, procedures in [bonk~] can be a guideline.

RAP-based Pd classes could be, for example:

a class [attackpoint~], taking an audio stream and an index stream, and producing cuepoint messages

a better [slicerec~]

a PSOLA pitch shifter

The above ideas sketch a very rough impression of what RAP could be. To design it's structure and implementation will be a delicate and time-consuming process. Do not expect it to be ready within days or weeks. If we take time to work out a decent structure, it will be possible to use the library for all sorts of purposes, and extend it over time. Like Pd, it should be BSD licensed to avoid limitations in the way how we can use it.

Katja

@katjav said:

Hello,

The RAP library would be a set of analysis routines written in C, which can be used in different combinations to create RAP-based Pd classes. The C routines should be of high quality and efficiency. For pitch tracking, procedures in [helmholtz~] are probably the best option (http://www.katjaas.nl/helmholtz/helmholtz.html). For attack detection, procedures in [bonk~] can be a guideline.

Mmm .. very interesting
Going back to the topic of pitch tracking, what would be the purpose? Is it your intentionfor this library to be only for monophonic real-time analysis?

I was reading one of your posts from a couple of weeks ago (about pitch-tracking), and you mentioned about downsampling by a factor of 8 before doing anything else. Sounds good to me.

The downsampling available in Pd at present does not include anti-aliasing (it justs drops samples) .. a linear interpolation is sufficient for analysis, I would say .. 0.25 0.5 0.25 for instance

Sounds like an interesting project .. monophonic or polyphonic?

@moog1 said:

Is it your intentionfor this library to be only for monophonic real-time analysis?

Good question. It brings me to an aspect of RAP which I forgot to mention. It seems to me that a new analysis library should only collect routines which have such good quality and efficiency that they can be used for realtime applications of (semi-)professional grade: in live performance setups, in music apps for portable devices, etc. For monophonic pitch tracking and transient detection, this is realizable with techniques available today, it is only a matter of meticulous implementation.

Polyphonic pitch tracking is not (yet) in this category, unfortunately. Nick Collin's PolyPitch object for SuperCollider reportedly consumes about 50% CPU time, and this is probably the state of the art as far as realtime / open source is concerned. It may need a few years of development before such a process is ready for a production library, if possible at all.

Moog1, I don't know how you think about it, but for me the focus is on solid technique rather than the latest academic inventions and hypotheses. That is just my level of operation. Many analysis projects are optimistically started, but abandoned while still in experimental state. As a result, we still do not have monophonic pitch detection and -shifting of satisfactory quality in Pd, while it is available in commercial hardware and software since decades. Therefore I believe that RAP should be compact and concentrate on refinement of proven procedures, so we'd have stuff to go on stage with.

@moog1 said:

The downsampling available in Pd at present does not include anti-aliasing (it justs drops samples) .. a linear interpolation is sufficient for analysis, I would say .. 0.25 0.5 0.25 for instance

Indeed for analysis the filter requirements need not be so strict as for audio output. I suggested downsampling by factor 8 earlier, but when testing [helmholtz~], I found that factor 4 would be the maximum for the full fundamental range of most instruments, and factor 8 would be possible for bass instruments. The filter coefficients you mention, is this per octave or for all octaves? With [basspitch~] you go down even further to factor 16. But do you run the filter more than once on the input signal, or not? This was the most dense and enigmatic part in your code, I could not follow it completely. Anyway, it would be best to set up objective tests with different FIR kernels and check where the margins are. We could modify your impulse response patch to make a FIR kernel generator tool.

Katja

Hi Katja,

I have found sigmund~ to track note attack better than bonk~ (particularly concerning string motion). Wondering if you have tried sigmund~ with the "note" argument?

Ricky

The filter coefficients you mention, is this per octave or for all octaves? With [basspitch~] you go down even further to factor 16. But do you run the filter more than once on the input signal, or not?

Right .. here goes..

for (i=0; i<4; i++) {
        in_p = out_p = out;
        while( in_p < out+len ) {
            samp = x->filt_state[i] + (*in_p++ * 0.5);
            x->filt_state[i] = *in_p++ * 0.25;
            *out_p++ = samp + x->filt_state[i];
        }
        len=len/2;
}
'out' is  a pointer to a buffer. len is initially equal to block size. the while loop downsamples by a factor of 2 with anti-aliasing, so halving the num of samples in a block. This is repeated in the 'for loop' 4 times, making an overall factor of 16, with the output being pointed to by 'out' of size len/16.

The coeffs. are just linearly interpolating the 2  would-be dropped samples as well (0.25) .. that's all..
Understand now? :)[/i][/i][/i]

@ricky said:

I have found sigmund~ to track note attack better than bonk~

I remember you mentioning that on the forum. You're pointing to an interesting aspect of note detection. Some notes are marked by a pronounced transient, but it isn't always the case. A change of pitch can sometimes better indicate a note start. [bonk~] checks for shifts in spectral content, but only at the level of a few filter banks. [sigmund~] of course has a much better spectral resolution.

An integrated library including pitch and attack detection could combine all the info to make up the abstracted data you need. Here's an example: a pronounced transient can often be detected within a few milliseconds, much shorter than the analysis frame of a pitch tracker. A transient hit could then temporarily speed up the overlap of the pitch tracker, so you would have pitch info as soon as possible after the transient phase has ended. Vice versa, pitch info can indicate a note start even if doesn't show a sudden RMS amplitude rise.

Katja

@moog1 said:

The coeffs. are just linearly interpolating the 2 would-be dropped samples as well (0.25) .. that's all..
Understand now?

Wow, a filtering decimator in not even ten lines of code.

Yeah, I'm starting to understand it, slowly. So you apply this filter four times, one for each of four downsampling stages, and effectively keeping 1 out of each 16 input samples.

Later on you analyze only the lowest 44 bins in the 512 pt spectrum, so that also means a form of filtering. For an autocorrelation-based method, you can't leave fft bins out. I think that much longer FIR kernels would be needed in that case. We have to find out.

Katja

@katjav said:

Wow, a filtering decimator in not even ten lines of code.

Thanks

I was being a bit lazy, really..
This is probably faster/better:-

t_sample samp, state;
for (i=0; i<4; i++) {
        in_p = out_p = out;
        state = x->filt_state[i];
        while( in_p < out+len ) {
            samp = state + (*in_p++ * 0.5);
            state = *in_p++ * 0.25;
            *out_p++ = samp + state;
        }
        x->filt_state[i] = state;
        len=len/2;
}
..only saving the state for continuity when necessary .. probably more understandable, as well ;)

[quote]So you apply this filter four times, one for each of four downsampling stages, and effectively keeping 1 out of each 16 input samples.[/quote]That's it!

[quote]Later on you analyze only the lowest 44 bins in the 512 pt spectrum, so that also means a form of filtering.[/quote]Indeed !

[quote]For an autocorrelation-based method, you can't leave fft bins out. I think that much longer FIR kernels would be needed in that case. We have to find out.[/quote]Hmm .. It depends what you're trying to do .. as we've aready established, you can only determine a monophonic tone with autocorrelation, or know what you're looking for.
I mean, what's the point of doing all the filtering, and then throwing away the results :-D

One or the other .. helmholtz~ is excellent for monophonic tracking .. I doubt whether it can be beaten[/i][/i]

Here is a impression of the operations needed to get a _single_ pitch report over 1024 samples using SNAC, hold your breath:

• fft: order N * log(N) = order 1024 * 10 = order 10240 operations
• computer power spectrum: 1024 multiplications
• ifft: again order 10240 operations
• normalisation: 870 * 2 multiplications and 870 divisions
• peak interpolation: dependent on number of peaks

Tens of thousands of operations for one single datum! When you look at it from this perspective, it seems ridiculous. On the other hand, it is a few dozen operations per sample. That is not so bad for an analysis routine. Actually, SNAC is one of the most efficient pitch tracking methods in existence, because it does not reduce the effective analysis length by Hann windowing.

Now continuing the estimation in case we decimate:

Say we would use a ~20 pt FIR filter for decimation, a common length used for this purpose. Half of the calculations to compute the inner product are redundant and can be skipped in an efficient implementation. That would leave 10 multiplications per sample. In the next stage of downsampling you have half of the samples left to filter, and the number of multiplications is 5 per original sample point. This done, the number of samples is reduced to 1 in 4 for the price of 15 multiplications per original sample. The SNAC can thus be done at 1/4th of it's original price. Say it was 40 operations originally, now 10. Summed with the 15 filter multiplications, the result is: order 25 operations per original sample.

Not a dramatic performance gain, it seems. But do not forget: you'd want to overlap analysis frames to improve the message output rate. With 4 times overlap, we are again at order 40 operations per original sample for the SNACs, plus 15 for the filter. Considering the fact that filtering improves the quality of autocorrelation, you'd implement a filter anyway.

According to my current experience, this is about the best efficiency you can get for monophonic pitch tracking. If the filter kernel can be shorter than proposed here, you might gain a bit on performance but it can't make a big difference. The point with analysis is, if it's used for processing, the message output rate must be sufficient to give a smooth processing result. Therefore it can never be really cheap.

[helmholtz~] does 0.6% per instance with default settings and no downsampling on my 2 GHz Core2Duo. Moog, can you report what it does on your PentiumII?

Katja

Downloaded helmholtz~ and will test out over the weekend. An exciting development.

I think that I'm out of my depth .. I will watch with interest how the project proceeds

I think that I understand your approach .. you intend to analyse the various characteristics, and isolate the periodic waveforms so as to get an accurate pitch measurement with limited chaos..

Is that right?

PS. I'm not sure what you mean by 0.6% per instance, but I can assure you that [helmholtz~] performs very well, with no excess cpu load..

I'm not trying to criticise, only be helpful

@moog1 said:

I'm not sure what you mean by 0.6% per instance

Ah stupid me, I meant to write 0.6% CPU time. An object's CPU load is best found like so: instantiate (for example) 50 objects of one type on a patch and find the CPU load difference compared with Pd's idle load, then divide by the number of objects. Depending on the object type, CPU load may vary with input type.

Moog1, I appreciate your comments a lot, it is through evaluation of each other's work that our (shared) knowledge increases.

Katja

I've done a few impressions of how an integrated analysis library could be set up. I used Pd to draw the flow charts. In the end, it must all be written in C to get a good integration.

Katja

@katjav said:

..I meant to write 0.6% CPU time.
Katja

Using the built-in load meter (from the Media menu), I get about 50% cpu load for 10 instances, and 122% cpu load for 20 instances.

As 122% isn't possible, I'm not sure what that means :\

However, from trusty old 'top', I get 77% cpu load for 20 instances, against 1% for nought.
About 3.75% per instance .. hope that helps

@moog1 said:

About 3.75% per instance .. hope that helps

Thanks, that helps. 3.75 / 0.6 = a factor 6.25 of performance difference between our computers. And my computer is not even the newest. So there may be a factor 10 performance difference between hardwares used for Pd. Meaning that optimization is an issue, if we want to make Pd code work fine for everyone. I'll keep that firmly in mind.

Katja