@Cuinjune Hi. Do you know if that is possible? They do that with Shadertoy and it seems also possible with Open Frameworks.

The audio needs to be transformed to a 2D Texture:
“The FFT signal, which is 512 pixels/frequencies long, gets normalized to 0…1 and mapped to 0…255. The wave form, which is also 512 pixels/smapled long, gets renormalized too from -16387…16384 to 0…1. FFT goes in the first row, waveform in the second row. So this is a 512x2 gray scale 8 bit texture.”

https://forum.openframeworks.cc/t/passing-fft-audio-data-into-a-shader-as-a-texture2d-object-shadertoy/13756

Here are some examples:
https://www.shadertoy.com/view/Xds3Rr
https://www.vertexshaderart.com/art/TYoTaksHA6DWsP4aD
https://www.vertexshaderart.com/art/PponkZdtktgJHhipC

This way I can create the 2d array described above if i am right (not totally sure).
Also not sure if I would need the real or the imaginary fft part.
And the question is, if this is right, how to pass it to Ofelia.

2DArray.pd

@Jona Only a maybe.... it is not my specialist subject.
https://stackoverflow.com/questions/53858208/are-textures-the-only-way-to-transfer-audio-data-into-glsl The link to https://noisehack.com/build-music-visualizer-web-audio-api/ at the top has some (maybe useful...?) code for audio to single pixel shaders.
David.

@whale-av thank you for the links. They also use a 2d texture for the audio information.
The problem is I am really a beginner with Open Frameworks, and I do not know how to create a texture from those 2 arrays.
But another solution could be just to pass the two arrays as arrays to the shader.
Here is an example for that method:
https://github.com/ViRGiL175/sound-visualizer/blob/master/Shader_spectrum.frag

Theoretically this should work this way:

"Sometimes it is necessary to pass to the shader not just a single float value but an
array of floats. To do this, just declare the array in the shader's code as follows:

#define N (256)
uniform float myArray[N];

Now bind the array from the openFrameworks project' code:

shader.setUniform1fv( "myArray", myArray, 256 );

In the preceding line of code, myArray is a float array with 256 elements."

I can access the Pure Data arrays with the pdArray method.
But while it works to set single values for setUniform1f it seems not possible to set arrays for setUniform1fv.

I get this error message:

ofelia: [string "package.preload['1011-glslBlend2Images'] = ni..."]:68: Error in ofShader::setUniform1fv (arg 3), expected 'float const *' got 'table'

For this Line:

shader:setUniform1fv("array2", array2:get(0), 512);

While "array2" is one of the pdArrays.

@Jona "To do this, just declare the array in the shader's code as follows:"
I have always disliked the word "just".
There was something in the first link I posted about "uniform" arrays being limited to an arraysize of 64 in almost all hardware.
David.

I think this could be one approach:

1. Make a PD table with the results of the FFT (see example in 3.audio.examples/I02.Hann.window)
2. Make a LUA script with Ofelia to convert a PD table into an image. The image should be 1 by [size of PD table] pixels, each pixel should have a normalized gray value representing the amplitude for each frequency.
3. Make a FBO of that image and use it as a texture in your shader.

Step 1 and 3 seems pretty straightforward. Step 2 should be doable but requires a bit of LUA scripting skills (which I don't have).

@whale-av even an array with just 4 values does not work (the way I tried it)...
At least for webGL the uniform max size is 256, i think that is the reason why they use textures instead for the audio information...
https://stackoverflow.com/questions/51624865/what-is-the-max-size-for-an-array-to-be-uploaded-using-gl-uniform1fv
@dom1817 thanks. that could be a good way to go. And I will have a look at the hann window example, was not sure about my attempt.

Basically it works know, at least I create a 2d texture from the audio information.
I think only the first row of the audio texture is used for this shader.
The main problem is now that the audio output is crackling while the ofelia window is open.
audiovisualshadertest.zip
You can see the effect and hear the crackling in the video:

@Jona This might not be true in this case...... but.....
In Gem the object [pix_sig2pix] said as a part of the help file.....

"[pix_sig2pix~] will interprete the 4 incoming audio-streams as values for the red, green, blue and alpha channel of an image. Each image corresponds to an audio-block (signal-vector). The default blocksize in pd is 64 samples. If you want images with more than 64 pixels, you will have to change the blocksize with the [block~] object. The default image-size is calculated from the blocksize (eg: 64samples -> 8x8 & 128samples -> 16x8), but you can set it either with arguments or the [dimen( message. If your imagesize is bigger than the blocksize, the rest will be undefined.

If you have large images (pe: 256x256=65536) you will also need large audio-blocks. However, since 65536 samples are more than 1 second (at 44.1kHz) you might want to set an overlap or upsampling factor with [block~] to get decent framerates."

David.

@whale-av hmm, I think in the case of glsl the audio information get scaled to the screen size.
i got the example from shadertoy working: https://www.shadertoy.com/view/Xds3Rr
the waveform needed also a range from 0 to 255.
the "expensive" part is iterating the arrays into the audio image (i think there come the audio drop outs from):

function ofelia.bang();
for i = 0, 255 do;
image:setColor(i, 0, ofColor(array1:getAt(i)));
image:setColor(i, 1, ofColor(array2:getAt(i)));
end;
image:update();
end;

Because of that i reduced the size of the audio image to 256 x 2.
It is also not drawn every draw circle anymore (you can change the rate).
Perhaps there is another way to optimize it.
There is also a compatibility warning: "global variable gl_FragColor is deprecated after version 120".
But if I try to change it to the newer syntax, it does not work anymore (possibly because of my lacking knowlegde ).

There is still some slight audio distortion here and there, even if the cpu has not too much work.
I also tried to solve that with a subprocess [pd~] but it seems not to make a difference.

audiovisualshader_waveform.zip

Cool, I'm happy that it works! I think the «waveform» row, as intended by ShaderToy, is a simple [tabwrite~] of the signal samples with no FFT analysis.

I'm familiar with the audio distortion problem in Pd + Gem. There are two options I'm aware of. The first one is to set an higher latency in Audio preferences (something around 25 should work). The other way is to run two instances of Pd, one for audio and the other for video. The two can communicate using [netsend] or something similar. This works even with very complex patches but it's a bit tricky, maybe there are better ways to do it.

@dom1817
I tried to optimize the patch again. In addition the drawrate is reduced to 30 fps. Now it is almost without artifacts (but to have fun with the patch it should be without).
I have the feeling the subprocess version improves the result a bit, but not really.
The same with higher latency.

audiovisualshader_waveform_opt.zip

Perhaps I will try the netsend method. I think in the Egregore patch they did that in a very nice way: http://chdh.net/egregore_source.php

I would like to know if the netsend method works better than the ~subprocess method (in this case).

I have to correct myself, the pd~ subprocess is the solution, it works great even with nogui
The audio image has again the size from shadertoy (2*512) and it draws with 60 fps.
It works now because I send a list with length 1024 instead of an audio stream to the subprocess. A bit less information
The only thing is that there are very small drop outs if another window gets the focus (not inside the patch), but that is fine for me.
This could also work for GEM projects. And it is possible to build audiovisual GLSL shader in Pure Data (that is what I wanted to find out).
audiovisualshadertest_waveform_subprocess.zip