This is a gem Concept I've been working on. Scans a pix for grey value, then re-constructs it with different sizes and shades of circles.

Any ideas as to what I could do with the raw pixel matrix data? For instance, this patch creates a 400-point (20X20) array representing grey values, this could easily be placed in a table and used for timbre-stamping, waveshaping etc...

but

Is their a better way to store this data? How can I use it as audio control data and still retain it's 2-dimensional properties?

--Nestor--

"Is their a better way to store this data? "

Yes but no. There's plenty of ways of reducing a matrix (effectively compressing it) but since it is to be used immediately in its raw form I can't see the point of a fancy datatype. 20x20 is very manageable.

"How can I use it as audio control data and still retain it's 2-dimensional properties?"

Audio signals are really 2D anyway if you take time as part of the vector. Choosing how to map the dimensions of your input data onto synthesis/scoring parameters is the nub of the problem for composers using data from visual domain. As quick suggestions:

1. Amplitude and filter cutoff
2. Note duration and note pitch
3. Modulator frequency and index in an FM instrument

But actually - if you have a square matrix of grey values you actually have 3D data, p x q x r where r is the greyscale value.

I find it's nice to plot as a surface to give you ideas for how "shape" maps onto a sound or score.

I'll attach my patch this time...

@obiwannabe said:

But actually - if you have a square matrix of grey values you actually have 3D data, p x q x r where r is the greyscale value.

Yeah, I guess that is what I meant... I was thinking of it a little differently, here goes:

It would be easy to treat a matrix of 16X16 as 16 lines of 16 points (16 2-dimensional arrays); Columns representing time, as in a 16-step sequencer, Points could represent amplitude, pitch, whatever. In this way one could control 16 "instruments" or 16 "parameters" of any number of instruments.

The problem with this is when using video, the events would occur at 16X the frame rate, at 24 frames, this is 384 events in a second. Also one does not recognize the visual congruity from one frame to the next in the audio stream.

I think that the best use I have experimented with is in FFT convolution. Perhaps the answer for me is in creating some complex audio signal with the above method, and then resynthesising based on the contents of the array.

@obiwannabe said:

I find it's nice to plot as a surface to give you ideas for how "shape" maps onto a sound or score.

How would one do this? In Gem?

http://www.pdpatchrepo.info/hurleur/tiles3.pd

I see what you mean. For video, with the images passing by so quickly there's a mismatch between the rate of data in the visual domain and how fast it can be rendered and assimilated in the audio domain. It's not really my thing, but lurking and listening to VJs and animators a few ideas seem to recur. One is to do threshold detection of some kind - perhaps look for groups of pixels that change suddenly in intensity in a certain area and have those trigger an audio event. Another is to do edge detection, comparing the brightness of adjacent pixels and triggering an event whenever the difference between a group or line of them passes a threshold.

For surface plots I dump arrays or lists to Gnuplot (or Jgraph/Xgraph), but that is slow. I'm sure a Gem/Gridflow expert will give you ideas on how to make a histogram or height plot in real time - out of my depth with all that stuff I'm sorry to say.