I've progressed a bit, here is what I could do so far. Thank you all for your advices and tips, it really helped me!:

Desktop graphics sonification using ManyCam:
https://www.sugarsync.com/pf/D6163802_05628859_7827373

The patch uses ManyCam (or similar software, tip above by @Johnny Mauser thanks again!). The desktop is fed to the GEM. Mouse position is extracted and it's position used to sonify pixels under mouse pointer.

The patch still needs a lot of work, especially to tune the waveforms for the individual colors (RGB). There are some other things like clipping output etc. that need to be fixed. I need to move on with my research project right now, but I'll post the updated version whenever they'll be ready.

Known glitches:

• Y position is incorrectly reflected to the GEM window, X position is ok. I need to dig into this if this is gemmouse issue or something else.

• Sound clipping, not much time to fix this right know.

• Lots of other small things

• White pixels sound is too loud and clips the output (max amplitude for all color synths, this probably needs to be adjusted or somehow limited maybe by expression that will reduce the output if all amplitudes are max.

Thanks @EEight @LiamG & @Johnny Mauser for very useful suggestions, I will give them a try!

@EEight That's right, maybe I'll try Python. I decided to make demo in PD capable to sonify only pictures. I have one issue though. I am looking for a function that will return me the color of specific pixel under the mouse pointer (in GEM window). Right now I am using [gemmouse] to determine the mouse position in the window, but I don't know how to get info about the pixels on the mouse position.

Thank you @EEight, I am running the PD-extended on the Windows 8.1. I cannot get the shell object running (seems like my PD does not know it, I read elsewhere in the forum that it does not work under the Windows). I found some Windows Powershell scripts for taking screenshots, but that might be rather slow if I want the sonification to be dynamic.

Is it possible to run some C or C# code from PD patch? I found some C# code that can directly determine the color of the pixel under the mouse pointer. I have experience only with Java and PHP but I think this might be better solution.

Hello everyone!

I wanted to use the Pure Data to create a desktop or picture sonification tool (dynamic conversion of the graphic properties to sound). For now my main question is whether there is a way to get the graphical data from the screen (get something like printscreen) and work with them using GEM. Right now I was only successful with loading an individual images using the pix_image.

I guess that I might need some external script for this, or do you have any other ideas?