Now that it is possible to generate shader visuals with sound information in Pure Data (it was before with Gem, right?),
I was wondering how it can work the other way around. To generate sound or at least sound information with shaders and send it to Pure Data. There are some very interesting examples of how to use the GPU for sound generation. This one is Shadertoy again:
https://stackoverflow.com/questions/34859701/how-do-shadertoys-audio-shaders-work https://www.reddit.com/r/musicprogramming/comments/2cbd2s/shadertoy_has_added_glslsynthesized_audio/
This seems also interesting: https://www.fsynth.com/
Anyway, I read out one horizontal pixel line per frame and use the rgb information for generating MIDI notes.
I also use the mScale abstraction from @ingox for scaling the notes.
The player generates also MIDI notes from images, it is only silent if the whole scanned line is very dark or very bright .
My first approach was to generate audio from the pixel information like in the examples above, but the sound was always distorted (but I am sure it could work... ).
I am also interested in other methods to generate sound from visuals and visuals from sound.

GLSL_Video_Effects_V01_MusicPlayer.zip

Here is an example from the MusicPlayer:

Hi everyone. I have frequently revised designs for polyphonic envelopes. i've often misunderstood things about vline~ and scheduling voices in such a way to avoid unwanted clicks while also keeping things on time and snappy.

i'd be really happy to know what your methods are for envelopes.

i submit this patch, a reflection on envelope practices and how i address certain challenges. envwork.pd

this patch makes these assertions:

1- because vline~ maintains sample accuracy by scheduling events for the next block, you can switch dsp on in a subpatch with block~ while sending a message to vline~ and the dsp will be active by the start of the vline~ output. This also works if you need to configure a non-signal inlet before triggering a voice. send a message to such an inlet concurrently with a vline~ message and the parameters will update on the block boundary before the vline~ plays.

2- accounting for note stealing can cause issues in a polyphonic patch. if the stealing note has a slow attack and the envelope of the stolen note is not closed, there will be a click as the pitch of the new note jumps. the voices in my patch apply slight portamento to smooth out this click. if, however, the attack time of the stealing note is faster than this slight portamento it is counterproductive and will soften the attack of stolen notes. Stolen notes need every bit of snap they can get because the envelopes may be starting at a non-zero value. so i limit the time of the portamento to the attack time.

3- to make sure a note that is still in its release phase is treated as a stolen note, it is necessary to monitor the state of the envelopes like so:

switching the dsp off too close to the end of the release causes clicks. after testing, my system liked a full 50ms of extra delay after the end of a release before it was safe to switch off dsp. I don't think this is attributable just to the scheduling delay of vline~ but it's a small mystery to me. possibly there's a problem with my voices.

This all gets a little more complex when there are multiple envelopes per voice. The release time that affects the final output of the voice must reset all envelopes to when it is finished and before dsp is switched off. Otherwise an envelope with a long release affecting something like filter frequency can be held at a non-zero value when dsp is switched off and spoil the starting state of the vline~ on a new note.

finally, on vline~ and sample accuracy and timing, let me type out what i believe is the case. i could be wrong about this. if you programmed a synth using line~ for the envelopes, it would be faster than vline~ but not all notes equally faster. all notes would sound at the block boundary. Notes arriving shortly after the last block boundary might take 90% of the block period to sound. notes arriving just before the block boundary might take 10% of the period to sound.

vline~ will always be delayed by 100% of the block boundary. but the events will be scheduled sample-accurately, so the vline~ will trigger at exactly the real time intervals of the input. a synth with line~ envelopes will trigger any two events within a single block at the same time.

this should mean that vline~ envelopes can be accurately delay compensated and stay absolutely true to input timing, in the case of something like a Camomile plugin.

however, if one was to build a synth for something like a raspberry pi that will act as hardware, would it be better to use line~ envelopes and gain a little bit of speed? is the restriction of locking envelopes to block boundaries perceptible under normal playing conditions?! i could test some midi input and see if the notes in a chord ever achieve a timing spread greater than the block period anyway...

Based on the midi generator https://forum.pdpatchrepo.info/topic/10791/markovgenerator-a-music-generator-based-on-markov-chains-with-variable-length that i made with @ingox some time ago I made a Lua markov chain generator that generates endless markov chains from midi files.
The patch needs the Ofelia, Cyclone and Zexy libraries.

I have one question: Is there a way to keep the "original" sounds from the midi file? Perhaps if i store the midi control data together with the other values (pitch, velocity, program change, midi channel)? Somehow the midi program change values do not have the "right" sound information. If I play midi files with the [seq] object it sounds like it should. So for now i choose the sound manually but it would be nice if it sounds like it is meant to.
LuaMidiMarkov.pd

A silly example:

I finished the Ofelia / Lua Markov Generator abstraction / patch.
The markov generator is part of two patches but can easily be used as an abstraction.
I want to use it for pattern variations of a sequencer for example.
It just needs a Pure Data list as input and outputs a markov chain of variable order and length.
Or draw into the array and submit it to the markov generator.
The first patch is an experiment trying to create interesting sounds with the markov algorithm.
In addition I used the variable Delay from the Pure Data help files:
LuaMarkovGeneratorSynthesizer.pd

The second patch creates markov chains at audio rate, it is quite cpu heavy but works until the 10th markov order.
It is quite noisy but I was courius how it will sound:
LuaMarkovGeneratorAudioRate.pd

And here is the Lua code.
The core of the code is adapted from this python code: https://eli.thegreenplace.net/2018/elegant-python-code-for-a-markov-chain-text-generator/
A few things that I do not really understand yet, but finally it works without errors (it was not easy sometimes ):

-- LUA MARKOV GENERATOR;
function ofelia.list(fv);
;
math.randomseed(os.time()- os.clock() * 1000);
;
print("LUA MARKOV GENERATOR");
local markovOrder = fv[1];
print("Markov Order: ", math.floor(markovOrder));
;
-- make dictionary;
;
local function defaultdict(default_value_factory);
;
local t = {};
local metatable = {};
metatable.__index = function(t, key);
if not rawget(t, key) then;
rawset(t, key, default_value_factory(key));
end;
return rawget(t, key);
end;
return setmetatable(t, metatable);
end;
;
-- make markov matrix;
;
local model = defaultdict(function() return {} end);
local data = {};
for i = 1, #ofelia.markovInputList do;
data[i] = ofelia.markovInputList[i];
end;
print("Data Size: ", #ofelia.markovInputList);
for i = 1, markovOrder do;
table.insert(data, data[i]);
end;
for i = 1, #data - markovOrder do;
local state = table.concat({table.unpack(data, i, i + markovOrder - 1)}, "-");
local next = table.unpack(data, i + markovOrder, i + markovOrder);
model[state][next] = (model[state][next] or 0)+1;
end;
;
-- make tables from dict;
;
local keyTbl = {};
local nexTbl = {};
local prbTbl = {};
for key, value in pairs(model) do;
for k, v in pairs(value) do;
table.insert(keyTbl, key);
table.insert(nexTbl, k);
table.insert(prbTbl, v);
end;
end;
;
print("Key: ", table.unpack(keyTbl));
print("Nex: ", table.unpack(nexTbl));
print("Prb: ", table.unpack(prbTbl));
;
print("Make a Markov Chain...");
;
function ofelia.markovChain();
;
-- make start key;
;
local startKey = {};
if ofelia.randomStart == 1 then;
local randomKey = math.random(#keyTbl);
startKey = randomKey;
else;
startKey = 1;
end;
;
local markovString = keyTbl[startKey];
local out = {};
for match in string.gmatch(keyTbl[startKey], "[^-]+") do;
table.insert(out, match);
end;
;
-- make markov chain;
;
for i = 1, ofelia.markovChainLength do;
;
-- weighted random choices;
;
local choices = {};
local weights = {};
for j = 1, #keyTbl do;
if markovString == keyTbl[j] then;
table.insert(choices, nexTbl[j]);
table.insert(weights, prbTbl[j]);
end;
end;
;
-- print ("choices:", table.unpack(choices));
-- print ("weights:", table.unpack(weights));
;
local totalWeight = 0;
for _, weight in pairs(weights) do;
totalWeight = totalWeight + weight;
end;
rand = math.random() * totalWeight;
local choice = nil;
for i, weight in pairs(weights) do;
if rand < weight then;
choice = choices[i];
break;
else;
rand = rand - weight;
end;
end;
;
if math.type(choice) == "integer" then;
choice = choice * (1.0);
end;
;
table.insert(out, choice);
local lastStep = {table.unpack(out, #out - (markovOrder-1), #out)};
markovString = table.concat(lastStep, "-");
end;
;
return {table.unpack(out, markovOrder + 1, #out)};
end;
end;
;

The markov generator works now. It was just a small formatting error.
Here is a small experiment with the markov generator, but basically i want to use it for sequencing and it can be used for anything else, just needs string formatted data as input. i think it is quite a bit faster than the generator that i build together with @ingox some time ago. https://forum.pdpatchrepo.info/topic/10791/markovgenerator-a-music-generator-based-on-markov-chains-with-variable-length/1
lua_markov_generator3b.pd

Hello. I'm messing about with creating envelope generators in Pd using arrays.

I've tried to make the array's settings easily changeable to allow it to work for amplitude, pitches and other uses, using variables sent to a message for the array.
But I've found that most of the time with this setup I just patched up I will, with a lack of better words, break Pd, get a bunch of error messages in the terminal and number boxes will stop showing changes, alongside dragging in values on the array.

The patch in questions and the error message is shown here:

The values shown in the picture work fine, those are set up as an amp envelope. Changing certain parameters will break it, I can't quite be sure which. Sometimes I will just get the "graph: empty bounds rectangle" message, which will usually be followed by the more verbose error message.

The changes seem to be made, but it just won't show it. I have to completely quit Pd and open it again for it to return to normal.

As soon as the error message occurs, the number boxes will freeze and add "..." to the end, as if the number shown exceeds the width of the box, which is not the case.

I assume I'm running Pd into a wall, placing values that are out of bounds for the given array, but I can't tell how and why. It's my first venture into using graphical envelopes, I've usually just used the more primitive version of line ramps and delayed signals to get regular adsr envelopes.

Can anyone tell me what I might be doing wrong here?

I'm running vanilla 0.47.1 on a Macbook Pro running El Capitan.

Thank you.

Thanks for all the suggestions about this. I'm going to keep plugging away and report back when I find the fix, though I'll probably ultimately take th8a's suggestion of using tables for the envelope segments so I can eliminate the hacky way I'm using threshold~.

I've been building in lots of print and env~ objects to try to figure out where the exact problem is. I believe I've successfully determined that it's NOT the voice stealing or the threshold~ object. All the midi info seems to be getting routed to the right voices, and the decay segment seems to be playing correctly even in notes that get stuck on, which it's the purpose of the threshold~ object to trigger.

For some reason the midi off control seems to be sent to the envelope, but it isn't registering it. I can fake another off event with messages and the envelope releases normally at that point. I've been playing around with a slightly more complex synth patch that uses the same envelope abstraction, which is actually a little more helpful because it has multiple oscillators for each voice, and they do NOT all seem to get stuck on at once. In fact, the base oscillator seems to be the only one that gets stuck on, whereas the 2 suboscillators do not, which suggest to me that I might have screwed up the construction of how the oscillator patch interacts with the envelope. But I'm still only closer to figuring it out; I haven't had much time to mess around with it lately.

@weightless I like this idea of bracketing strings (I like the visual language too). I guess that the brackets as delimiters would behave as part of the generating shapes that seed further generations propagated from that point. the thing is, if the L system is iterating something with simple fractal rules like a cantor set, and this is deterministic and the rules themselves don't change over time, an isolated branch would be then similar to another piece from a lower order of magnitude anyway. a segment of a future generation will be a segment of the generative string in the first place, just "out of phase" (in a different order due to a different starting place). so, in that sense it is like what I was trying to get at before, a loop where you could run the process to some level of complexity then return back and do it again. I guess that is actually not at all a hard thing to do, I was just excited by the way you implemented this, super great.

this website might be of interest: http://algorithmicbotany.org/
there is a whole book in pdf on there about generating fractal plant morphologies with L-Systems

Hello!

I am trying to make an modulation envelope that works with the lop~ filter. The lop~filter uses a control inlet for the filter frequency where vcf~ and bp~ uses a signal inlet. And the envelope I have made works perfectly for the vcf~ & the bp~ but not the lop~.

I tried using snapshot~ to convert from signal to control, but the result is not good, even when I feed the snapshot~with a very fast metro to refresh it rapidly.

My patch looks like this:

The subpatch on the left side is the sequencer so you can see what signal I feed into the envelope. When I am live with the patch I can see that the envelope receives a trigger signal. So I am pretty sure the inlet/trigger part is working, so I dont think that is the issue.

On right side is the envelope that I cannot get working with the lop~ filter.

Does anyone have a suggestion to:
a. What I can do to either get the decay envelope I all ready have working in another way than using snapshot~.

b. Make a new decay envelope that uses control signal. Or maybe there is all ready one? I havent been able to find one....

Any help appreciated!

Thanks, Jaffa!

If you wanted to do this, you could use [until] and hook it up to a counter to generate 0-n numbers, and then feed it into your formula, like this:

[until]
l
[f] x [+ 1]
l
[my formula]

But I wouldn't really recommend doing this if you want to control the parameters in real time. The formula will update every point each time it receives a new value, which is very inefficient and will possibly lead to audio clicks.

A better solution is to generate the envelope in real time with line level objects. [vline~] will give simple linear output; you can square it if you want a curve. You can pack all the variables you want into a list and then send them to the object each time you want to generate a beat.

If you want exponential curves, I posted a way of doing this here: http://forum.pdpatchrepo.info/topic/10336/exponential-adsr-envelope-using-lp-filter