To be honest, I'm not sure how clear this is going to be, because the best way to make it clear is to build it in Pd, and I don't have time to do something complicated.

Anyway... consider a "data stream" interface. Some part of the code asks the stream for the next value, and -- in text languages -- the value is returned to the caller. There may be dozens of places that need to access the stream, and the rule is, whoever asked gets the result.

I had mentioned parsing. If the input string is held in a stream wrapper (in SC, CollStream), then this object can be passed around the system as a function/method argument, and everywhere that needs the next character can just ask for it.

The challenge, then, in graphical patchers is that there is no concept of returning a value to the caller. A message traversing a patch cable is calling into the next object. "Returning from a function" is, in patching terms, backtracking to a prior object to see if it has more output cables that haven't been handled yet. But data don't travel backward up the stack, only down.

A file handle can be treated as a stream. Here, Pd recognizes the issue and allows shared file handles to be "define"-d: [file define xyz], and then every "caller" can have its own [file handle xyz].

If the data source is in memory, though, then the current position in the stream has to be maintained within the abstraction -- and there can be only one of those, because Pd/Max don't have a concept of a pointer to an abstraction that's loaded somewhere else. E.g., a really simple data queue (didn't implement "rewind" here) -- as an abstraction, everything would have to go through this.

So then there has to be a way to request a value from a specific part of the patch and get back to that part of the patch.

... where oid said, "In that situation I would say that [pd my-stateful-thing] should have a send method" -- seems to me that's what the above is (just not folded into the subpatch).

Regarding your live-coding dialect, would be curious to see it if you shared it anywhere.

https://github.com/jamshark70/ddwChucklib-livecode -- see https://github.com/jamshark70/ddwChucklib-livecode/blob/master/cl-manual.pdf for full documentation.

The parser (with heavy use of CollStream throughout) is at https://github.com/jamshark70/ddwChucklib-livecode/blob/master/parsenodes.sc

Example:

Most of what I do in pd is designing languages

Quite curious about that. Anything public to see?

hjh

seems to me that's what the above is (just not folded into the subpatch).

You already have the send method, you just are not using it, [list store] gives it to you for free, as do [float], [int] and [value] , [text] and [array] have the send method to get a pointer to their data. I find it difficult to think of a situation were we can not build our abstractions around these objects and easily get access to the internal state of an abstraction to send data backwards or even "rewind" the patch but you have to plan it that way. Am I missing something? I am good at missing the point when it comes to contrived examples and fail to see beyond the example.

Not sure about pointers to abstractions, think I would need an example of your use case to address that.

Quite curious about that. Anything public to see?

Current project is actually getting a few of them into a shareable state, required me making a set of abstractions which is essentially a dialect of Tk since interacting with a text based language is awkward at best in pd. Believe the last hurdle in those abstractions has been passed and will finish them all up and document/upload them this coming week so I can get onto reworking the languages to use the Tk abstractions. They will get posted in the abstraction section as they become usable, I suspect only the Tk abstractions will have any real interest but I am curious to see if anyone uses my languages or they get people to develop their own languages. Thanks for the links, will dig into them this weekend.

Edit: forgot pic. And I did not quite plan that one out well, could easily cause an infinite loop or overflow depending on where [s backwards] actually sends to earlier in the patch. But I think it shows the logic?

You already have the send method, you just are not using it,

Here, though, if [list store] holds the queue's contents, then it's necessary to perform an operation to pop the top value off of the queue. Calling "send xxx" on the [list store] will send the entire queue contents to the target, but we don't want that. We want just the top value. Otherwise it's just a variable, it's not a queue.

After performing that operation ([list split 1] in my example), then you just have a datum, and the way to send a datum to a receiver is [send].

Part of the problem is that iterators are commonplace in object oriented programming, where patching doesn't easily support this idea of "invoke the iterator from anywhere." If your primary frame of reference is patching, then this is going to seem a bit vague or weird.

A simpler scenario: if you have a note sequence, but you want 3 different players to request notes from the stream following their own independent timing. I guess that a more typical Pd approach would be a master [metro] driving the sequence, with logic below the sequencer to route the values among the different players. But that isn't independent timing -- each player needs its own [metro], and a way to pull from the stream. In SC:

(
var noteSequence = Pshuf([48, 52, 53, 57, 59, 62, 64, 67, 72, 74, 79], inf).asStream;

~threePlayers = Ppar(
    Array.fill(3, { |i|
        \instrument, [\synth1, \synth2, \synth3][i],
        // one stream, shared among voices
        \midinote, noteSequence,
        \dur, Pexprand(0.2, 0.8, inf),  // independent per voice
    }),
    inf
).play;
)

I guess I'll have to build it out in Pd for this to make any sense, but I can't do it right now.

Not sure about pointers to abstractions, think I would need an example of your use case to address that.

Purely hypothetical, not worth pursuing.

Current project is actually getting a few of them into a shareable state, required me making a set of abstractions which is essentially a dialect of Tk since interacting with a text based language is awkward at best in pd.

Looking forward to examples -- I can't quite get the gist from this.

hjh

If your primary frame of reference is patching, then this is going to seem a bit vague or weird.

Give us a working patch in SuperCollider with an example, not too complicated thoufh

I guess I'll have to build it out in Pd for this to make any sense, but I can't do it right now.

I could build it too, but give us a patch that actually makes sound please. Again, try to be as minimal as you can

If I get it, you got one sequence and 3 independent playing from this sequence. I can envision a simple [clone] with 3 instances...

Looking forward to examples -- I can't quite get the gist from this.

You can do stuff like unpack the canvas from the patch window and fill the window with Tk widgets instead, or have the Tk widgets and the canvas tiled or tabbed, make new windows, embed other patches into your current patch, lots of fun stuff. The abstractions do all the hard work and setup the stuff for communicating with your patch.

Edit: That was a silly place to connect the [text get].

each player needs its own [metro], and a way to pull from the stream.

like this?

All of this can easily be worked with each instrument being an abstraction using the right inlet on the array get or arguments or sending the pointer. Skipped some of the details but duration and getting the stream into the individual abstractions is straight forward enough and more of the same. I can't tell if I am missing something, life outside of data flow has become weirdly abstract for me, I am one of those sorts who really took to it. I am very interested in know what I am missing if I am missing something, I have been trying to cure my data flow myopia, it has become a bit of a disability in the last year or two.

Edit: Or is it that you just don't like that each needs its own metro? If so I think your issue is not with patchers/data flow but low level languages, pd is low level. But that is what abstraction and externals are for.

So here is what I mean:

triple-arp.pd -- plugdata, and uses else and/or cyclone for audio, apologies to vanilla-land.

The notes are to play in the same order continuously, but distributed among the three players. If I print out the symbols being accessed and add note indices, it's:

next: symbol instr3 -- play note 0
next: symbol instr2 -- note 1
next: symbol instr1 -- note 2
next: symbol instr1 -- note 3
next: symbol instr2 -- note 4
next: symbol instr1 -- note 5
next: symbol instr3 -- note 6
next: symbol instr2 -- note 7
next: symbol instr1 -- note 8
next: symbol instr1 -- note 9
next: symbol instr2 -- note 10
next: symbol instr3 -- note 11 etc.

@porres 's "one sequence and 3 independent playing from this sequence. I can envision a simple [clone] with 3 instances..." sounds like it will clone the sequence -- that's not the same thing.

@oid 's version will skip notes, because it's advancing through the stream on a fast timing that is independent of the three players' timing (and the order isn't deterministic -- whereas SC's Pshuf repeats the same shuffled ordering). That's also a misinterpretation. The advancement through the stream should be passive, in response to requests from the players.

("Why would you want to do that?" Why not... it's a reasonable algorithm.)

Now, why do I care about this? Because I've done a lot with state-machine parsers and found it to be really nice to have, for instance (and speculating in Pd terms) an abstraction that parses an identifier, and another abstraction that parses a number, and another that parses some kind of bracketed group containing identifiers and numbers, etc. How do these different parts of the patch know where you are in the source string? This is easy if there is a shared object that streams out the characters one by one, and the character goes to the part of the patch that requested it. Distributing a single stream among multiple players is just a different version of the same problem -- I was hoping that putting it in musical terms would make it clearer, but that seems not quite to have worked.

It's mainly out of curiosity to see how patching handles this.

• I'm suggesting [send] with a variable target: the request tells the stream where to send its "next" result.
• Or @solipp has suggested to spray the value everywhere in the patch that it might be needed, and then the caller bangs only the one it's interested in.

I feel a bit like the "spray everywhere" solution would require more object copy/paste -- mine just has "; next symbol instr1" where the other needs a [t], a [s], a [r], and an [f]. Also my [pd stream], if it used a list instead of an array, could be type-agnostic. (I only used an array here to avoid more lectures about the perils of list manipulations, which are not quite relevant I think.) But YMMV.

hjh

I feel a bit like the "spray everywhere" solution would require more object copy/paste -- mine just has "; next symbol instr1" where the other needs a [t], a [s], a [r], and an [f].

well, you can wrap it into a subpatch and copy/paste that : )
Essentially both methods are not complicated, they just have to be implemented in a low level language as @oid pointed out.

Seems to me English is not up to the challenge here.

We just needed to identify the actual issue here, which is not patching but low level language. To do the sorts of things you want to do means you have to define an interface and a protocol, those things which high level languages like SC take care of for you. Pd has no concept of an instrument and if you want to use such a paradigm it is up to you to implement it. One way would be to send the instruments args to a master abstraction which then takes the necessary actions, a quick and simple example which is not fully fleshed out but should show the logic:

Started adding in my $0s halfway through. You would probably want a [text] in [register] to keep track of all this stuff and it can also keep track of values you can query for in the various abstractions and what not through their send methods. [stream] would need some more logic, needs a [receive] to start/stop it and probably would want a way to change its settings as well as an [iemguts/canvasdelete] so you can clean things up as your patch changes. [synth1] seems pretty complete and a solid instrument, I wouldn't change it.

We just needed to identify the actual issue here, which is not patching but low level language. ... Pd has no concept of an instrument... it is up to you to implement it.

The instrument isn't the point (and [pd instr1] etc. in my example is an implementation, though a minimally simple one).

But... as I read your design... if I "; register asStream synth1" then synth1 gets a new instance of the stream abstraction, and if I subsequently "; register asStream synth2" then synth2 gets a separate instance of the stream abstraction... which is precisely not what I'm talking about.

In SC, if I define multiple sequences based on patterns, then each sequence makes its own stream:

(
var pattern = Pn(Pseries(0, 1, 15), inf);

var seq = Pfindur(24, Ppar([
    // this can be written with less code,
    // but I'd better expand it out because
    // the shorter version hides the multiple
    // references to 'pattern'
    Pbind(
        \instrument, \instr1,
        \degree, pattern,
        \dur, 0.25
    ),
    Pbind(
        \instrument, \instr2,
        \degree, pattern,
        \dur, 0.25 * 4/3,
    ),
    Pbind(
        \instrument, \instr3,
        \degree, pattern,
        \dur, 0.25 * 4/5
    )
]));

m = SimpleMIDIFile.fromPattern(seq, inTracks: 3, inTempo: 60, trackNames: [\instr1, \instr2, \instr3]);

m.write("~/tmp/parallel-streams2.mid".standardizePath);
)

If, on the other hand, a single stream is made at the top, and shared among the sub-sequences, then the notes are interleaved in time order.

(
var stream = Pn(Pseries(0, 1, 15), inf).asStream;

var seq = Pfindur(24, Ppar([
    Pbind(
        \instrument, \instr1,
        \degree, stream,
        \dur, 0.25
    ),
    Pbind(
        \instrument, \instr2,
        \degree, stream,
        \dur, 0.25 * 4/3,
    ),
    Pbind(
        \instrument, \instr3,
        \degree, stream,
        \dur, 0.25 * 4/5
    )
]));

m = SimpleMIDIFile.fromPattern(seq, inTracks: 3, inTempo: 60, trackNames: [\instr1, \instr2, \instr3]);

m.write("~/tmp/shared-stream.mid".standardizePath);
)

The latter is what I'm after (and is in fact what my example patch does). And it's not even that I care that much about the musical result, but that this is a musical context for a general programming problem.

hjh

@ddw_music Does this behave like a shared stateful function that returns to the caller?

It does (and is basically identical to the approach in my example patch above).

So perhaps this really is the best way.

hjh

(and is basically identical to the approach in my example patch above).

Oh sorry, I missed that. TBH I've been barely hanging on I'll go back and read why you were hesitant about endorsing this approach. But don't you think that in the end, all languages have things that are convenient and things that are inconvenient, and so....so what? Before I retired I was studying functional programming languages like Haskell, and I didn't get far enough along to understand how the hell you could program without state! I understood that it was great for multiprocessing, but wow.

The latter is what I'm after

Oh, that is simple enough to do, just replace the [f ] in the stream abstraction with a [v ] shared between the abstractions like the array. Getting there?

Obviously you would want asStream to create the [array define]s as needed and the symbols for the array and value having their own iterators so it can manage multiple streams but that is simple enough. Oh, and register its array and value symbols so you can grab them as needed and append to the array and what not.

(I guess?) I still lean slightly toward the targeted-send approach, over the spray-everywhere approach. One reason is, if you're using lists, sending a sub-list to one place is likely to be more efficient than sending it to potentially dozens of places. It's a valid concern, about redundant list allocations; one way to deal with that is to be less promiscuous about sending data to places where they won't be used.

I do appreciate everyone's indulgence. I've certainly gotten more comfortable with patching over the years, but I'm never quite sure if what I'm doing is really idiomatic.

hjh

Not sure what you mean by "spray everywhere," these are targeted approaches and there is no copying. Does reallocate on resizing the array but that could be avoided easily enough, just use an array bigger than you will ever need and only use what you need.

Improved version below.

Added some usage notes in the patch

don't see it, where are the help files?

@ddw_music Value has been mentioned multiple times in both threads

I know, and I've used value in other places. I hadn't thought to connect it to the current index, though.

Not sure what you mean by "spray everywhere," these are targeted approaches and there is no copying.

Sure, that's a misreading. I see also (on closer inspection) that folding the metro into the stream abstraction makes the stream effectively its own caller, sidestepping the return to caller problem. Though this means that your approach could be used only in self-timed situations, where SC streams and Python iterators are more general.

Haskell people like to say that a programming language isn't worth learning if it doesn't change the way you think. Pd has actually done that for me, to some extent.

hjh

asStream.pd
stream.pd

@ddw_music said:

Though this means that your approach could be used only in self-timed situations,

Just need to change [obj 10 10 stream $1 $2 $3...( to [obj 10 10 $1 $2 $3...( and add in some simple logic and now asStream can load the abstraction you specify through methods/arguments. As I said earlier, you design the interface and the protocol which is what those sends and receives and values and dollar arguments in asStream are, an interface and protocol for communicating with the stream.pd abstraction or any other abstraction designed to work with that interface. Object names themselves can also be filled in through arguments, [$1 $2] can be either a metro or a receive or anything else but [$1 $2 ....] are actually [list]s so they only give you two inlets and one outlet, if you need more than that you need to resort to editing the abstraction in a text editor, patch it together with any object with suitable io and then open it in a text editor and edit that object into $1 $2... And there is plain old dynamic patching to programmatically create the required patch. Lots of option and they can all be mixed.

@porres No help file, usage notes in asStream.pd, this thread is the documentation.

You see, I'm interested in this thread, I suggested that it would be created. I'm interested in the problem and how to make it work in Pd. I read the thread. This that you're sharing is not enough for me. I get that you may be satisfied and that you worked a lot and just about as much as you care on this, but I just wanted to make this clear. Ok, I know, I could do more of my homework, but I'm born and bred in Pd-land, all this just sounds way too complicated and like explaining 'snow' to me, who's also born and bred in a tropical country (I was at the beach yesterday by the way), so more help would be appreciated.

I have brought myself into such Pd x SC challenge other times, it's not the first time that things don't get too conclusive. I hear that it's impossible to do something "x", or too hard, and I hear about complex programming paradigms, and then it comes down to a simple solution like using [value]. My idea is to maybe try and expand Pd to improve it and open it up to what some people that come from elsewhere struggle with, and maybe offer a new object in my ELSE library... but all I'm getting so far from this thread is that you can share the index of an array with [value] in multiple subpatches or abstractions.

I was interested in studying this abstraction and see if maybe I could add it if I really got what it did and what it is for, but for now it's just quite abstract and over my head.

cheers

I don't think extending pd would solve this issue, the question was not really how to do this in an isolated situation but about the methodology. How do you do something like functionA (functionB (1 2) 849 functionC ($1 $2) asdf) in pd? In a one off situation it is generally a fairly simple problem to solve but to make it so at anytime you can do the equivalent of [abstractionA [abstractionB 1 2] 849 [abstractionC $1 $2] asdf] is a much more complicated issue and one which externals don't solve and often make it seem more complicated than it actually is because they have a tendency to make pd feel like a higher level language than it is, keep the user from ever really learning the nuances and techniques of the low level dataflow paradigm that is pd. The one off solutions which often come up in these threads result in what I think @ddw_music refers to as "spray everywhere," to try and build that solution into every abstraction so you can use it everywhere but to get the equivalent of abstractions calling abstractions would require you to fill your abstractions with [v ]s and [send]s and [receives]s in the vain attempt to cover every possible situation and very complicated/convoluted bit of inter abstraction communication. All my posts since the first asStream screenshot post have been about methodology, how to build higher level paradigms in a low level language like pd and avoid spraying everywhere, this is what asStream is meant to show. Like any well founded low level language pd is quite good about giving you the tools to build out higher level functionality and adapt it into your own language. At least that is what I got from this thread.

asStream has the potential to be a really useful abstraction but still has some issues, I can help you understand it and flesh it out, just ask your questions, they will probably make it a better abstraction than either of us could make on our own. As an exercise I will attempt to repatch it in your heavily subdivided patching style on my next break from my projects, I will probably fail but it will be interesting, I get the logic of the style but it obfuscates the dataflow for me and anytime I want to understand the work of a subdivider I have to remove all that subdivision, I can't even see how it could be subdivided into discrete, well named subpatches of highly singular/specialized function, for me it already is that.

for now it's just quite abstract and over my head

I made a concrete example -- with scores! If the programming-ese terminology is too difficult, maybe focus on that example.

I hear that it's impossible to do something "x", or too hard, and I hear about complex programming paradigms, and then it comes down to a simple solution like using [value].

That's a good point. Sometimes it takes effort to find the simplest way.

@oid

I don't think extending pd would solve this issue

Yeah, the core of the issue is that Pd's stack works differently from most text languages since Fortran.

Tbh I think variable-target [send] is fine, once it's been established that [send] is going to be involved in pretty much any Pd solution. "Return to caller" as in any structured or object-oriented language is nicer, but [send] is not bad.

what I think @ddw_music refers to as "spray everywhere,"

Here, I'm referring specifically to https://forum.pdpatchrepo.info/topic/15085/return-to-caller-followup-from-symbols-explicit/8 , where the [send] pushes the value out to all matching receivers -- to every potential caller -- where it's swallowed by an [f]'s cold inlet, and only the specific caller pushes the value forward. If you have 50 potential callers, then the data have to be replicated 50 times.

If the data are sub-lists instead of single floats, that would be a lot of multiple allocation. My/jameslo's solution doesn't do that; nor does yours.

to get the equivalent of abstractions calling abstractions would require you to fill your abstractions with [v ]s and [send]s and [receives]s in the vain attempt to cover every possible situation

[list-map] in list-abs shows a nice approach, though, decoupling the iteration logic from the user-operation: the abstraction outputs a trigger to perform the operation, and then the result of that operation is put back into the abstraction to finish the job. That could inspire a super-minimalist approach (which has the benefit of assuming nothing about the returned value's type, whereas most other solutions posted here seem to assume it will always be a single float):

Now, I didn't think of this until the discussion went through a few iterations... see porres's comment about the eventual solution often being much simpler than the problem seems at first.

hjh