@Il-pleut Yes.... 250 and 252.
This might interest someone...... https://github.com/IndustrieCreative/Harmonync
I haven't uploaded it so as not to upset @Eeight
Although it was built a long time ago for extended the heavy midi lifting was done including sysex and sysex bulk dumps with checksum (maybe why no-one is having much luck with bulk dumps at the moment)...... sysex.zip
Though mostly Vanilla of course.
David.

@oid said:

@ddw_music Long sysex messages work just fine if you treat them as what they are and what pd is quite good at dealing with, streams.

I do understand what you're saying, and I began with the same assumption -- that [midiout] should be just a low-level conduit -- byte goes in, byte gets sent out.

Unfortunately it doesn't work that way in practice.

My test scenario looks like this. Launch SuperCollider and Pure Data. It helps in my case that I'm testing in Linux, where I can create and remove MIDI connections arbitrarily in qjackctl's ALSA tab.

SC code:

MIDIClient.init;

MIDIIn.connect(0, MIDIClient.sources.detectIndex { |src| src.device.containsi("pure") });

m = MIDIOut(0).connect(MIDIClient.destinations.detectIndex { |dst| dst.device.containsi("pure") });

m.connect(MIDIClient.destinations.detectIndex { |dst| dst.device.containsi("superc") });

MIDIdef.sysex(\a, { |... args| args.postln });

Then I need to go to qjackctl and connect Pd's MIDI output to Pd's MIDI input.

At this point, I can send MIDI from either Pd or SuperCollider, and both SC and Pd will respond.

Pd patch, implementing "stream of bytes" (btw I tested both [sysexin] and [midiin], no difference in the results):

sysex-stream.pd

(Note also: Because I am collecting the MIDI-in bytes into a list, I also checked whether there is a maximum list length. There may be, but we won't hit any such limit in these tests -- I could go up to 5000 numbers in a list, no problem. So, in case of any message truncation, it must be happening in the MIDI objects, not in list handling.)

The test case will be to send a packet of n 14-bit integers (MSB first). The total packet size will be n*2 + 2, so if we want to produce exactly b bytes, n = (b / 2) - 1.

SuperCollider sending:

(
f = { |n|
    var out = Int8Array[240];
    n.do { |i|
        out = out.add(i >> 7 & 127).add(i & 127);
    };
    out.add(247);
};
)

// small packet
m.sysex(f.value(10));  // SC OK, Pd OK (22 bytes)

// 512 bytes
m.sysex(f.value(255));  // both OK

// 514 bytes
m.sysex(f.value(256));  // SC OK, Pd did not print

And my results for Pd sending:

• 22 bytes (n = 10): Both OK
• 512 bytes (n = 255): Both OK
• 514 bytes (n = 256): SC did not respond. Pd printed only 512 bytes (truncated).

Conclusions:

• When SC sent 514 bytes, SC received 514 bytes, so we know SC is sending a complete packet.
  • Pd did not respond, suggesting that both [sysexin] and [midiin] simply stopped listening after 512 bytes, did not process the ending delimiter, and thus did not output the packet at all.
• When Pd was asked to send 514 bytes, Pd received only 512 bytes. This suggests that [midiout] is buffering the incoming bytes, waiting for a complete packet before sending, but it must have a limit of 512 bytes, causing the message to be truncated. (And SC didn't respond at all, which suggests that the closing delimiter was never sent.)

So again... in principle, you're correct -- it's technically possible for some [midiout] object to relay the data out, byte by byte. But neither [midiin] nor [midiout] are implemented that way.

hjh

Last one for now -- I found that SuperCollider can send long sysex messages.

This means [midiout], [midiin] and [sysexin] in Pd seem unable to handle long sysexes.

It's a kludge, but you could run a background SuperCollider process, and send open sound control to it when you need it to send one of these big sysex messages.

hjh

@lo94 said:

I think I understand what you're saying but because of the formatting of the template file I'm having a hard time applying it to this use case (as I'm used to relatively short sysex messages like the one shown in the forum thread, where this one is massive). I attached the file in case it would be useful/relevant to see

In theory it would be something like the attached. But, I haven't gotten the entire sysex to be received on the other end. (EDIT: When you click 'dir' at the top, the number box will show 240... not done in this screenshot.)

sysex.pd

The right hand [print] shows that the entire file is being sent to [midiout].

For receiving, I tried two tests:

• SuperCollider, like before. If I send separate messages [240 1 2( and [3 4 247(, then SC's sysex handler does assemble them into a single packet -- so there should be no problem with sending sysex in chunks. But when I send the entire file (in chunks), SC never receives the terminating 247 -- then, if I send 247 on its own, it outputs all of the bytes stored so far!

• Pd. In qjackctl (I'm in Linux), I connected Pd's MIDI output port to its own input port, and try to collect sysex bytes in Pd itself. ([midiin] gives access to raw incoming MIDI bytes, which SC doesn't.) [midiin] gets only 512 bytes.

There are two possible explanations for the 512-byte limit. One is that perhaps [midiin] buffers the input -- but I think we can reject that explanation, because [midiin] outputs the bytes one by one, suggesting no internal processing.

The other would be that [midiout] is trying to collect an entire sysex packet, and that this imposes a limit of 512 bytes. The fact that neither Pd nor SC got the 247 terminator is consistent with this.

Unfortunately, that's bad news for your use case. There appears to be no alternative to [midiout] here, and AFAICS it would require a software change to extend the outgoing packet size.

hjh

Thanks for the replies. I will have to spend some time staring at this as I am relatively new to the world of sysex. I've sent sysex messages for simple cases like program changes, but with this one I have a rather large file as I realized a workaround to a potential limitation with my MIDI Controller (Novation SL25 MK1) with templates (limited number) and PureData is that PureData could basically just send the giant template sysex file to the controller based on MIDI messages. These templates are stored in temp memory, so basically all my custom PureData patches don't need to actually be saved within the synth, and I can save the template storage for my hardware synths. Essentially just pass the template file to the synth as needed from PureData, which automatically loads it up in temp memory when it receives it

I think I understand what you're saying but because of the formatting of the template file I'm having a hard time applying it to this use case (as I'm used to relatively short sysex messages like the one shown in the forum thread, where this one is massive). I attached the file in case it would be useful/relevant to see

FirstButtonCC.syx

@whale-av said:

@lo94 [midiout] requires decimal....... and a sysex file will be in Hex (almost certainly)....... https://forum.pdpatchrepo.info/topic/377/sysex/32

Well... maybe.

For instance, https://mido.readthedocs.io/en/latest/syx.html has a section for "SYX files -- reading and writing," and another section for "Plain Text Format" sysex files. So, apparently, there are two ways that sysex could be written to disk -- a sequence of bytes (numeric), or as text, where each byte has been encoded as two hex digits.

"read_syx_file() determines which format the file is by looking at the first byte. It raises ValueError if file is plain text and byte is not a 2-digit hex number."

Sysex messages are supposed to be introduced by a byte F0, and terminated by F7. So you could do as this site suggests -- read one byte, and if it == 240, then treat it as a byte stream.

I have no idea whether binary or hex-coded sysex dumps are more common in the wild, only that at least one reference page implies that both exist.

It is possible that a whole sysex dump can be passed in one message (still decimal though) to [midiout], but I have no means for testing that.

I just tested this by having SuperCollider print out sysex messages -- that is, there is no need to have a MIDI device receive sysex.

MIDIClient.init;

// '5' is Pd's MIDI source in the device list
// may vary in your system
MIDIIn.connect(0, 5);

MIDIdef.sysex(\y, { |... args| args.postln });

[240 1 2 3 4 247( --> [midiout] then caused SC to print: [ Int8Array[ -16, 1, 2, 3, 4, -9 ], 8454145 ] = data block, and device UID (where -16 = 0xF0 and -9 = 0xF7, as signed 8-bit ints).

hjh

@lo94 [midiout] requires decimal....... and a sysex file will be in Hex (almost certainly)....... https://forum.pdpatchrepo.info/topic/377/sysex/32
Hex/decimal conversion can be done easily in Pd since version 0.48
Hex.pd
You will need to drip [list-drip] the symbols into the "symbol representation" box and group the messages that you send to [midiout] starting a message when you drip 240 (F0 hex) and ending the message when 247 (F7 hex) has dripped.... rinse and repeat (see link above).
[list-drip] [select] and the messages [set( and [add2( will be your friends.
handle.zip (Hex.pd included)

It is possible that a whole sysex dump can be passed in one message (still decimal though) to [midiout], but I have no means for testing that.
David.

@fina said:

I'm especially worried about multi mapping the controls to different pages/layers and how the controllers behave if the pot/encoder is in a different position

This can be a pain, faders (unless they are motorized) will jump, if you are at midi value 10 and switch too a different mapping that has that fader at max, when you move it it will jump down to 10, quite annoying but can be exploited and was a common trick on the early synths with patch memory that used analog controls, but this was generally more a hindrance than a help. Most modern controllers use endless rotary encoders which will update their values, in absolute mode you map out the controls on the controller itself and make presets, switch the mapping and the new values are there so no jump when you turn the knob but this has the disadvantage that they generally are limited to the low resolution 0-127 midi values. In relative mode the encoder sends only a plus or minus value so you can have unlimited resolution but you need to do more in pd since you need to add or subtract those plus or minus values from a stored value.

Personally I find mapping controls on a controller to be slow and I do need greater resolution so I do it all the mapping and the like in pd with relative mode, I made some abstractions to take care of all the work and they do some useful things like the first tick is ignored and just highlights the parameter being edited so when you forget what knob does what you can find your way without altering anything that is going on. This also means you are not limited by the controllers memory for mappings, I just do that in the patch so I have a virtually unlimited number of mappings. I use an Arturia Beatstep as a controller, 16 pads and 16 encoders, so each pad selects a mapping giving me 256 parameters I can edit or if I somehow needed more I could arrange the pads as 8 banks of 8 and 1024, but I have yet to need to do that. It also has the sequencer mode which is very limited but handy as an easy way to easily test sounds out. I can upload these abstractions if you decide on going with relative encoders, been on my list to get those uploaded but I tend to drag my feet on documentation so they have yet to get a proper upload, they have been uploaded a few times in various threads just not everything with actual documentation.

One thing to keep in mind, some controllers have software editors to make it easier to map stuff out, these software editors often do not work in linux even in Wine, so if you use linux you will want to make sure that the sysex commands have been published either by the maker or by someone who has sat down and figured them all out. For what ever reason some companies just will not release the sysex (Arturia being one of them). It is not terribly difficult to figure out the sysex on your own, just time consuming.

@oid How raw is [midiout].... ??
Did you establish that it cannot build a midi2 message...?
https://forum.pdpatchrepo.info/topic/13662/raw-midi-midiout

Just wondering because it can build NRPN... but does it have list constraints? RAW suggests "anything" in the message string that is just going to be converted to HEX.
It looks as though midi2 can carry both midi1 and midi2 messages, so maybe there is just one initial status byte to change.
David.

@jaffasplaffa I don't see why you don't just use the reference .txt you have, that seems way easier than having to export a sysex and compare it for every parameter.
Anyways, the sysex will be serial so "which parameter" it is will be based on what # of message it is in the input stream when you read the sysex. (though idk what you're going to use for getting the sysex file in - [mrpeach/binfile], or sent over midi using something like sysex librarian https://www.snoize.com/SysExLibrarian/)

the full 32-patch format is different than than the single-patch format though so the parameters will have different positions in each.