Hey all,
Sorry for overflowing these threads with questions haha. Been posting a lot lol

Was curious if there's an easy way to send an entire .syx file via midi. When looking through the forums for answers I mostly found questions pertaining to sending individual sysex messages, but I'm looking to simply route a .syx file like MIDIOX or similar software would. Would I have to parse the message somehow or is there a ready made abstraction for this sort of thing?

As always, thanks in advance

@lo94 said:

Was curious if there's an easy way to send an entire .syx file via midi.

A [file handle] object should give you access to the bytes in the file. Then pass those to [midiout] I guess. I think you'd just dump the bytes sequentially into [midiout]'s left inlet.

hjh

@ddw_music I'll give this a shot. Thanks, didn't even know about that object

@lo94 [midiout] requires decimal....... and a sysex file will be in Hex (almost certainly)....... https://forum.pdpatchrepo.info/topic/377/sysex/32
Hex/decimal conversion can be done easily in Pd since version 0.48
Hex.pd
You will need to drip [list-drip] the symbols into the "symbol representation" box and group the messages that you send to [midiout] starting a message when you drip 240 (F0 hex) and ending the message when 247 (F7 hex) has dripped.... rinse and repeat (see link above).
[list-drip] [select] and the messages [set( and [add2( will be your friends.
handle.zip (Hex.pd included)

It is possible that a whole sysex dump can be passed in one message (still decimal though) to [midiout], but I have no means for testing that.
David.

@whale-av said:

@lo94 [midiout] requires decimal....... and a sysex file will be in Hex (almost certainly)....... https://forum.pdpatchrepo.info/topic/377/sysex/32

Well... maybe.

For instance, https://mido.readthedocs.io/en/latest/syx.html has a section for "SYX files -- reading and writing," and another section for "Plain Text Format" sysex files. So, apparently, there are two ways that sysex could be written to disk -- a sequence of bytes (numeric), or as text, where each byte has been encoded as two hex digits.

"read_syx_file() determines which format the file is by looking at the first byte. It raises ValueError if file is plain text and byte is not a 2-digit hex number."

Sysex messages are supposed to be introduced by a byte F0, and terminated by F7. So you could do as this site suggests -- read one byte, and if it == 240, then treat it as a byte stream.

I have no idea whether binary or hex-coded sysex dumps are more common in the wild, only that at least one reference page implies that both exist.

It is possible that a whole sysex dump can be passed in one message (still decimal though) to [midiout], but I have no means for testing that.

I just tested this by having SuperCollider print out sysex messages -- that is, there is no need to have a MIDI device receive sysex.

MIDIClient.init;

// '5' is Pd's MIDI source in the device list
// may vary in your system
MIDIIn.connect(0, 5);

MIDIdef.sysex(\y, { |... args| args.postln });

[240 1 2 3 4 247( --> [midiout] then caused SC to print: [ Int8Array[ -16, 1, 2, 3, 4, -9 ], 8454145 ] = data block, and device UID (where -16 = 0xF0 and -9 = 0xF7, as signed 8-bit ints).

hjh

Thanks for the replies. I will have to spend some time staring at this as I am relatively new to the world of sysex. I've sent sysex messages for simple cases like program changes, but with this one I have a rather large file as I realized a workaround to a potential limitation with my MIDI Controller (Novation SL25 MK1) with templates (limited number) and PureData is that PureData could basically just send the giant template sysex file to the controller based on MIDI messages. These templates are stored in temp memory, so basically all my custom PureData patches don't need to actually be saved within the synth, and I can save the template storage for my hardware synths. Essentially just pass the template file to the synth as needed from PureData, which automatically loads it up in temp memory when it receives it

I think I understand what you're saying but because of the formatting of the template file I'm having a hard time applying it to this use case (as I'm used to relatively short sysex messages like the one shown in the forum thread, where this one is massive). I attached the file in case it would be useful/relevant to see

FirstButtonCC.syx

@lo94 said:

I think I understand what you're saying but because of the formatting of the template file I'm having a hard time applying it to this use case (as I'm used to relatively short sysex messages like the one shown in the forum thread, where this one is massive). I attached the file in case it would be useful/relevant to see

In theory it would be something like the attached. But, I haven't gotten the entire sysex to be received on the other end. (EDIT: When you click 'dir' at the top, the number box will show 240... not done in this screenshot.)

sysex.pd

The right hand [print] shows that the entire file is being sent to [midiout].

For receiving, I tried two tests:

• SuperCollider, like before. If I send separate messages [240 1 2( and [3 4 247(, then SC's sysex handler does assemble them into a single packet -- so there should be no problem with sending sysex in chunks. But when I send the entire file (in chunks), SC never receives the terminating 247 -- then, if I send 247 on its own, it outputs all of the bytes stored so far!

• Pd. In qjackctl (I'm in Linux), I connected Pd's MIDI output port to its own input port, and try to collect sysex bytes in Pd itself. ([midiin] gives access to raw incoming MIDI bytes, which SC doesn't.) [midiin] gets only 512 bytes.

There are two possible explanations for the 512-byte limit. One is that perhaps [midiin] buffers the input -- but I think we can reject that explanation, because [midiin] outputs the bytes one by one, suggesting no internal processing.

The other would be that [midiout] is trying to collect an entire sysex packet, and that this imposes a limit of 512 bytes. The fact that neither Pd nor SC got the 247 terminator is consistent with this.

Unfortunately, that's bad news for your use case. There appears to be no alternative to [midiout] here, and AFAICS it would require a software change to extend the outgoing packet size.

hjh

Also, I should clarify -- in that version of the patch, the chunk settings are ridiculous. I was trying to see if maybe the problem was flooding [midiout] -- would it make a difference if I slowed it to a trickle? Unfortunately it didn't -- so you could change those to more reasonable settings, like [128( instead of [32(, and [delay 5].

hjh

Last one for now -- I found that SuperCollider can send long sysex messages.

This means [midiout], [midiin] and [sysexin] in Pd seem unable to handle long sysexes.

It's a kludge, but you could run a background SuperCollider process, and send open sound control to it when you need it to send one of these big sysex messages.

hjh

@ddw_music Long sysex messages work just fine if you treat them as what they are and what pd is quite good at dealing with, streams. I generally read them right into an array with each byte banging a counter to increment the array and send them out by iterating the array with a counter banged by an until. My guess would be this is a limitation of the midi objects and how they convert messages to streams, but messages are not meant for streams anyways so I do not think this would be a bug?

@oid said:

@ddw_music Long sysex messages work just fine if you treat them as what they are and what pd is quite good at dealing with, streams.

I do understand what you're saying, and I began with the same assumption -- that [midiout] should be just a low-level conduit -- byte goes in, byte gets sent out.

Unfortunately it doesn't work that way in practice.

My test scenario looks like this. Launch SuperCollider and Pure Data. It helps in my case that I'm testing in Linux, where I can create and remove MIDI connections arbitrarily in qjackctl's ALSA tab.

SC code:

MIDIClient.init;

MIDIIn.connect(0, MIDIClient.sources.detectIndex { |src| src.device.containsi("pure") });

m = MIDIOut(0).connect(MIDIClient.destinations.detectIndex { |dst| dst.device.containsi("pure") });

m.connect(MIDIClient.destinations.detectIndex { |dst| dst.device.containsi("superc") });

MIDIdef.sysex(\a, { |... args| args.postln });

Then I need to go to qjackctl and connect Pd's MIDI output to Pd's MIDI input.

At this point, I can send MIDI from either Pd or SuperCollider, and both SC and Pd will respond.

Pd patch, implementing "stream of bytes" (btw I tested both [sysexin] and [midiin], no difference in the results):

sysex-stream.pd

(Note also: Because I am collecting the MIDI-in bytes into a list, I also checked whether there is a maximum list length. There may be, but we won't hit any such limit in these tests -- I could go up to 5000 numbers in a list, no problem. So, in case of any message truncation, it must be happening in the MIDI objects, not in list handling.)

The test case will be to send a packet of n 14-bit integers (MSB first). The total packet size will be n*2 + 2, so if we want to produce exactly b bytes, n = (b / 2) - 1.

SuperCollider sending:

(
f = { |n|
    var out = Int8Array[240];
    n.do { |i|
        out = out.add(i >> 7 & 127).add(i & 127);
    };
    out.add(247);
};
)

// small packet
m.sysex(f.value(10));  // SC OK, Pd OK (22 bytes)

// 512 bytes
m.sysex(f.value(255));  // both OK

// 514 bytes
m.sysex(f.value(256));  // SC OK, Pd did not print

And my results for Pd sending:

• 22 bytes (n = 10): Both OK
• 512 bytes (n = 255): Both OK
• 514 bytes (n = 256): SC did not respond. Pd printed only 512 bytes (truncated).

Conclusions:

• When SC sent 514 bytes, SC received 514 bytes, so we know SC is sending a complete packet.
  • Pd did not respond, suggesting that both [sysexin] and [midiin] simply stopped listening after 512 bytes, did not process the ending delimiter, and thus did not output the packet at all.
• When Pd was asked to send 514 bytes, Pd received only 512 bytes. This suggests that [midiout] is buffering the incoming bytes, waiting for a complete packet before sending, but it must have a limit of 512 bytes, causing the message to be truncated. (And SC didn't respond at all, which suggests that the closing delimiter was never sent.)

So again... in principle, you're correct -- it's technically possible for some [midiout] object to relay the data out, byte by byte. But neither [midiin] nor [midiout] are implemented that way.

hjh

Ahhh hah... this may be specific to the ALSA MIDI backend.

s_midi_alsa.c:

//the maximum length of input messages
#ifndef ALSA_MAX_EVENT_SIZE
#define ALSA_MAX_EVENT_SIZE 512
#endif

But I can't quickly find where Pd's source is interfacing with CoreMIDI (macOS) or ... whichever MIDI framework in Windows.

In any case, it's good to be aware of the behavior, in case a patch needs to be cross-platform compatible.

hjh