Hi there, I'm working on a project that involves streaming audio from an AVPlayer video player object into libpd. For the process loop of the tap, I used PdAudioUnits render callback code as a guide; but I realized recently that the audio format expected by libpd is not the same as the audio coming from the tap — that is, the tap is providing two buffers of non-interleaved audio data in the incoming AudioBufferList, whereas libpd expects interleaved samples. Does anyone know of a way I can work around this?

I think that I need to somehow create a new AudioBufferList or float buffer and interleave the samples in place; but that seems expensive to me. If anyone could give me some pointers I would greatly appreciate it!

static void tap_ProcessCallback(MTAudioProcessingTapRef tap, CMItemCount numberFrames, MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut, CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut)
{
    OSStatus status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut);
    if (noErr != status) {
        NSLog(@"Error: MTAudioProcessingTapGetSourceAudio: %d", (int)status);
        return;
    }
    
    TapProcessorContext *context = (TapProcessorContext *)MTAudioProcessingTapGetStorage(tap);
    
    // first, create the input and output ring buffers if they haven't been created yet
    if (context->frameSize != numberFrames) {
        NSLog(@"creating ring buffers with size: %ld", (long)numberFrames);
        createRingBuffers((UInt32)numberFrames, context);
    }
    
    //adapted from PdAudioUnit.m
    float *buffer = (float *)bufferListInOut->mBuffers[0].mData;
    
    if (context->inputRingBuffer || context->outputRingBuffer) {
        
        // output buffer info from ioData
        UInt32 outputBufferSize = bufferListInOut->mBuffers[0].mDataByteSize; // * 2 solved faint avplayer issue
        UInt32 outputFrames = (UInt32)numberFrames;
//        UInt32 outputChannels = bufferListInOut->mBuffers[0].mNumberChannels;
        
        // input buffer info from ioData *after* rendering input samples
        UInt32 inputBufferSize = outputBufferSize;
        UInt32 inputFrames = (UInt32)numberFrames;
        UInt32 framesAvailable = (UInt32)rb_available_to_read(context->inputRingBuffer) / context->inputFrameSize;
                
        //render input samples
        
        while (inputFrames + framesAvailable < outputFrames) {
            // pad input buffer to make sure we have enough blocks to fill auBuffer,
            // this should hopefully only happen when the audio unit is started
            rb_write_value_to_buffer(context->inputRingBuffer, 0, context->inputBlockSize);
            framesAvailable += context->blockFrames;
        }
        rb_write_to_buffer(context->inputRingBuffer, 1, buffer, inputBufferSize);
        
        // input ring buffer -> context -> output ring buffer
        char *copy = (char *)buffer;
        while (rb_available_to_read(context->outputRingBuffer) < outputBufferSize) {
            rb_read_from_buffer(context->inputRingBuffer, copy, context->inputBlockSize);
            [PdBase processFloatWithInputBuffer:(float *)copy outputBuffer:(float *)copy ticks:1];
            rb_write_to_buffer(context->outputRingBuffer, 1, copy, context->outputBlockSize);
        }
        
        // output ring buffer -> audio unit
        rb_read_from_buffer(context->outputRingBuffer, (char *)buffer, outputBufferSize);
    }
}

@whale-av, here is a log running pd with -lib Gem -verbose.
tried both 32bit and 64bit pd 0.48-1...

tried ./Gem.m_i386 and failed
tried ./Gem.dll and failed
tried ./Gem/Gem.m_i386 and failed
tried ./Gem/Gem.dll and failed
tried ./Gem.pd and failed
tried ./Gem.pat and failed
tried ./Gem/Gem.pd and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem.m_i386 and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem.dll and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem/Gem.m_i386 and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem/Gem.dll and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem.pd and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem.pat and failed
tried C:/Users/Raphael Isdant/Documents/Pd/externals/Gem/Gem.pd and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem.m_i386 and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem.dll and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem/Gem.m_i386 and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem/Gem.dll and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem.pd and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem.pat and failed
tried C:/Users/Raphael Isdant/AppData/Roaming/Pd/Gem/Gem.pd and failed
tried C:/Program Files/Common Files/Pd/Gem.m_i386 and failed
tried C:/Program Files/Common Files/Pd/Gem.dll and failed
tried C:/Program Files/Common Files/Pd/Gem/Gem.m_i386 and failed
tried C:/Program Files/Common Files/Pd/Gem/Gem.dll and failed
tried C:/Program Files/Common Files/Pd/Gem.pd and failed
tried C:/Program Files/Common Files/Pd/Gem.pat and failed
tried C:/Program Files/Common Files/Pd/Gem/Gem.pd and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem.m_i386 and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem.dll and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem/Gem.m_i386 and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem/Gem.dll and succeeded
D:\\pd-0.48-1.windows.64bit\\extra\\Gem\\Gem.dll: couldn't load
tried D:/pd-0.48-1.windows.64bit/extra/Gem.pd and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem.pat and failed
tried D:/pd-0.48-1.windows.64bit/extra/Gem/Gem.pd and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem.m_i386 and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem.dll and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem/Gem.m_i386 and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem/Gem.dll and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem.pd and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem.pat and failed
tried D:/pd-0.48-1.windows.64bit/doc/5.reference/Gem/Gem.pd and failed
Gem: can't load library```

I realized recently that Gem has some very similar (and nice) shader examples, like a game of life, or a multi pass rendering (I learned that this is what I am doing with the effect chain in the video player).
And the timbreID library has some nice Gem patches for visualizing audio.
And Gem has of course a video player.

I ask myself now, because I do not want to reinvent the wheel, what are the advantages and disadvantages if I compare Gem to Ofelia?

I did some experiments with Gem some time ago but at a certain point I got stuck, whether because of my lacking knowledge or because the patch was getting slow.

So it could be possible to do exactly the same with both libraries?

Here I found a very old paper about Gem that states Gem is using openGL up to 1.2.
https://puredata.info/downloads/gem/documentation/manual/pub/zmoelnig2004gem.pdf
Has that changed? Or could that be a difference? I think the openGL versions that Ofelia can use depend only on the hardware.

What I like about ofelia so far is (in comparison to Gem):
To compile the patch as a standalone application.
To make nice interfaces.
(but I think those points are not openGL related...)
That it gives access to lua and open frameworks (which could make it easier to do complex graphics calculations / interfaces?).

Is anyone else in the position to compare those libraries, or has an opinion about them?

@Joseph-Mikkelson That's great.
So I looked in [pd G] and separated out the very left hand part...... see...... [left part]
Then I tidied it up....... [left-part-tidy]
I left in the string / cord that looks like it might be a mistake.
Then I copied all the audio part into another patch called [buffer]
And then I used that to make a new left part........ [left-part-buffer-abstraction]
That "mistake"...? cord I have left out of the last patch.

In that last patch is [buffer 1 0]
The 1 is an argument, and replaces $1 in an object inside [buffer]
For the next (second) part you would use [buffer 2 0] and the delwrite inside would become [delwrite~ buffer-2 1000]......... and then arguments 3, 4, etc. so that the delay names are all different, and the messages receiving the frequency etc. are different for each one too.
That solves the buffer created multiple times problem, that you have solved already I see..

I have added a second argument to [buffer].
When it is 0 [buffer] will use [vline~] and when it is 1 it will use [del]
So actually for the 2nd part of [pd G] you would use [buffer 2 1] so that [del] is used instead of [vline~]

The $1 in the envelope message to [vline~] is receiving a bang when it needs a float...... I don't know what you intended there. The output of [expr] is sending the value for $1, and so banging the message through, and I don't think you need the extra bang at the same time.
A $1 in a message does not take the argument of the abstraction. It takes the first float or symbol sent into the message.
That is the $1 that is complaining in the terminal....... because it has not been sent a value, just a bang.

I hope that helps you to see where any other mistakes might be.
Abstractions are great. If you change [buffer] and save it then every copy will change, so if you want to add some effects into it...... that is very easy.... you only have to do it once.
all_that.zip ..... don't forget to unzip.
David.

Generally, the "blocksize" in the audio setting really is the hardware buffer size and not Pd's global blocksize (which is always 64 samples).

Modern ASIO drivers are multiclient which means that the device can be used by more than one app at the same time. When you set the buffer size in your device app (e.g. Focusrite MixControl) this will set the internal blocksize and consequently the minimum latency possible for all clients. Usually you would want to set the internal buffer size as low as possible (without getting audio glitches).

Individiual clients (e.g. a Pd instance) can request a buffer size from the device. This can't be smaller than the above mentioned internal buffer size, but it can be larger (if you need extra latency). Most DAWs control the latency via the buffer size while Pd has it's own buffering mechanism ("delay").

Note that for Pd the hardware buffer size affects the overall timing resolution (when DSP is on). Pd will calculate several blocks of 64 samples as fast as possible until it has enough samples to send to the device. If the hardware buffer size is 256, Pd will calculate 4 blocks in a row (4 * 64 = 256) and then wait until the device asks for another 256 samples. This causes jitter! (Try [bang~] with [realtime] and [print] to see it yourself). The lower the hardware buffer size, the less jitter you get, so generally it's better to keep "blocksize" as low as possible and rather increase "delay" if you need more latency.

BTW, in case of old single client ASIO drivers, "blocksize" would directly set the hardware buffer size of the device. This is still the case e.g. with ASIO4ALL.

I tried making your patch and it worked if the block size was lower than 256

edit: it's because your write buffer length is less than 256 samples. Increase past 1 ms and it works.
not sure why it would work with block sizes of 64 & 128 but not with 256 though.. and it seems like it should work anyways..

edit2: For those curious (I was) after digging around what happens is that it is possible for the writer to write over its starting point in the buffer before the reader has a chance to read those values if the buffer size < the block size. Because the "real" delay time (that is, the starting read point in the circular buffer) is supposed to be < the write buffer size but is also supposed to be >= the block size, the delay is set to the buffer size so when it reads the buffer it starts at the buffer size and reads the buffer circularly until the block finishes.
In this specific case, because 1 ms is 176 samples, pd rounds the write buffer size up to 240 (for some reason it adds 64 samples). (And snapshot~ takes the last sample in a block)

@jeffgrips I have it running with vanilla in Windows7.... but it was hard...... and it's a long story....
I cannot see any reason that it would not work for widows10.
I have not found any other easier better solution.
(((SEE below though for the fact that the -lib needs to be declared...!
That could be your only problem? But I doubt it..!!! )))

I used the 64-bit GEM from here (0.94-test3.zip)..... https://github.com/avilleret/Gem/releases/

I unzipped it and put it in a folder (which I called "dot") on my desktop.

Because I am using many different versions of Pure Data I left it there, and I start Pd with a shortcut that also points to the GEM folder and loads it as a library (It has to be loaded at startup as a library...... and that could be why you don't have it working.... but probably not...).
This is the contents of the line "target" in the shortcut.......

"C:\Users\David\Desktop\Pd Vanilla\pd-0.47-1.msw\pd\bin\pd.exe" -lib C:\Users\David\Desktop\dot\0.94-test3/Gem -font-size 10 -font-face "Consolas"

Be careful with forward / backward slashes.

I am pretty sure that I tipped the contents of the "QT" folder out into the "0.94-test3 folder" where gem.dll is (from within the "QT" folder inside "0.94-test3").
If you find some .dlls are missing then post back with a list, and I can upload them....... but you might not have the correct C++ redistributables from Microsoft.

But if you want to start Pd normally by double-clicking a patch then you will need to declare the library in Pd.. edit....preferences.... startup......
If you have GEM in the extra folder you should only need to put "-lib Gem" where you see that I have put "-noautopatch".......

The gem abstractions could not be found though (the "parts of Gem... .like [pix_image] pix_video] [gemwin] etc. worked fine) I copied the contents of the Gem folder from "extended" into a folder on the desktop called "GemAbstr" and set a path to it in Pd preferences.
I removed gem.dll from that folder (no conflicts....!).
The abstractions now create, but the help files for embedded objects cannot be found (different "search" for embedded objects) so I have to open my "GemAbstr" folder and open them manually.

Everything works. All the patches from https://puredata.info/downloads/extended-view-toolkit create and run correctly. The camera on my computer even works (I had never had it working before).
And it works with Spout.

If you go this route then I wish you good luck.......

@zerino has confirmed that it can work with widows10........ https://forum.pdpatchrepo.info/topic/11450/pix-and-geometric-objects-not-being-loaded-in-pd-vanilla/7
........and resolved the "help" file problem.....

For the time being you are lucky that you are using Windows.
This solution is not available for other systems.
David.

Ok, here in Puppy-Linux apt-get is not installed.
Now I installed libftgl2_2.1.3 successfully with the package manager.

Doing the same as before
[Gem] creates! Yes!
In the commandline (linux-console) [shortened]:

pd-l2ork -noprefs -nostdpath -nrt -verbose
Pd-L2Ork version 2.4.6 (20171213-rev.46ffba3)
compiled 22:30:53 Dec 13 2017
port 5401
guidir is /usr/lib/pd-l2ork/bin
"/usr/lib/pd-l2ork/bin"/nw/nw --user-data-dir="/root/.config/purr-data" "/usr/lib/pd-l2ork/bin" 5401 localhost pd-l2ork "/usr/lib/pd-l2ork/bin" xf3c100Waiting for connection request... 
[0426/002010.733670:WARNING:chrome_main_delegate.cc(565)] final extension: 
... connected
/etc/pd/gem.conf: No such file or directory
/root/.config/pure-data/gem.conf: No such file or directory
./gem.conf: No such file or directory
load plugins 'film' in '/usr/lib/pd-l2ork/extra/Gem/'
pattern : /usr/lib/pd-l2ork/extra/Gem/gem_film*.so
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_filmAVIPLAY.so'!
library loading returned: dlerror 'libaviplay-0.7.so.0: cannot open shared object file: No such file or directory'
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_filmGMERLIN.so'!
library loading returned: dlerror 'libgmerlin_avdec.so.1: cannot open shared object file: No such file or directory'
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_filmMPEG3.so'!
library loading returned: dlerror 'libmpeg3.so.2: cannot open shared object file: No such file or directory'
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_filmQT4L.so'!
library loading returned: dlerror 'libquicktime.so.2: cannot open shared object file: No such file or directory'
[...]
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_imageMAGICK.so'!
library loading returned: dlerror 'libMagick++-6.Q16.so.5: cannot open shared object file: No such file or directory'
[...]
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_imageMAGICK.so'!
library loading returned: dlerror 'libMagick++-6.Q16.so.5: cannot open shared object file: No such file or directory'


[...]
dylib loading file '/usr/lib/pd-l2ork/extra/Gem/gem_recordQT4L.so'!
library loading returned: dlerror 'libquicktime.so.2: cannot open shared object file: No such file or directory'
[...]

There are some more libs missing, maybe I should install them too?

All the examples of Gem seem to work now!
Even the teapod and openGL stuff.

And my final patches, also work. So nice!

Now I will review what those flags actually do:
https://puredata.info/docs/faq/commandline

But I am wondering, if I was lucky, that Gem is programmed for printing messages in the commandline and other non-working libraries probably won't do so?

Thank you so much

Welcome to Purr Data
warning: your system's font stack is not optimal
Pd has started the GUI
canvasinfo: v0.1
stable canvasinfo methods: args dir dirty editmode vis
pdinfo: v.0.1
stable pdinfo methods: dir dsp version
classinfo: v.0.1
stable classinfo methods: size
objectinfo: v.0.1
stable objectinfo methods: class
[import] $Revision: 1.2 $
    [import] is still in development, the interface could change!
    compiled against Pd-l2ork version 2.4.6 (20171213-rev.46ffba3)
input channels = 2, output channels = 2
working directory is /usr/lib/pd-l2ork/extra/Gem
input channels = 2, output channels = 2
audio buffer set to 50
opened input device name hw:0
configuring sound input...
Sample width set to 4 bytes
configuring sound output...
Sample width set to 4 bytes
error: audio I/O dropout
tried but couldn't sync A/D/A
tried /usr/lib/pd-l2ork/extra/Gem/Gem.l_ia64 and failed
tried /usr/lib/pd-l2ork/extra/Gem/Gem.pd_linux and succeeded
verbose( 1):/usr/lib/pd-l2ork/extra/Gem/Gem.pd_linux: libftgl.so.2: cannot open shared object file: No such file or directory
tried /usr/lib/pd-l2ork/extra/Gem/Gem.pd and failed
tried /usr/lib/pd-l2ork/extra/Gem/Gem.pat and failed
tried /usr/lib/pd-l2ork/extra/Gem/Gem/Gem.pd and failed
 Gem
... couldn't create

libftgl is missing?

@jancsika said:

ls /usr/lib/pd-l2ork/extra/Gem/Gem*

ls /usr/lib/pd-l2ork/extra/Gem/Gem*
/usr/lib/pd-l2ork/extra/Gem/Gem.la
/usr/lib/pd-l2ork/extra/Gem/Gem-meta.pd
/usr/lib/pd-l2ork/extra/Gem/Gem.pd_linux
/usr/lib/pd-l2ork/extra/Gem/GemPrimer.pdf

du -h /usr/lib/pd-l2ork/extra/Gem/Gem*

du -h /usr/lib/pd-l2ork/extra/Gem/Gem*
4.0K /usr/lib/pd-l2ork/extra/Gem/Gem.la
4.0K /usr/lib/pd-l2ork/extra/Gem/Gem-meta.pd
5.6M /usr/lib/pd-l2ork/extra/Gem/Gem.pd_linux
568K /usr/lib/pd-l2ork/extra/Gem/GemPrimer.pdf