Does anybody know how to change the ofelia video example code so that i can use it like the gem [pixfilm] object?
I want to load videos with (kind of) [openpanel], have stop and play buttons with variable speed. And I want to build "effect chains": connect the video example object to a blur object for example.

i think it is possible after trying a bit.
what seems important:
-use photo JPG (or something similar) as the codec for the video. a good program for converting is http://www.squared5.com/
-use setPosition and not setFrame for scrubbing or playing reverse through the video (it is much faster)
-playing with setPosition is without sound
-for playing with setPosition the video needs to be paused
it is adapted from the video example and can be optimized a lot to work like the [pixfilm] object...
video4.pd

if type(window) ~= "userdata" then;
window = ofWindow();
end;
;
local canvas = pdCanvas(this);
local clock = pdClock(this, "setup");
local videoPlayer = ofVideoPlayer();
;
function ofelia.bang();
ofWindow.addListener("setup", this);
ofWindow.addListener("update", this);
ofWindow.addListener("draw", this);
ofWindow.addListener("exit", this);
window:setPosition(50, 100);
window:setSize(800 + 40, 600 + 40);
if type(window) ~= "userdata" then;
window = ofWindow();
end;
;
window:create();
if ofWindow.exists then;
clock:delay(0);
end;
end;
;
function ofelia.free();
window:destroy();
ofWindow.removeListener("setup", this);
ofWindow.removeListener("update", this);
ofWindow.removeListener("draw", this);
ofWindow.removeListener("exit", this);
end;
;
function ofelia.setup();
ofSetWindowTitle("Video Player");
ofBackground(0, 0, 0, 255);
end;
;
function ofelia.moviefile(path);
videoPlayer:load(path);
videoPlayer:setLoopState(OF_LOOP_NORMAL);
videoPlayer:setPaused(true);
end;
;
function ofelia.setSpeed(f);
videoPlayer:setSpeed(f / 100);
end;
;
function ofelia.play();
videoPlayer:play();
end;
;
function ofelia.stop();
videoPlayer:setFrame(0);
videoPlayer:setPaused(true);
end;
;
function ofelia.setPaused(f);
if f == 1 then pause = true;
else pause = false;
end;
videoPlayer:setPaused(pause);
end;
;
function ofelia.setFrame(f);
videoPlayer:setFrame(f);
end;
;
function ofelia.setPosition(f);
videoPlayer:setPosition(f);
end;
;
function ofelia.setVolume(f);
videoPlayer:setVolume(f / 100);
end;
;
function ofelia.update();
videoPlayer:update();
end;
;
function ofelia.draw();
ofSetHexColor(0xFFFFFF);
videoPlayer:draw(20, 20, 800, 600);
local outputList = {};
outputList[1] = videoPlayer:getPosition();
outputList[2] = videoPlayer:getCurrentFrame();
outputList[3] = videoPlayer:getTotalNumFrames();
return outputList;
end;
;
function ofelia.exit();
videoPlayer:close();
end;

here is the player with the (similar) effect from the ofelia example:
videowitheffect.pd

i think it would be really nice to have effects like that as independent abstractions, so that you can chain them together like with gem (a draw object at the end of the chain). otherwise the code could become a bit confusing...

@Jona Great work! I will add the ofVideoPlayer abstractions to ofelia soon.

@Cuinjune that is nice, the ofVideoPlayer works very well.
but i think for what i wanted i do not even need the abstractions (of course, still nice to have them).
for now i think it is fine to have the video player as a single object.
my idea was just to seperate ofWindow, ofSetup, and ofdraw. And to seperate the ofdraw chain into individual objects.
I have somehing like that in mind, not sure if it makes sense (it is just a non working dummy):
edit: i think it would not make sense do draw the videos if i want to multiply them, instead if of that i should use the getPixels() method and multiply the pixel values.
ofdummy.pd

here is kind of a pixel effect and the testvideo from the example decoded with the photo-jpeg codec:
videoeffect_pixelB.pd
fingers2.mov

when i recalculate 64*64 pixel with 25 fps it already starts to slow down (while the cpu load is only around 10 %), is there a way to make it more efficient?

@Cuinjune do you think it is possible to add shader filters like this to the video output? https://github.com/selflash/ofxSketchFilter

@Jona Processing a video frame pixel by pixel in real-time is a heavy operation and Lua being a scripting language is nowhere as fast as compiled languages like C/C++.

In order to create such video effects, the 3 ideas comes to my mind.

1. Compile ofelia externals using LuaJIT which will make Lua run much faster but then users will need to use old Lua 5.1 instead of the latest 5.3. (which currently ofelia uses) Also, I heard LuaJIT is almost no longer being maintained which is not good.

2. Create classes and functions for heavy pixels processing in C++ so users can use them in Lua script. (Just like how pdFilter() class currently works) The heavy parts will be done in C++ so it will be much faster.

3. Use Shader to process pixels from the GPU which I think will be the most efficient solution. I will add ofShader class abstractions to ofelia soon.

@Cuinjune thanks for your explanations.
so in comparison to Gem lua is the bottleneck in ofelia when processing video in real time?

1. Okay, that does not make much sense then...
2. That sounds like a good solution, at least for very heavy and often used tasks...
3. Having a shader class sounds great. Actually the sketch filters in the patch above are shader effects, i tried to understand with your shader examples how to implement them, but no luck so far...

I added 21 GLSL effects to the videoPlayer. And finally the pixel effect, which was the reason why i looked into GLSL.
I modified some effects that I found, so it is not my code. Still try to understand GLSL.
Would be nice to have a bigger video glsl collection.
ofelia_video_glsl.zip

I updated the collection. There is now also a magnifying glass effect that works with the mouse.
Fascinating what is possible with a few lines of code in GLSL.
@Cuinjune do you know if it would be difficult to make a shader chain? So that one shader effect is one [ofelia] object.
My question regarding that is how to use a shader output as the input for the next shader...

@Jona I don't know how easy or difficult it would be to design the shader effect chain but If you can implement multiple shader effects in one [ofelia] object, it won't be difficult to split it into multiple [ofelia] objects that work in chain. So first, I would suggest you try to implement two shader effect chain in one [ofelia] object and then try to split it into two [ofelia] objects.
I think this post might be helpful: https://gamedev.stackexchange.com/questions/22216/using-multiple-shaders

@Cuinjune thanks for the link and your suggestions. so it sounds like it is possible, but also not the easiest task.
It seems the easiest to put every effect of the desired chain into one shader.
But this sounds promising:
"You just bind one shader, render all the objects using that shader, then bind the next shader, render the objects using that one, etc.
You can have as many shader objects (shaders loaded into memory and compiled) as you want; only one can be bound (active) at a time."
They also write about framebuffers, perhaps that is what I would need for a video delay, but that is another topic...
And I am really new to GLSL, so not everything that is possible with openGL is possible for me (yet).
I will give it a try perhaps...

A few websites with GLSL content and code:
https://www.interactiveshaderformat.com/
https://thebookofshaders.com/
https://www.shadertoy.com/
https://gl-transitions.com/
http://glslsandbox.com/

i emulated the pixfilm metaimage object. I had to write the shader by myself this time, because I did not found anything similar. I am sure the code can be optimized, but it works
ofelia_video_glsl.zip

// fragment program 

uniform sampler2D Tex0;
uniform vec2 resolution;
uniform float colorFade;
uniform float MosaicNumber;

void main() 
{
    vec2 texCoords = vec2(floor(gl_TexCoord[0].s * MosaicNumber) / MosaicNumber, floor(gl_TexCoord[0].t * MosaicNumber) / MosaicNumber);
    vec4 color = texture2D(Tex0, texCoords);
    vec3 lum1 = vec3(0.299, 0.587, 0.114);
    color = vec4(vec3(dot(color.rgb, lum1) * (1.0 - (colorFade*(-1.0)+1.0)) + (color.rgb*(colorFade * (-1.0) + 1.0))), color.a);
    vec2 uv = (gl_FragCoord.xy - 20.0) / resolution;
    uv.y = uv.y * (-1.) + 1.;
    uv.xy *= MosaicNumber *(ceil((1.-uv.x)));
    vec4 color2 = texture2D(Tex0, mod(uv, 1.0));
    vec3 lum = vec3(0.299, 0.587, 0.114);
    color2 = vec4(vec3(dot(color2.rgb, lum) * (1.0 - colorFade) + (color2.rgb * colorFade)), color2.a);
    vec4 color3 = (color * color2);
    gl_FragColor = color3 * 2.;
}

and this are the effects that i collected so far ( also finding out about the possibilities...):

@Cuinjune apparently I could not figure out how to implement two shaders in one object.

Is there a simple trick or is it some openGL magic?

From your link:

"The simple answer is you change them between each draw call. Set a shader, draw a teapot, set another shader, draw another teapot."
But than the draw function needs to be faster than what is drawn on the screen? And somehow I need to get the render result from shader 1 to shader 2?

Or is that an easy solution?
"In terms of implementation, in every frame I use glUseProgram(1) and glUseProgram(2) to change shaders?"

The Framebuffer Object https://www.khronos.org/opengl/wiki/Framebuffer_Object sounds very interesting, but that is over my head...

Or perhaps it is really the the best solution to have all effects in one shader?
Something like that:

uniform int shaderType;
const int kShaderType_X = 1;
const int kShaderType_Y = 2;
const int kShaderType_Z = 3;

vec4 doShaderTypeX()
{
    // ... do whatever you need for the first shader type
}

vec4 doShaderTypeY()
{
    // ... do whatever you need for the second shader type
}

vec4 do ShaderTypeZ()
{
    // ... etc.
}

void main()
{
    vec4 result;
    if (shaderType == kShaderType_X)
    {
        result = doShaderTypeX();
    }
    else if (shaderType == kShaderType_Y)
    {
        result = doShaderTypeY();
    }
    else if (shaderType == kShaderType_Z)
    {
        result = doShaderTypeZ();
    }
}

I will think about it...

I think I found a solution for mixing shaders.
First I tried to use 3 fbo´s (one for each video effect). That did work, but not for mixing but for layering the results.
Now I use only one fbo. Perhaps that is really the simple trick
I am not totally sure because, there are some video artefacts, with some effect combinations.
But I think it is the interaction of the (for interaction wrongly programmed) shaders, because the artefacts are reproduceble and not random.
It is not really modular yet, but I think it could be possible to build such a system.
Right now it is kind of a glitch generator for videos...

ofelia_videoeffect_shader_3_effect_mix.zip

The glitches are gone and the mixer works now as expected
It is now also an effect chain, effect one goes into effect two and so on.
I think in the glitch version everything was mixed together...
Very nice that this is possible with Ofelia (Pure Data / Lua / Open Frameworks/ GLSL).
I use 4 fbo´s now (one for the video player and one for each effect), just did not know how to use them right.
But somehow I found some of the old glitches quite fascinating.
I would be happy about feedback and suggestions (how to improve the patch)...
One question: Right now the functionality depends on the creation order of the Ofelia objects, because that also seems to be the draw order.
Can I create a constant draw order somehow?
Edit: I think the fbo's are also the solution for a video delay, i just need to find out how to grab for example every 5th frame into 10 fbo buffers. Then I can mix the fbo's with a GLSL file...

GLSL_Video_Mixer.zip