Hello there!
I am working on a project which be looking to use a video camera to observe the movement of fireflies at dusk, and use this data to trigger audio in real-time.
IE: position of firefly light on X and Y axis (or within a particular area of a grid imposed on the video) could trigger a specific note or sample, brightness of light could affect volume, movement on X and Y axes could cause a glissando.
I am feeling good about using the data once it's actually in PD, but I could really benefit from some suggestions for good ways to get started in order to read the data from a live camera feed. I figure there's a good chance that something very similar to this has been done before, but can't find it in the forum. Any ideas will be gratefully received, thank you!

@Dizzy-Dizzy Easiest if GEM is working for Pd for your system........ https://curiousart.org/digital_proj/pd_eBook.pdf .....see Chapter 5.
Maybe can be done with Ofelia externals for Pd.... which is worth learning how to use if you have the time..... but I think not...

Otherwise http://reactivision.sourceforge.net/ .... see "finger tracking"..... might get you position data that it will send as OSC messages that can be received in Pd...
David.

Last semester, I was giving a class demo about video camera input in Pd.

For a single blob, pix_blob does reasonably well (though, as it's a single weighted average, everything tends toward the center).

"... movement of fireflies" implies tracking multiple points.

Gem has pix_multiblob, but on my system, this was unacceptably slow. YMMV depending on hardware, image resolution etc., but in my case, it simply wasn't usable.

So instead, for the demo, I found a Processing sketch online for multiblob tracking, and added a function to send Open Sound Control messages to Pd. Worked a treat.

Ofelia is likely to be more performant than Gem, but it's much harder to use.

hjh

Thank you @ddw_music and @whale-av! I have some reading to do, will report back.

OK, so I did some reading! Thanks to both posters who responded.

To recap, I'm hoping to track the movement of firefly lights to produce data that I can use to trigger or influence sounds. This will happen live, to create an aleatoric soundscape that a musician can improvise with. We will be staging these performances at dusk or shortly thereafter, so I suspect a big part of this will be making use of the fact that the fireflies are brighter than anything else- hopefully some form of threshold level will help me eliminate the other movement of leaves in the wind, etc.

@whale-av I have been working with the GEM patch from that textbook on my old macbook, and it's going pretty well, especially as the example patch in the book has a threshold control. However, I just found out find that GEM will not work on my new silicon M1 macbook (Ventura OS), which is the one I was hoping to use for the performance. I could use my old laptop, but it's a bit creaky, from 2012. I've looked through the forums and it seems GEM is no longer supported on newer macs, and Ofelia is way more complex to use (and there's no handy textbook for me to follow like with GEM)?

@ddw_music I am just now really reading your post. I'm definitely hoping to track multiple points (fireflies). You think I should look into Processing instead? I have some experience with Arduino sketches, but I'm not a skilled coder. Could you maybe share the sketch that you used, please?

Thanks all!

@Dizzy-Dizzy There is a chance that it can work on your M1....... maybe....... https://www.reddit.com/r/puredata/comments/yq36kc/gem_on_a_mac_with_m1_chip/
David.

It took a few minutes to compile Gem, here is a pretty stable version of Gem for M1 I did in februrary:
https://drive.google.com/file/d/1xIM5EElTXiDh9USz0siK3vqxHCIQUxGs/view?usp=share_link

Since Gem is still maintained, you can expect bug fixes and new features in the last master on github, but they are untested.

I remember the last version in github fixed a fullscreen issue under MacOS, but I did not have the time to test it further...

Thanks both - I downloaded the zip file and moved the files into my externals folder (replacing the Gem externals I already had) but Gem objects aren't able to be created, just a dotted outline. Any hints on this? Running Ventura 13.2.1. Thank you!

@Dizzy-Dizzy In the Pd console...... what are the messages just before the "cannot create" error messages for the objects? i.e. does Pd find the objects, or are they incompatible with your system?
I imagine that you still have the entry "-lib Gem" in your file,preferences,startup,startup flags, and the path to the Gem folder set in file,preferences,path.
David.

@whale-av thanks as ever for your help
In the console I'm just getitng the name of the object followed by '... couldn't create'.
For example:
pix_buffer_read vid
... couldn't create
gemhead
... couldn't create

@Dizzy-Dizzy It seems probable that you do not have the correct binary for your system.
Try putting a [declare -lib Gem] object in your patch and saving and re-opening it.
If that still doesn't help then you should try deken (help - find externals) again in case some updates have been coded (but I don't think that will help)..... and then, if it still fails the last resort will be to compile from source (make install) for your system.
At least, when you do that, you should see messages about failures, and if there are only dependancies missing you should be able to get them if they are compatible.
You will need the "latest, bleeding edge" release for that I think.
See the readme here........ https://git.iem.at/pd/Gem

Did you try opening Pd from a terminal after typing

arch -arch arm64e zsh

into the terminal?
David.

I think the file I create are blocked by apple security system, since their system automatically tag downloaded binaries to block them... you can type this in Terminal to untag the files:

sudo xattr -cr path/to/Gem

But good news, there is now automatic builds up on deken... Here is the news from IOhannes, who is actively maintaining Gem:

just for the record: since yesterday there's snapshot builds of Gem available on deken.

• this is a snapshot of the current git 'master'. there hasn't been a
  release. things might be more broken then in a normal release (on the
  upside: if you report bug, they are fixed faster than in a normal release)
• there is only a single version "0.94-snapshot". the package will be
  updated whenever a i change something in the git repository. there is no
  turning back (if things worked with yesterday's snapshot, but not with
  the one downloaded today, then you are out of luck)
• Windows: both 32bit and 64bit binaries are available
• macOS: the binary is universal (amd64 and arm64). they should run at
  least on "Big Sur" (i've tested them[*] on Catalina/amd64 and Monterey/M1)
• Linux: there are binaries available, but unlike with Windows/macOS
  binaries, there are no dependencies included. I do advise Linux people
  to just build Gem themselves (or use a distribution that offers packages
  for Gem's git snapshots). this is because on Linux you typically have a
  proper package manager (that allows you to easily install all the stuff
  needed, starting with a compiler), whereas on macOS/Windows you don't
  (unless you count the variants of "app stores" - which I don't).
  the pre-built binaries will most likely not work out of the box on
  your system. (having said that: the binaries where produced on a
  Debian/bullseye system, so if you are running something similar it
  should be possible to just install the missing dependencies with your
  package manager; a good start is to just install the "gem" package that
  comes with your distribution, as this should pull in most dependencies)

gfds
IOhannes