Hello pd land. Greetings from sc land.

I wanted to share with you all an external project that I have been working on for the past 9 months or so call rtneural_plugin:

https://github.com/spluta/RTNeural_Plugin

This project is designed to be a real-time safe, general use-case neural inferencing engine (using the open source rtneural inferencing engine) that runs at both audio rate and control rate. It is cross platform and runs in pd, max, and sc.

All training is done in python. Any rtneural shaped network should be able to be loaded. See Jatin's repository here for more details on the shapes of networks. There are lots of help files outlining some possibilities, but since this is a general inferencing engine, there are many more things it can do

Compiled releases for Mac and Windows are available on the releases tab. Linux users have to compile themselves. Instructions are provided.

There are 3 demo videos on YouTube outlining how this works:

General overview (everything but LSTM time series prediction):

LSTM note prediction in SC:

LSTM note and rhythm prediction in Max:

A pd video for the LSTM prediction will happen, but we probably need to make an object that deals with the N parameter softmax output of the external. If anyone has a better idea, please let me know.

Let me know if there are any questions or issues. Bugs or questions can also be submitted on GitHub.

Sam

(Also, right now the windows build of the pd rtneural~ object is a bit wonky. If anyone wants to help debug that I would be grateful. The only pc I have is parallels on an ancient iMac, so it is a bit tedious and I don't trust it.)

This is a great addition to Pd's AI capabilities! I had thought of creating a Pd external based on RTNeural some time ago, but it was too much for me to handle. I'm grateful you took the initiative and did this! Being able to train NNs in Python and run them natively in Pd opens up a lot of possibilities!

Thanks @alexandros. Hopefully you can get some use out of it. I see you have your own project as well. BTW - in my post I forgot to thank Christof for his help, both on the pd and sc plugins.

Sam

please explain to me like I'm a 5 year old what you youngsters are doing with this kind of thing

@spluta I do have my own project which is an external that enables the creation of neural networks, all written in C from scratch, but I've gone as far as autoencoders, struggling to get it to create variational autoencoders. RTNeural though takes advantage of all the development that has been done in Python packages (the go-to language when it comes to AI), so this external is really nice to have!

@porres The main issue that I see with neural networks is the immense amount of data they need to train properly. Apart from being hard to assemble a proper training dataset, there are all these copyright issues and (un)ethical use of data and AI. But in some cases they can be very useful, like controlling lots of parameters with just a couple of knobs (or the mouse pointer position) or messing with sound with variational autoencoders and their latent space, to transform sound in various ways (check the example of this external where the sinewave is turned into a sawtooth - not very clean, but it's just an example), just a couple examples of their use.
Still, I do believe that AI and NNs are a hype that evenrually will phase out and will be replaced by something else that will become the new thing.

I am being serious, I have no idea what this is for and what can be done.

Turning a sine into a sawtooth was the only example, but not what I was expecting

I feel AI could really do some quite cool things if it;s OPEN SOURCE and we get to tweak it in interesting ways. Cause most of these new hyped cool things that generate music, for instance, generate very common ground things (which they were trained the most with to be able to generate output). But what if I wanna go crazy and tweak it? I guess we need to be nerds that play with Pd/Max/SC, ok, but for someone who has no idea about it, really, how does it go? And I'm so busy with what I'm doing, I'm not stopping until I see some pretty cool thing going on. Turning a sine into a sawtooth wouldn't be it :/

Like, I know that, in general, I can use AI for "deep fakes", so I can make mike patton with boobs sing carmina burana with his voice, but I guess 'rtneural' can't do that, right? I have "some ideas".... but I don't even know how far off I am on this.

And the actual hype with AI and chat GPT gonna take over the world, for what I've been seeing and reading and checking, it's a hoax. The whole thing it's pretty bad. For reference, I'm hearing what ed zitron has been saying about this and folks like sam altman... seems legit!

Anyway, really, explain like I'm a 5 year old

@porres From one 5 year old to another....
A Neural Net attempts to replicate a brain..... treats snippets of language and information as if they were stored in neurons.
Starting with some instructions (rules) from a programmer it then looks for likely links between those "neurons" and builds new rules based on data matches and the probability that it will return the correct output. We don't know how the brain works, so the maths involved is somebody's best guess.
Given the high percentage of garbage LLM's scour on the net they have a quite high failure rate, which gets worse as the model grows and feeds also on garbage outputs. For a while now its language construction is excellent, causing humans to believe that far more of its output is correct than is actually the case.
Unfortunately newer models are also proving to be worse than their predecessors.

A neural net trained on a smaller data set in Pd might be better, and the initial rules easier to fathom...?
David.
.

I have to say that as someone who writes code as a 5 year old does, AI has been helpful to help me to some things I couldn't do/fix on my own. It surprises me. Though sometimes I can clearly see it doesn't know what it's doing and is just wasting my time, others I get to know where it's failising ridiculously and I can put it back to the right track to help me.

I'm hearing what ed zitron has been saying about this

Nice to see this has triggered a discussion. I will be the first to admit that I don't know what this is for. As with most things AI, this is a solution in search of a problem. Here are the examples that I have provided in the repository:

1. LSTM/GRU emulation of guitar pedals (this is the same inference engine that is used in Proteus and Aida-X guitar distortion modeling engines - you can even load those models with rtneural~)
2. RNN note and rhythm prediction
3. Multi-layer perceptron as control (similar to FluidMLPRegressor)
4. Mult-layer perceptron as a wavetable oscillator
5. GRU sine wave to saw hocus pocus

RNN distortion and RNN time series prediction were, as far as I know, not possible before in pd. They definitely weren't possible in sc, which is where I mostly hang out. Thus, people haven't yet had time to play around with them. And I think people will figure out what to do with them given the time. That is the point of the project: to provide a framework that will work on any network if you train it correctly and save it in the right format. If someone wants to build a WaveNet or AutoEncoder, they can train it in python and now they can run it on real-time audio or as data in pd.

The sine wave to sawtooth example is just there to show a magic trick that is really difficult with traditional dsp (btw - it isn't just a sawtooth, but is a band-limited sawtooth, which opens up far more possibilities). This is where much of the possibility lies - in dsp development. The guitar distortion models are the best example of the possibilities. Lots of development has happened in this arena because it is low hanging fruit that is highly marketable. To be able to contort a signal like these trainings can, with one object, is pretty amazing. What other kinds of distortions are possible? I don't know!

The other thing that could be very useful for people right off the bat is RNN time series prediction, predicting notes and rhythms or whatever. Siri is an LSTM, so we are all familiar with this technology. As far as I know, this kind of inference was not available before in pd.

Last thought - conflating the big tech LLM stuff with small-scale AI technologies is not correct. These are different technologies and different ecosystems. Their data sets, training times, trajectories and uses are completely different. This is a small-scale, open source audio project with no aspirations to take over the world. But hopefully it can open up some new possibilities in music creation and dsp.

Sam