Nice to see this has triggered a discussion. I will be the first to admit that I don't know what this is for. As with most things AI, this is a solution in search of a problem. Here are the examples that I have provided in the repository:

1. LSTM/GRU emulation of guitar pedals (this is the same inference engine that is used in Proteus and Aida-X guitar distortion modeling engines - you can even load those models with rtneural~)
2. RNN note and rhythm prediction
3. Multi-layer perceptron as control (similar to FluidMLPRegressor)
4. Mult-layer perceptron as a wavetable oscillator
5. GRU sine wave to saw hocus pocus

RNN distortion and RNN time series prediction were, as far as I know, not possible before in pd. They definitely weren't possible in sc, which is where I mostly hang out. Thus, people haven't yet had time to play around with them. And I think people will figure out what to do with them given the time. That is the point of the project: to provide a framework that will work on any network if you train it correctly and save it in the right format. If someone wants to build a WaveNet or AutoEncoder, they can train it in python and now they can run it on real-time audio or as data in pd.

The sine wave to sawtooth example is just there to show a magic trick that is really difficult with traditional dsp (btw - it isn't just a sawtooth, but is a band-limited sawtooth, which opens up far more possibilities). This is where much of the possibility lies - in dsp development. The guitar distortion models are the best example of the possibilities. Lots of development has happened in this arena because it is low hanging fruit that is highly marketable. To be able to contort a signal like these trainings can, with one object, is pretty amazing. What other kinds of distortions are possible? I don't know!

The other thing that could be very useful for people right off the bat is RNN time series prediction, predicting notes and rhythms or whatever. Siri is an LSTM, so we are all familiar with this technology. As far as I know, this kind of inference was not available before in pd.

Last thought - conflating the big tech LLM stuff with small-scale AI technologies is not correct. These are different technologies and different ecosystems. Their data sets, training times, trajectories and uses are completely different. This is a small-scale, open source audio project with no aspirations to take over the world. But hopefully it can open up some new possibilities in music creation and dsp.

Sam

Thanks @alexandros. Hopefully you can get some use out of it. I see you have your own project as well. BTW - in my post I forgot to thank Christof for his help, both on the pd and sc plugins.

Sam

Hello pd land. Greetings from sc land.

I wanted to share with you all an external project that I have been working on for the past 9 months or so call rtneural_plugin:

https://github.com/spluta/RTNeural_Plugin

This project is designed to be a real-time safe, general use-case neural inferencing engine (using the open source rtneural inferencing engine) that runs at both audio rate and control rate. It is cross platform and runs in pd, max, and sc.

All training is done in python. Any rtneural shaped network should be able to be loaded. See Jatin's repository here for more details on the shapes of networks. There are lots of help files outlining some possibilities, but since this is a general inferencing engine, there are many more things it can do

Compiled releases for Mac and Windows are available on the releases tab. Linux users have to compile themselves. Instructions are provided.

There are 3 demo videos on YouTube outlining how this works:

General overview (everything but LSTM time series prediction):

LSTM note prediction in SC:

LSTM note and rhythm prediction in Max:

A pd video for the LSTM prediction will happen, but we probably need to make an object that deals with the N parameter softmax output of the external. If anyone has a better idea, please let me know.

Let me know if there are any questions or issues. Bugs or questions can also be submitted on GitHub.

Sam

(Also, right now the windows build of the pd rtneural~ object is a bit wonky. If anyone wants to help debug that I would be grateful. The only pc I have is parallels on an ancient iMac, so it is a bit tedious and I don't trust it.)