AI is creating never before heard sounds! 🎵

AI is being used to transform the most personal instrument we have, our voice, into something that can be “played.” This is fascinating in and of itself, but Yotam Mann from Never Before Heard Sounds is doing so much more! In this episode, he describes how he is using neural nets to process audio in real time for musicians and how AI is poised to change the music industry forever. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. PSSC Labs https://pssclabs.com/practicalai – Solutions from PSSC Labs provide a cost effective, highly secure, and performance guarantee that organizations need to reach their AI and Machine Learning Goals. Learn more and and get a FREE consultation today at pssclabs.com/practicalai https://pssclabs.com/practicalaiThe Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Yotam Mann – Website https://yotammann.info, GitHub https://github.com/tambien, X https://x.com/yotammannChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Never Before Heard Sounds https://heardsounds.com/Yotam’s personal website https://yotammann.info/Holly+ https://heardsounds.com/holly-plusReal Time Hardware for Audio Processing https://heardsounds.com/hardwaregan.style https://heardsounds.com/gan-styleOnsets and Frames (Piano Transcription) https://arxiv.org/abs/1710.11153DDSP: Differentiable Digital Signal Processing https://magenta.tensorflow.org/ddspUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I have some kind of frustration with a lot of the way that machine learning stuff goes where it's like not fully attributed data sets. So we are trying to kind of do the hard tack in the opposite direction. All the data sets that we have are open source, or we hired our friends or musicians that we know to create a data set that is fully licensed for this purpose of creating a generative model. And then I think when people are able to en masse create their own generative models, then there is a really big opportunity for creating value for those musicians, like getting paid for the generative use of, or the licensing of that generator.

• 0:39Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:23Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R-U-D-D-E-R-S-T-A-C-K.com.

• 1:57Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at kanej.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:23Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:38Chris Benson?

  I am doing very well. How's it going today, Daniel?

• 2:40Daniel Whitenack?

  It's going great. I don't know if we talked about this. Are you a musician in your past at any point?

• 2:45Chris Benson?

  I have been a musician a lot in my past, but I'm not currently right now. But I love music.

• 2:50Daniel Whitenack?

  I dabble a bit, so I'm pretty excited today that we have Yotam Mann, who is a musician and programmer. He's the co-founder of Never Before Heard Sounds, which uses machine learning and AI for musicians, which is pretty exciting. Welcome, Yotam.

• 3:08Speaker 00

  Good to be here. Thank you for having me.

• 3:10Daniel Whitenack?

  Maybe we should just start out to let people know how machine learning maybe has been used before for music, or, you know, some things that people are trying out there. Just sort of set the landscape for us. How have people tried and maybe failed or succeeded in using machine learning in the realm of music?

• 3:32Speaker 00

  Sure. I mean, there's been, I guess, a lot of sort of statistical music generation for decades now. I wish I could remember this person's name, but automatic music generation, sort of in the MIDI form, using Markov chains, has been pretty successfully used to imitate, you know, classical-era composers, going back from the 80s and 90s.

• 3:55Daniel Whitenack?

  And MIDI, this M-I-D-I, for those that don't know, that's like a representation of like, a sort of markup representation of notes and lengths of notes and that sort of thing, right?

• 4:07Chris Benson?

  Sadly, I remember all this all through the world.

• 4:11Speaker 00

  Yeah, no, I, thanks for stopping because I might speak in some other music-specific jargon. Yeah, MIDI is, is notes onset and offset. And, you know, more recently, a lot of stuff has been in the audio classification realm. Certainly, people want to know, you know, when a piece of audio, someone's talking, or there's music playing, or in music information retrieval, which is a massive field. So I'm, this is just all like super broad overview, and then we'll get into this little niche corner that we occupy. Music information retrieval, which is about figuring out the chord structure of a song is, maybe even the genre of a song.

• 4:53Speaker 00

  This has also been going on for, for, you know, more than a decade at least.

• 5:00Speaker 00

  More recently, sort of deep learning stuff has been in the realm of music generation. So really trying to get some really accurate, you know, sort of composition-length audio that maybe isn't a specific style of an artist, or a period, or a location. And then the realm that, that we're focused on at the moment is in the audio generation realm. So forgetting the sort of the pitch MIDI format stuff, and jumping straight into how can we make new and interesting sounds using machine learning.

• 5:36Daniel Whitenack?

  I'm wondering, as I'm thinking about this, we, we sort of talked about MIDI a little bit, and then we have like, you know, I think people are familiar with at least like audio files, like they might have MP3 files or WAV files or something. But in terms of AI models and these generative models, like how is, how is audio represented in most of these models in terms of like data input? And generally, how, how it's processed?

• 6:04Speaker 00

  Yeah, great question. I would say, you know, the, the sort of the MIDI to audio jump is a similar kind of leap. As say like text to image, just a substantially more data to deal with and much more complex models. So MIDI can be represented in a number of ways. You can think of it as if you've ever seen sort of like a piano roll or a music box where you have kind of all of the notes in one dimension and then time in another dimension. Or you can represent it just as sort of discrete onset events and offset events. So there's a great piano transcription model called Onsets

• 6:46Speaker 00

  and Frames and it is able to kind of reduce the piano transcription output to a pretty kind of compact format by just giving you the onset and the offset of the notes as opposed to, you know, you can imagine sort of this 2D array of data is a much bigger thing to represent. On the audio side, there is a few ways to do it. The models that I'm typically working with are outputting sample by sample. So audio, digital audio, you know, like a WAV file is an array of floating point values usually, one for each channel, and that's typically sampled at 44,000 times a second. So we're talking about quite a lot of data for things

• 7:29Speaker 00

  that are generating in the raw audio domain. It makes it kind of a challenge, you know, especially a challenge to do it quickly because it's a lot of, a lot of data to generate.

• 7:40Daniel Whitenack?

  Yeah, I assume that, you know, it's not that satisfying to have a generative model where maybe you hit a note on a keyboard and then like seven seconds later you get the output, right?

• 7:54Speaker 00

  Yeah.

• 7:55Chris Benson?

  It's delayed gratification there.

• 7:58Daniel Whitenack?

  You know, I don't know that that would promote like a lot of jazz improv and that sort of thing.

• 8:04Speaker 00

  Yeah, you can imagine how hard it would be to learn an instrument where say you sit down at a keyboard and you play for a minute and a half and then it's totally silent and then you come back and you can hear what your minute and a half sounded like. It would be a pretty laborious way to, to learn anything. So specifically the models that we focus on are, in the audio domain are realtime. So they both need to generate faster than realtime and they need to have small enough little buffered chunks to be able to keep up with a low latency. So there's kind of two, two parts of realtime audio processing. One is the realtime, which is like It takes less than a second to produce one second of audio.

• 8:45Speaker 00

  That would technically be a real time model, but then a low latency real time model would be as soon as you, you know, ask for the beginning of the audio generation. How long does it take to, uh, to give you the beginning of the response?

• 8:58Chris Benson?

  You'd kind of have to be Beethoven, you know, being the deaf composer that he was to, to be successful. Otherwise, you know, if you, if you were having that delayed period all the way through there.

• 9:07Speaker 00

  Yeah, that would be tough.

• 9:08Daniel Whitenack?

  These models that you're working with, these generative models. I mean, maybe the first thing that comes to people's mind when they think of generative models is GANs, so generative adversarial networks. Is that sort of the realm of things that you're working with? Or are you, what's the sort of range of those types of either frameworks or architectures that you're exploring in the audio space?

• 9:30Speaker 00

  We had some success with, with GAN-based models. We actually made like a little website where we show off two of these models called gan/style. You drop in a YouTube link and then it replaces the audio of that YouTube link with the audio generated from one of these models.

• 9:49Chris Benson?

  That sounds exciting.

• 9:50Speaker 00

  Yeah. So the, the two models that we got running in the GAN style are a choir model that's generated from, I think 11 plus hours of, of kind of choral church music and a string quartet model, which is from a university computer science slash music crossover department that they released a paper with this, with this data set. So those are both trained, uh, using, uh, GAN and a discriminator for the loss. We also had some luck losing the discriminator part. It trains a lot faster and there's other kinds of losses that we were able to use. I think if you look at the paper out of Magenta called DDSP,

• 10:33Speaker 00

  you'll see a really successful version of this sort of timbre transfer, the same kind of technique, turning one sound into another. And they don't, they don't use a GAN approach. They just use, yeah, different, different loss, FFT loss.

• 10:47Chris Benson?

  So I have, I have an almost a side question because I'm, I'm fascinated by this and it's a use case for GANs that we haven't talked about yet, you know? Which seems kind of weird at this point since, you know, we all love music and everything, but. I can't wonder, what drove you into this? I mean, I guess you were doing music first and getting it. You know, how did you combine all of this to go do this? And what was the motivation to get you going? Because as you've been explaining this, I've been wondering that along the way.

• 11:15Speaker 00

  Yeah. How does someone end up here? Good question.

• 11:17Chris Benson?

  How do you end up there? Yeah, I mean, really. And how can I end up in the same place? It sounds like a lot of fun. He was much smarter about it than we were. Yeah.

• 11:26Speaker 00

  Yeah, I actually never even really thought about like learning how to program or anything like that until I was kind of into my, I guess, junior year of college. I studied music. I played piano and I studied jazz piano. And that was kind of my whole focus. I was really interested in kind of music production techniques and different kind of things, but I had never expanded to the realm of sort of like entirely generative and especially not like machine learning generated.

• 11:55Chris Benson?

  That wasn't just a thing that popped into your mind, studying music in school?

• 11:59Speaker 00

  Yeah, so I was lucky that I went to a school that had this computer music department, which I didn't know as a field and which is, you know, like a very vibrant field over the past many decades with a few kind of like bases all around the world. And one of the bases is at UC Berkeley, the Center for New Music and Audio Technology. And I just was sort of enamored by the way that people were making music there. And this entirely new seemed like really innovative, seemed like the stuff that they're making is like a huge departure from anything that you could do using different techniques. So that kind of led me down this route of like, okay, how can technology kind of like make new music?

• 12:44Speaker 00

  How can it kind of keep progressing and pushing music forward? And, you know, I had done many, many projects sort of in music technology, building synthesizers, building even like, you know, music generator systems, doing some like generative music for video games, different things. And maybe about

• 13:06Speaker 00

  Four years ago or something, I started to get more into, okay, you know, like machine learning, this is the next big technology innovation. What's this gonna do for music? And so it kind of like, you know, I, I just saw a huge, huge opportunity in all of the potential. And so about two years ago, I started a company with a partner, Christine, or called Never Before Heard Sounds, basically entirely dedicated to this question of, okay, what is AI ML gonna do for music? And specifically with the angle of like, you know, there's a lot of people doing, not a lot, but music generation systems, you know, like for ads or for, I totally understand the like utility of that,

• 13:49Speaker 00

  but that for me doesn't progress music, like that doesn't push the envelope of what's possible. It sort of puts music in this like utilitarian category of like, okay, music serves a function, you know, helps you study, uh, you know, like works really well in this ad video and things like that, but it doesn't kind of like, you know, the, the way to, to progress music is to put those tools in musicians' hands to build their instruments. Yeah.

• 14:17Chris Benson?

  So, so you're there in school, you're a junior, you have not previously done this, you discover this. But I gotta say, I, and I'm not just talking about music here. I'm thinking back to myself and, and probably most people to leap into machine learning and to suddenly be able to be a practitioner in the space. That's a tall order. That is a significant leap right there. Can you tell us a little bit about, you know, you're this musician, you're doing this. You discover this, you get this exposure. How did you make that jump into being a practitioner? How did you accomplish that?

• 14:50Speaker 00

  You know, I graduated over 11 years ago, 10 years ago. So it's been a slow transition. Uh, it definitely was not something overnight.

• 14:59Chris Benson?

  Okay.

• 14:59Speaker 00

  First started with, there was this great, you know, my, my like primary language that I got super into after college where it's like, it's all Python and Java and list for some reason. So, you know, then the thing that I was most fascinated with is JavaScript. Cause it's like, you program in JavaScript, people can run your applications on like billions of devices and great language for that reason. So I got into, the first thing that I looked at with machine learning was this. Really simple neural net implementation in JavaScript. And I wish I could remember the name. I remember the author's first name was Heather.

• 15:38Chris Benson?

  I think I saw that. I don't remember either, but I think I remember.

• 15:41Speaker 00

  From years ago. Yeah, yeah.

• 15:43Chris Benson?

  I think I remember what you're talking about.

• 15:44Daniel Whitenack?

  If you know what we're talking about, let us know and we'll put it in the show notes.

• 15:48Chris Benson?

  That's right.

• 15:48Speaker 00

  Yes, please. Thank you. So it was this great, like, compact library and just like being able to see those things in this JavaScript is a fairly simple language. Being able to see those concepts like kind of laid out in that simple way. That was my, that was my start. And like in the beginning, I wasn't able to really produce anything interesting or fascinating with this, but like I kept trying, I kept seeing that potential.

• 16:30Speaker 02

  With advancements in AI and deep learning evolving at lightning pace, it's more important now than ever to research the best options suited to your unique needs. This is particularly true when building custom systems and those systems that are GPU-heavy. Not only do the applications running on the system matter, but your AI infrastructure and budget constraints need to be front of mind as well. PSSC Labs, which is an HPC and AI custom solutions provider based in California, has been creating high-performance computing systems to meet their clients' unique enterprise computing challenges for more than 25 years. And with cloud computing costs growing at astronomical rates, plus companies increasingly losing control of their data security,

• 17:14Speaker 02

  it is no wonder that enterprises and government agencies need to continually look for ways to take back control of their data. Solutions from PSSC Labs provide a cost-effective, highly secure, and performance guarantee that organizations need to reach their AI and machine learning goals. For more information and a free consultation, please visit pssclabs.com-practicalai. Once again, that's pssclabs.com-practicalai.

• 17:51Daniel Whitenack?

  We were chatting before starting the recording about some of the projects that you're working on. And I think you had mentioned this one AI vocal technology that you just released, which I think is really interesting. And not only is it interesting technology, but I think it has a lot of elements in it that I think people, you know, may have follow-up questions on in terms of like data and the motivation and use and, you know, all of those things. So could you just introduce that? I believe it's called Holly Plus. Tell us a little bit about that and how it came about.

• 18:26Speaker 00

  Yeah, so I had mentioned this past project, GAN Style. And really, for me, the interesting thing with these models that we've been developing is, you know, like we can find cool data sets and we can train cool models. But really, I think the big leap is in letting people train their own models. And now you have a generator of your specific sound. You know, your grandma's piano with the one squeaky note and like the out of tune upper register, or your specific guitar playing technique where you slap certain notes and you pluck others. And obviously, the most, the most personal instrument that we all have is our voice.

• 19:09Speaker 00

  It seems like the voice models are like really the most interesting personal model. So while we were developing GAN Style, we're thinking about, okay, like it'd be cool to find some musical artists that we could collaborate with. And I always had Holly Herndon in mind, just because she's done a lot of AI music projects. So I actually just basically tweeted up her musical partner. And we were DMing and I was like, or no, actually, he had first posted on our launch day. Is there anyone out there who could help us out with machine learning? You know, like any, any data scientist, people, ML programmers who could help us out with the project?

• 19:51Speaker 00

  I immediately responded in DM and I was like, yeah, what are you looking for? And he's like, well, we want to build a website where anyone, where we could train a custom model on Holly's voice. And we could have anyone upload an audio file, transform it into that custom model and then kind of like have the results. And we had just released this dance style, which is, you know, a lot of the same mechanism. And I had already transformed one of her songs into the choir. So I sent it to him and he's like, I think his response was Holly and I are freaking out. Can we talk?

• 20:25Speaker 00

  Yeah, fast forward a few months, we had sort of repurposed this, you know, GPU backend that we had developed for, or not repurposed, but augmented now to, to include this Holly Herndon model. And trained a model from her, you know, a bunch of recordings that she gave us of her voice. And she's got a, one reason that this worked kind of successfully is that her vocal style is, is not acoustic. It's very digital. It's a lot of like harmonized, a lot of vocoded stuff, a lot of like digital glitches. And you know, like one thing that machine learning is really good at is producing some digital glitches. So yeah, it worked out pretty well with her voice model.

• 21:07Speaker 00

  And then last week, we, yeah, we released it to the world where anyone could drop in an audio file and have it transformed into Holly Herndon's voice.

• 21:18Daniel Whitenack?

  I have so many follow-up questions on this. I don't think we'll get to them all in our, in our interview. So it's, so you'll have to keep, keep me on track, Chris. Maybe first off, like you were talking about the voice being one of the most personal sort of instruments that we have. But also, you know, a musician's voice, it's sort of like, it's a big part of their livelihood, right? And how they, how they make money. So what are your sort of thoughts? And what were your discussions with Holly and maybe others as you were entering into this project? And the future of these types of projects around like, hey, I could, you know, I get Taylor Swift's voice and I can produce my own backing track.

• 22:02Daniel Whitenack?

  And now I have my own Taylor Swift song and release it on and stream it on some service. And now I'm making money off of it. Right? So there is a lot of like messy stuff that could come up here. So what were some of your thoughts on that? And what discussions have you had there?

• 22:20Speaker 00

  Yeah, this was really for Holly Arnden and Matt Dryhurst. This was the purpose of the project was like the age of this, exactly what you're describing, Daniel, this is coming. And they wanted to kind of control that story and offer a version of how this could go. So they have a whole mechanism for how things that are created with this, you know, they're free and they're, they're able to be used, but the ownership is actually through like a DAO, a decentralized autonomous organization. That's their version of how this kind of rights management could go. But I think that those kind of are the most relevant questions when you think about,

• 23:04Speaker 00

  about these types of generative models. Our version of this and how we've been producing and thinking about the future models that we produce is really like full transparency, full attribution. And, and also when we get to it, being able to kind of pay the artists back that went into this model. So. I have some kind of frustration with a lot of the way that machine learning stuff goes where it's like not fully attributed data sets or like a lot of scraped data sets and things like that. And so we are trying to kind of do the hard tack in the opposite direction. The all the data sets that we have are

• 23:47Speaker 00

  open source or we we hired our friends or musicians that we know to create a data set that is fully licensed for this purpose of creating a generative model. So and then I think when people are able to kind of en masse create their own generative models, then there is a really big opportunity for that being sort of, you know, creating value for those musicians, like getting paid for the generative use of or the licensing of that generator. So that's kind of how I see that potential future.

• 24:19Daniel Whitenack?

  So Holly was a part of this process and maybe had a vision for the future of how this data was going to be used and potentially how, you know, her career and could be benefited benefited from it, maybe how her bottom line and her finances could be benefited from it. Whereas like if I go to Spotify and just use some tool to capture a bunch of audio from an artist, they have no knowledge of of that particular use. And so, you know, part of it, I guess, is on the data side and maybe part of it is on how the model is is released. I know we've been seeing a lot more models even, you know, outside of this genre of models being released as APIs where there's

• 25:02Daniel Whitenack?

  sort of more control in terms of who's using that for what and how that exchange of value is happening and and that sort of thing. So yeah, do you see a lot of these models maybe in the future being released in that sort of API form or or, you know, because I could also just throw the model up on S3 or something and then like anybody can use it to generate generate a voice, maybe.

• 25:29Speaker 00

  Yeah, true. For the time being, we are kind of in full control of of how these models are all used. So, you know, they are behind an API that we control. They are they are on our servers. And we are not yet at the place where we intend on open sourcing these models or their weights. Partially because of these reasons of, you know, like being able to kind of.

• 25:54Speaker 00

  Control who uses it and like give value to the musicians effort that went into this and kind of, you know, honor that by not just giving it away for free everywhere.

• 26:04Chris Benson?

  You know, it's, it's interesting is that this is an area like, like pretty much everything we've seen in the AI ML space that because it's a totally new way of doing things and clearly there will be lots of people. I mean, this is, I mean, you're pioneering something that I think will be huge going forward. And yet there's the kind of AI ethical consideration that is kind of built into everything that we do. You know, we've had this conversation across so many topic areas and stuff. It's pretty cool. I mean, so you're recognizing that early on and there are questions that have to be figured out and like, like every other field, but, and, and it sounds like you're taking a kind of a careful, you know, kind of respecting that, that process upfront. How do you shape that as you, as a pioneer

• 26:49Chris Benson?

  yourself in this field and you're looking at an industry that where other people will start doing either the same or very similar things and exploring their own creativity, just like we've seen in, in convolutionals with, you know, artwork creation, you know, in terms of visual things. How do you think that will roll out going forward to, to try to, to try to have a whole new industry in music that is taking advantage of this and, you know, merging it with existing, you know, approaches?

• 27:19Speaker 00

  As a small company, what all we can do is kind of like lead by example and, and hopefully make this sort of public enough information that it, people might expect that from other companies. Full attribution of the data sets or that, you know, the musicians involved are paid. I guess part of it is really education of how this stuff works. I think a lot of times the story that's sort of spun is like the AI is some agent that's like, you know, like a magical character that's off in the cloud doing some magic up there. But, you know, the story that we're trying to make clear is like, no, that's, that's actually not really how it works. There's a bunch of musicians, you know, there's kind of this condensed mathematical model and that is a generator that you can then

• 28:04Speaker 00

  play with. But in the end, that model acts as a conduit between you, a musician and the data set musicians. So making those data set musicians as kind of clear as possible and making that narrative as clear as possible is our approach to it, you know, so that when other data sets come out, I think the immediate question for consumers is hopefully, okay, well, where'd you get the data, you know, who's involved, what are their names, were they paid for it, and those kind of things. And yes, putting a face on that data set is our is our approach that hopefully catches on.

• 28:38Daniel Whitenack?

  And that data set that you gathered or worked on with Holly and her collaborators, what did that end up looking like? What data was needed to actually make this happen in the end in terms of like, how much and like, you know, someone could put in almost any kind of audio into this thing, right? We actually just had a conversation, I think, in the last release episode about, you know, out of distribution input into into models. And sort of here, you've got like this whole range of whatever audio could be that could come into this. So what did it take to put the data set together and get it behaving reasonably for for a sort of wide variety of audio input?

• 29:22Speaker 00

  It's the right question, which is like, how do you how do you train something? And really, it all just comes down to the data set. So for us, we it was a few iterations. You know, we had definitely a bunch of models that that didn't really come together in the end. And the initial ask was for Holly and Matt was, okay, can you give us roughly two hours of audio, you know, WAV files that are self similar? And the self similar part is like kind of an abstract notion. But, you know, we're like, okay, whatever is the the Holly sound like, give us give us those. So so we had a few back and forths and we trained probably about a half a dozen models on different permutations of their data

• 30:03Speaker 00

  set. We built a few tools that do, you know, sort of data set analysis and throw out things that are too far, or the thing that ends up being really far, oftentimes in audio data sets is like silence. So, you know, automatic silence, kind of trimming and things like that. And so, yeah, roughly about two hours of audio is what we've been using as our rule of thumb for these data sets, but it all depends on kind of how sort of self-similar it is. And then what you're asking for it to do on the output. Um, so because this was, you know, something that we wanted to be able to handle pretty much any audio file that people throw into it. We also just trained it for, for a really long time to try and kind of get out all of

• 30:46Speaker 00

  those little weird tweaks and edge cases that happen when you ask it for something that it knows nothing about.

• 31:02Speaker 02

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt-in reward system so you can get paid to view privacy respecting ads, then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 31:44Daniel Whitenack?

  Yo Tom, I wanna, I wanna return to a comment you made much earlier, which is that one of your big focuses in your work is real-time audio processing. And you sort of described what that meant in terms of like, oh, maybe you have around 44,000 samples per second and you're wanting to be real-time and low latency. So what has that journey been and the tricks that you've found and, and that sort of thing in, in order to actually reach that level of performance in, in a reasonable way that you, that you can support? Because I'm sure there's a lot of people out there that are interested in that, you know, real.

• 32:25Daniel Whitenack?

  Time, deep learning or AI side of things, maybe not even in the audio space, but in video or text processing from users or something. What's your journey been in that real time space? And what are some of the tricks and what you've had to learn in order to reach that performance?

• 32:45Speaker 00

  Basically trying to, you know, we use convolutional models. Those run the fastest in general. And they also will give you oftentimes, you know, because of the different convolutional layers, they'll give you sort of a bunch of garbage. I don't know how else, but a bunch of random stuff on either end where there was padding added. And what we need is, you know, which is fine if you're generating, it doesn't make, you don't even really hear it or notice it if you generate, say, five minutes of audio. But if you generate a little tiny, like, say, 256 sample chunk, where a large percentage of that chunk is going to just be this sort of convolutional padding, garbage on either end.

• 33:28Speaker 00

  So there are a few different techniques to kind of deal with that. The most simple is you just sort of trim it off. The next more complicated one is you, you know, you do what it would have done internally, which is you pass the end back in as the beginning and sort of repeat that over and over again. And so you're always just kind of swapping things around. That's how, that's how we've been able, you know, most models aren't made to kind of stream in this way. What I'm describing is how you make these, these convolutional models stream. Most of them are just made for, you know, large batch processing. And we need tiny batches that stream really fast.

• 34:10Speaker 00

  Another thing that we've had a lot of luck with in terms of getting, you know, multi X speed up, I think maybe like four or five times faster, was when we converted everything to TensorRT. And so that, that really helped. That required also kind of changing.

• 34:28Speaker 00

  The architecture of our model a bit to fit what I guess TensorRT had or has implemented.

• 34:35Daniel Whitenack?

  Could you talk a little bit about the about that about making that conversion and what it was?

• 34:39Speaker 00

  I mean, there was some just basic stuff like the I used this library. Now they actually have a replacement that I think Nvidia maybe is more officially supporting, but I was using this other library called Torch to TRT that did the PyTorch to to TensorRT conversion. And it was a bunch of layers basically that it didn't have implemented. So I needed to kind of take a look at what their code was like, for example, like 1D convolution, which is is used all over in audio. But it's like kind of a niche thing. I think like when I submitted the the pull request for like, hey, I implemented 1D convolution. Like here, this is this is what it is. I basically just copy paste it. Um, you know, it wasn't anything crazy that I invented.

• 35:24Speaker 00

  I copy pasted their 2D code and made it work for 1D. And the person who responded to this question was, what do you use this 1D convolution for? And I think it's just like most, most people, you know, who are doing image or video stuff, aren't used to seeing 1D convolution. And I that's probably why they didn't even bother implementing the the converter. There was another thing that came up, which was they didn't have specific kind of padding that we were using implemented. So I needed to swap out a bunch of layers. It was, you know, it wasn't pretty work. It was a lot of just kind of like grinding away. And there wasn't any, it wasn't any like massive, you know, epiphany that I can give you listeners. It was really just like,

• 36:07Speaker 00

  yeah, I went through it step by step and like where things broke, I tried to figure out a way around.

• 36:14Daniel Whitenack?

  And so then do do all these style transfer and real time audio processing models in terms of making that real time? Is a GPU required at inference time basically for for all of these?

• 36:29Speaker 00

  Yeah, for ours currently, they all are. And that's what got us to actually like trying to get it on little tiny pieces of hardware. Nvidia makes.

• 36:40Daniel Whitenack?

  Yeah, I've seen pictures with like a little knob and such, which was really intriguing for me because I was like, that's, that's, that's really cool. I mean, you, I was wondering what's inside of that box, I guess, was where my question was leading. Because if, if all of that's requiring the GPU, you know, what ends up having to be in that sort of small box to make that happen? Because it almost, is it inspired by a sort of guitar pedal type profile? Is that?

• 37:06Speaker 00

  Yeah, exactly. That's, we're trying to meet musicians where, with the, with the tools that they're accustomed to and the things that they have. And what not a lot of people have is like a Linux machine with an Nvidia card and the specific version of CUDA installed and all this kind of stuff that. So we're like, okay, well, what we could do is, is, you know, have basically a little computer with a little GPU and have everything basically pre-installed on it. And Nvidia, they've been, actually now, I think, had it for a few years, but they made some, some big strides with their Jetson platform, which is exactly that. It's, it's like a Raspberry Pi with a little attached GPU.

• 37:49Speaker 00

  It's not even a discrete GPU, it's like an integrated GPU, but, you know, it runs all of the CUDA libraries. It does give you a big speed up. And so we've been targeting that for sort of our real-time hardware offering. And, and the idea is to get it, you know, fast enough and stable enough that we could, we could have a consumer product that, that that would work, yeah.

• 38:12Daniel Whitenack?

  Yeah, and I guess if you had that platform and, you know, that GPU integrated, it seems like in theory, you could create a little sort of pedal type device where you could load a whole bunch of different models on it. As you release new models, you could sort of load them on or swap and that sort of thing. So, because I think a lot of guitar pedals, it's like it does a specific thing, right? Maybe this one would be sort of flexible in that way. Is that kind of what you envision?

• 38:45Speaker 00

  Exactly, yeah. So I think that that to me is the really exciting part with all of this is like the models. Like, what can we train? What can we make a good generator of? So I think, you know, we'll train a bunch of models. We have trained a bunch of models, different instruments and data sets that we've either licensed or recorded or open source. But letting people train their own model and then passing it over to their friend and saying, "Hey, this is saxophone technique that I was playing with. Why don't you try performing violin through it?" I think that's like the real exciting future.

• 39:21Chris Benson?

  So when do you think, you know, beyond kind of the gimmick stage, when do you think we'll be going to like concerts, you know, major concerts at major venues? And have that integrated in to the, you know, the performance itself? And not as like, "Ooh, we're using AI on stage," but like you get past that and it's truly part of the art, you know, it's built in as much as the instruments they're already using. Do you think that's sooner or do you think we're still quite a ways from that at this point?

• 39:52Speaker 00

  I think if we have our way, then it will hopefully be pretty soon, yeah.

• 39:56Chris Benson?

  Okay, good answer.

• 39:58Speaker 00

  That's exactly what we're working towards. And that is, you know, I think that the AI as gimmick will hopefully fade away and the AI is interesting, nuanced, you know, like wild sounds and integral to the art.

• 40:16Chris Benson?

  Integral to the art, yeah, yeah. Part of it as much as every other part.

• 40:20Daniel Whitenack?

  Yeah, maybe that brings up another question, which I've been thinking about, which is, I mean, like you say, computers and synthesis and, you know, actually very complicated math and all of that has been used in music for a very long time and synthesizers and other things. What do you think, like, creative-wise, AI-based techniques provide for musicians in terms of their own composition and that sort of thing? Maybe different than like computer-based music in the past? What are the sort of qualities of AI-generated music

• 41:00Daniel Whitenack?

  that maybe distinguish it a little bit from what musicians might have a feeling for already in terms of synthesis?

• 41:07Speaker 00

  I mean, I think that there's a few answers. One is the, the big one is, I don't know, and I want to find out. Basically, like Jimi Hendrix was not possible before the electrification of the guitar. Like those two things kind of go hand in hand. Or, you know, so much of, of what you hear on the radio, people, you know, Young Thug being virtuosic with, um, with the autotune was not possible before autotune was a, was a tool. So I think really the most interesting part is like how people, what we've kind of laid out is like a utilitarian thing. Oh, I don't have a saxophonist, but I have a saxophone model. Uh, so let me, you know, use the saxophone model to generate a saxophone sound.

• 41:52Speaker 00

  But I think really the interesting part is like where it breaks. And like how that is, you know, kind of pushed to, stretched to its limits to make something actually truly new. So that's, that's what I'm most excited about. I think another kind of thing that, that AI affords um versus traditional techniques is, um, you know, like more of a abstract, like I'll call it sort of like a feature space. Like some, say like a saxophonist's tone is a really sort of like abstract nuance. Like a lot of things combined to create that specific musician and their specific tone. It's kind of way too many parameters to model. Uh, though people try and there's a lot of really cool saxophone models.

• 42:37Speaker 00

  But AI kind of lets you jump that whole modeling part and just have these learned features. And then maybe you can even sort of distill some of the learned features. Like maybe you can extract like attitude from the saxophone model and maybe replace it with something else. And so this is another kind of like really interesting, exciting area of how, of how these models can be used that, that is a, a big departure from what we currently have.

• 43:04Daniel Whitenack?

  Well, I know I'm super excited to watch all of these things unfold and also to get one of these little boxes to plug my guitar into whenever it becomes available. Oh, thank you so much for sharing all this with us. It's been a real pleasure and I love how, you know, just to see the creativity that people are using AI for creative tasks, I think is really interesting, but there's also creativity in how you've gone about applying AI, which is amazing. So great work and thank you so much for joining us.

• 43:36Speaker 00

  Yeah, thank you both.

• 43:41Speaker 02

  Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you at changelog.com, including our brand new show, Ship It with Gerhard and Lazo, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is in implementation detail. Subscribe now at changelog.com/shipit or simply search for Ship It in your favorite podcast app. You'll find it. Of course, the Galaxy Brain move is to subscribe to our master feed. It's all changelog podcasts, including Practical AI and Ship It in one place. Search changelog master feed or head

• 44:21Speaker 02

  to changelog.com/master and subscribe today. Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Breakmaster Cylinder. We're brought to you by Fastly, OnStarfleet, and Linode. That's all for now. We'll talk to you again next week.