AI in the browser

We’ve mentioned ML/AI in the browser and in JS a bunch on this show, but we haven’t done a deep dive on the subject… until now! Victor Dibia helps us understand why people are interested in porting models to the browser and how people are using the functionality. We discuss TensorFlow.js and some applications built using TensorFlow.js Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean now offers three managed databases https://www.digitalocean.com/products/managed-databases/ — PostgreSQL, MySQL, and Redis. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. Changelog Weekly https://changelog.com/weekly – We track, log, and contextualize what’s happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com/weekly https://changelog.com/weekly to browse the archives, subscribe, and push the easy button on your continuing education. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Featuring: Victor Dibia – Website http://victordibia.com, GitHub https://github.com/victordibia, LinkedIn https://www.linkedin.com/in/dibiavictor, X https://x.com/vykthurChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: data2vis https://github.com/victordibia/data2visTJbot http://www.research.ibm.com/tjbot/TensorFlow.js https://www.tensorflow.org/jsHandtrack.js https://github.com/victordibia/handtrack.js/ConvNet Playground https://convnetplayground.fastforwardlabs.com/#/JS Party https://changelog.com/jspartyUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com/Changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/Changelog. Again, do.co/Changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter. We're at Practical AIFM. And now into the show.

• 1:47Chris Benson?

  Welcome to another episode of Practical AI, where we try to make artificial intelligence practical, productive, and accessible to everybody. My name is Chris Benson. I'm Principal AI Strategist at Lockheed Martin. And with me today is my co-host Daniel, who is a data scientist at S I L International. How's it going today, Daniel?

• 2:06Daniel Whitenack?

  It's going great, Chris. It's a beautiful fall day here in Indiana and can't complain at all.

• 2:14Chris Benson?

  Cool. As we are looking into the fall here, I'm getting excited. I guess a couple of weeks after this episode comes out, I will be at Nvidia GTC DC in Washington DC. And I just wanted to say if any of our listeners are there, I'll be walking around a good bit of the time in a practical AI t-shirt with a jacket. And if you happen to see me, I hope you'll come up and introduce yourself and say hi.

• 2:39Daniel Whitenack?

  Sounds good. I can't wait to hear what you learned there and hear about some of the content that's presented. I'm sure it'll be good.

• 2:48Chris Benson?

  Sounds good. Well, we have a pretty good episode, I think, lined up today. We are going to be talking about artificial intelligence in the browser.

• 2:56Daniel Whitenack?

  Oh, exciting.

• 2:57Chris Benson?

  I know. I'm pretty excited about it. And I know both of us have done over time, a fair amount of web development separate from the AI stuff. So this is the episode where we get to start combining them together, hopefully. With us today, we have Victor Dibia, and he is a research engineer at Cloudera's Fast Forward Labs. Welcome to the show, Victor.

• 3:20Speaker 03

  Well, thanks a lot. It's really good to be here. And I totally look forward to discussing machine learning the browser.

• 3:28Chris Benson?

  Fantastic. Well, I guess if you would start off by telling us a little bit about yourself, kind of how you, you know, how you got into this area and kind of how you found yourself arriving at Cloudera Fast Forward Labs and so that you could start this interesting line of work.

• 3:43Speaker 03

  Sure, absolutely. And so I could talk about it in terms of my educational background and then ease into the whole professional track.

• 3:56Chris Benson?

  Sure, whatever works.

• 3:57Speaker 03

  Yeah, and so my my background is a mix of computer science, that's software engineering, and a bit of human computer interaction, and more recently, applied artificial intelligence. I have a master's degree from Carnegie Mellon University that's focused on software engineering, and some some management courses. And as part of that, I did this thesis on building a web-based public key crypto system that was introduced by my advisor. And and that was a nice experience, it's just that there was a lot of time spent implementing cryptographic functions in Java. And after that, I felt, you know, it will be nice to move away from just the technical

• 4:41Speaker 03

  aspects of computer science and also look at some of the human aspects of computer science. And right after that, I I made an interesting choice, I moved to Africa, Lagos, Nigeria, to be specific. And I started a company focused on making software focused on the African market. And I did that for about a year, and as part of that, I also I also taught at a university in Lagos, Nigeria, through some project that was sponsored by MIT and Google. And as at that time, I I figured out, you know, I'm really interested in human aspects of computer science. And I had all this experience building software tools within the framework of a startup and also teaching at a university.

• 5:25Speaker 03

  And then I decided to to do a PhD, and so my PhD was in information systems I did at a city university of Hong Kong. And it was, the main focus was quantitative user user behavior studies. And at some point during my PhD, I had the opportunity to do an internship at IBM Research, and that's kind of where I got into AI. And so I I interned with a group called the Cognitive Environment Lab at IBM.

• 5:55Speaker 03

  And most of what that group did was trying to figure out good user experiences for applied machine learning. So we spent time taking APIs built by other research groups, so APIs around speech to text, text to speech, and computer vision. And our goal was to use these tools and build them into interactive, in some cases, room scale experiences. And that's kind of where all of my interests with AI kind of started. I started out applying models and after a while I spent time implementing some of these models in TensorFlow and Keras. And essentially made the transition to start applying some of these

• 6:40Speaker 03

  custom built models to new problem spaces. And so I was at IBM Research for as a postdoc and then a research scientist. And earlier this year, I joined Cloudera Fast Forward Labs as a research scientist. And so it's a bit of a journey, but it's kind of how it all went down.

• 6:58Daniel Whitenack?

  Yeah, so when you're talking about like user interaction or interfacing with AI, are you mostly talking about the sort of experiences like in Gmail, like autocomplete or voice to text and these sorts of things? Or even deeper in terms of helping a user kind of understand AI or use AI more effectively? How would you classify that sort of field of the interaction between humans and AI?

• 7:29Speaker 03

  Right, so my work kind of cut across the two areas that you mentioned. And so one specific line of work has to do with using AI to make the user interaction easier. And so that might be in terms of reducing the cognitive load associated with specific tasks. And good examples of that, the kind of thing you see in Gmail autocomplete, just start typing out an email and some LSTM model recommends a few completions.

• 8:00Speaker 03

  And so an interesting project that I worked on in a similar line is something called Data to Vs. And so with that project, we designed a neural network, a sequence to sequence model that could take user data and based on user data, it will propose about 10 to 15 visualizations that made sense for that sort of data. And so the value here is that an analyst who perhaps has limited experience with offering visualizations or writing code for visualizations could take a tool like that, upload their data, and the model will actually generate code for about 10 to 15 visualizations that they could either accept or they could modify to get their task done. So that's one line of work.

• 8:45Speaker 03

  And so another interesting line of work at this intersection of HCI and AI has to do with tools that make AI more accessible and more essentially easier to use for software engineers or other type of technical or non-technical users who strictly don't have a background in machine learning and AI.

• 9:06Daniel Whitenack?

  I know some people talk about democratizing AI. Is that sort of what they're meaning, I guess?

• 9:12Speaker 03

  Yes, yeah, that's a good umbrella term to kind of describe that sort of work, democratizing AI. I'm always not in a hurry to use that because that term can be reloaded and people have used it in all sorts of ways. But yeah, it's a good related term, the whole idea of democratizing AI. And it has its advantages and disadvantages. But the goal here is that, you know, if we make AI more accessible, then there are a lot of benefits that can come up. And so, for example, we want people with various backgrounds, various interests, various domain expertise actually coming and using AI tools.

• 9:53Speaker 03

  And that way, you know, we can kind of increase the diversity that kind of comes into this space. And one example of a project that I worked on in that area was something called TG-Bot. That was while I was at IBM. It's actually a maker kit made out of cardboard. And so it's a Raspberry Pi on the inside. It's a cardboard piece that you could fold up into a humanoid-looking robot.

• 10:16Speaker 03

  And we had actually gone ahead and written a JavaScript library that made it very easy to take a bunch of IPM pre-trained machine learning models, integrate that into the bots and essentially prototype things like you would see on Siri or any other like AI-enabled hardware device. And so that had a lot of success with schools, teachers, designers actually using it to actually start to use AI tools in different ways.

• 10:41Chris Benson?

  So got a question for you. You know, you were talking a few minutes ago on UI about, you know, with Gmail and things and you've talked about JavaScript and I was kind of curious to even take a step back a little for a moment and ask, we're talking about the browser here, why do you think people would want to run machine learning and AI models in the browser versus other environments? What is it that that kind of drove that interest into the browser?

• 11:07Speaker 03

  Right, so machine learning in the browser, it's a fairly new area and I guess most of the time when I talk about it or discuss it with people, there's always a form of healthy skepticism and I think it's for good reason. And so if I'm gonna take a step back just to discuss the two interesting aspects of machine learning, and so there's training. So this is the part where you create a model, which is essentially a function that learns mappings between your input data and some kind of target. And essentially you get all your data collected, you have it cleaned up and you go through the process of kind of learning this function that solves a specific task. And so that's the training phase.

• 11:51Speaker 03

  And then the second part of machine learning is inference where at some test time, you get this model that's trained and you get it to actually perform the task. And typically we've used languages like Python, Java, C, R, Scala, Julia. And these are typically backend languages and they have a lot of nice functionalities like.

• 12:13Speaker 03

  Hardware, direct hardware access, multi-threading, and it makes them work really well for intensive computations like machine learning. However, typically, most of all, this has been done on the backend. There's been a clear separation between what you can do in the frontend in an environment like the browser, because the browser has a bunch of limitations. And so it's single-threaded, and it's a sandbox environment with very limited

• 12:43Speaker 03

  access to system-wide features. However, it turns out that there are a few benefits that kind of make this proposition interesting. And some of my favorite reasons have to do with three specific benefits. And so the first would be privacy. And perhaps this is the most compelling and interesting benefit that I really care about. And so if you could take a model and you could deploy that in the browser, then you could create an environment where the user data actually doesn't get down to any backend server. And I could give some examples of that somewhere down the line as we continue the conversation. The second interesting benefit, why machine learning might be interesting in the browser, has to do with the ease of distribution.

• 13:29Speaker 03

  And so a few years ago, I had a couple of friends who really wanted to get into machine learning. But they did give up because they spent a couple of days just trying to install TensorFlow. And so while over the last two years, the user experience has become a lot better, there's still a lot of challenges, especially if you want to get a machine learning model or an application that uses machine learning deployed in an end-user system. However, if you go ahead and do that in the browser, this is a much straightforward and much, much easier developer and end-user experience. And then finally, the last interesting feature has to do with interactivity and latency.

• 14:10Speaker 03

  And so off the bat, the browsers, the web is designed to be interactive, and it's really valuable for crafting rich, interactive experiences. And so if there are situations where you have a model and you want to easily tailor that around user data, make changes and personalize that for a user, then the browser is a really excellent environment to interactively do all of that. And so these are three interesting reasons why I think it makes sense to actually explore machine learning in the browser.

• 14:40Daniel Whitenack?

  So yeah, that's a great explanation. I'm wondering if kind of along with that explanation, since we're always trying to be practical here, and like you said, there's probably a lot of listeners who are familiar with Python or Java or whatever it is, the sort of backend languages. Could you just kind of describe the JavaScript ecosystem a little bit? So there's like JavaScript, but then there's probably things like Node.js and other things that people have heard of. Could you kind of describe, I guess, in general, what those things are and how like machine learning is kind of touching each of those? Or maybe it's specifically touching one thing like vanilla JavaScript or whatever it is?

• 15:25Speaker 03

  That's a great question. And so one way to think of what you could actually do in terms of machine learning in JavaScript is to think in terms of the tools that are available today. And right now, most of that is TensorFlow.js. For listeners who are not familiar with TensorFlow.js, it's a JavaScript library designed to enable machine learning in the browser and any other environment that's built with JavaScript like Node.js. And so most of my conversations around, you know, when I talk about essentially implementing machine learning in JavaScript, most of the time, I'm actually referring to implementing machine learning using the TensorFlow.js library.

• 16:09Speaker 03

  And so with regards to the environment and platforms that are supported, TensorFlow.js allows you to build, train, and perform inference both in the browser, the browser environment as part of a front-end web application. And it also lets you build, train, and perform inference as part of a backend Node.js application.

• 16:33Speaker 03

  And so the library, one way to think of it is to think of it as having a few different installation versions. And so there's a version that could be bundled into a web application. It could be a vanilla JavaScript application where you could just, in your webpage, you could include a minified version of the TensorFlow JS library. Or you could install it using build tools as part of a React or Vue JS application. And so that's for the front end browser. Similarly, on the back end, you could npm install TensorFlow JS. And essentially, for your backend applications built in the JS, you could integrate TensorFlow models.

• 17:15Speaker 03

  And you could also integrate, install the GPU version of that same backend library. So to the npm install TensorFlow GPU. And one other thing that I guess people will be interested in learning about would be around the performance of TensorFlow JS and both the backend and the frontend. And so in the browser, it turns out that TensorFlow offers a vanilla CPU backend and also something called the WebGL backend. And so Mr. Voss might be familiar with the WebGL standard. It's used for accelerated graphics compute. And the value here is that if you do have a GPU available on the machine

• 17:58Speaker 03

  through the WebGL standard, you can actually accelerate your computations right there in the browser. And so underneath TensorFlow JS in the browser will take advantage of optimizations already implemented in WebGL. And that's how it's able to accelerate computations in the browser.

• 18:31Speaker 04

  Hello there. This is Jared Santo, Managing Editor here at Changelog. The fact that you're listening to this means you are actively investing in your future in this industry. Things move fast and keeping up is hard work. Help us help you stay relevant by subscribing to Changelog Weekly. We track, log, and contextualize what's happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com/weekly to browse the archives, subscribe, and push the easy button on your continuing education. That's all from me. Once again, that's changelog.com/weekly.

• 19:22Chris Benson?

  So Victor, I was kind of curious. I know that you are involved with TensorFlow JS and have been using that. And I was really wanting to learn about what it is and kind of how it fits in to the world of regular TensorFlow. Or does it? You know, what's the relationship between the two?

• 19:42Speaker 03

  And so TensorFlow JS is one of the libraries or frameworks in the broader TensorFlow ecosystem. And the primary benefit it offers is allowing a developer design, build, train, and perform inference on for machine learning models using JavaScript, either in the browser on the front end or on the back end in the JS. And so with regards to how it plays with the rest of the TensorFlow ecosystem, I like to think of it, the TensorFlow JS workflows in three, using three main approaches. And so the first is what I'll describe as the online workflow.

• 20:24Speaker 03

  And so with this workflow, you can structure out your machine learning models. Essentially, if you're building a convolutional neural network, you would specify input layers, the convolutional layers and your pooling layers, all of that is in tense for JS. You could train your model directly in the browser, user data, and then perform inference. And so clearly, there are caveats around this. You probably want to do this with small models or models that didn't have a lot of data. So that's the first approach, something I describe as the online approach.

• 20:59Daniel Whitenack?

  And in this case, the data, there's no actual data that's leaving the client device, is that right?

• 21:05Speaker 03

  Yes, that is correct.

• 21:06Daniel Whitenack?

  And does that kind of fit into that like privacy advantage that you mentioned before?

• 21:13Speaker 03

  Yes, yes, absolutely. And in this case, imagine you had user data already available on their machine. And your application, the browser, could get access to that data, train the model, and then perform inference without any data being sent to any backend server. So that kind of fits in with the privacy benefits. So the second, potentially more common flow is something called the offline flow, where you could train your model using a large amount of data and large GPU clusters or whatever hardware you have available. And for this process, you could use TensorFlow Python, like Keras models. And so just how you would train your models traditionally in TensorFlow Python,

• 21:58Speaker 03

  you could go ahead and build your models, train them on your GPUs or TPU clusters. And then you could export that model, that's the output of that process. And then you could use the TensorFlow GS converter to then convert that into a format that can be loaded in JavaScript or in a JavaScript application and then perform inference on that. So that's What I would refer to as the second of the offline flow. And then finally, the hybrid flow would be super similar to the offline flow, where you train offline, you convert your model and you import it. And then in the browser, you could go ahead and fine tune that model using user data right in the browser.

• 22:42Speaker 03

  And so these are three potential flows that a developer could take advantage of when using TensorFlow JS. And I guess the interesting thing to note here is that for models that you've trained using the traditional TensorFlow Python, TensorFlow JS offers a converter software, a tool that lets you convert those pre-trained models into a format called the web format that can be loaded in a JavaScript application. And so there's that opportunity to integrate whatever work you've been doing with TensorFlow Python and then bring that into the JavaScript or the web application space.

• 23:22Daniel Whitenack?

  So when a user, like if I'm a developer and I'm thinking about maybe like privacy is important to me or maybe the latency issues are important to me and I'm thinking about which of these scenarios I should pursue, whether I want to be fully online or offline or the hybrid situation like you're talking about. I guess part of that could be driven by the privacy concerns. But in terms of performance, like how much data or how big of a model can you train like in the online scenario versus like offline? And also like are some models, maybe the latest ones that are like, you know, however many billions of parameters, maybe you can't actually,

• 24:04Daniel Whitenack?

  or can you optimize those and fit them into the browser to run? What are the sort of constraints with those things?

• 24:11Speaker 03

  Right, so experience wise, I think most of the time, people would only train models, low parameter models, small models in the browser. And I guess the reason is pretty clear. The browser is not the multi-threaded, high-performance environment. And it's perhaps not designed to train large models using large data sets, images, thousands of images right in the browser. So what I've typically seen people do if they were going to train models from Scry, and so they would use, uh, train smaller models, perhaps using tabular data with maybe a couple thousand records and not much more than that. With respect to actually deploying high-performance models in the browser,

• 24:56Speaker 03

  I guess the, the good best practice would be to think of model optimization uh, during your model construction phase. And there are a few ways to go about that. And the idea is train your model offline, apply a bunch of optimizations, uh, and this could be model quantization, this could be model compression. And the goal would be to export a model that's small enough that it doesn't hinder the web or interactive experience. And then you would typically then import that, but mainly just for inference, inference in the browser.

• 25:34Chris Benson?

  So I guess, you know, how you've kind of covered these different ways of using it and how they integrate. Do you have any insight into, uh, maybe with, you know, Fast Forward Labs clients or, or anyone else you've come across within the industry about how people are typically using TensorFlow JS and how they're fitting it in, in real life, kind of aside from the, the options that you've laid out? Do you know what people are actually uptaking on?

• 26:01Speaker 03

  Right, so there are a few interesting use cases I've seen across the community and have also been highlighted by the, the TensorFlow JS community.

• 26:11Speaker 03

  And I think there's a really interesting experience or application by Airbnb, where as part of their user onboarding process, the user has to upload a photograph, an image of themselves. And right in as part of their onboarding experience, they have a Tensor for JS model in the browser that could look through whatever image has been uploaded and could tell the user if this particular image contains sensitive content or not. And so in some cases, users might upload their driver's license or other type of images that has potentially sensitive content. And the value proposition here would be to tell the user, you know,

• 26:55Speaker 03

  I'm able to offer you this service telling you that you have potentially privacy sensitive information here. However, this data does not get to my server and I never see.

• 27:08Daniel Whitenack?

  Yeah, so you don't even have to worry about storing that. You're just providing a service to the user on their side.

• 27:14Speaker 03

  Yes, so I think this is a really good and strong notion of privacy. The alternative would be that, you know, companies would typically say, oh, we have your data. We don't store it, we see it, but even if we store it, we will only use it appropriately. And so I think it's a stronger notion to say, we're able to offer you a service, but we never actually see your data because it never gets to your service. So I think that's a really interesting application. And so for developers who have similar privacy, similar scenarios where, you know, having a strong notion of privacy is valuable. I think Tensor for JS is a really strong candidate. Another interesting area, which is part of some work I have done,

• 27:59Speaker 03

  has been around attempting to design interactive experiences in the browser using the camera. And so I have this library called HandTrackJS. Essentially, it's an object detection model that's able to run in real time. And what it does is that it's able to track the location of human hands in any figure frame or image frame right then the browser. And so the idea is you could use interactions like that to create more engaging experiences like using your hand to control the game. Or for artistic installations, you could rather than using the mouse to interact with that art installation,

• 28:43Speaker 03

  you could just have users stand in front of the computer and based on the feed that's coming from the camera, they could perform things like selections, touch, grab and all of that. And the good thing is before now, to perform this sort of engaging interactions, you had to have some kind of hardware sensor or some really complex backend server. But now with Tens for JS and a well-optimized model, you could actually do all of that in the browser with no additional hardware and just access the user camera. And so these are two interesting examples I think are engaging. I think it's the space that's still growing.

• 29:21Daniel Whitenack?

  Yeah, so I'm curious. So I definitely see the advantage on the privacy side. I see the advantage on the sort of interaction side, like you're talking about with the hand track JS, which I definitely want to get into here in a second. But I was wondering, I anticipate that maybe some companies that like their AI models or their machine learning models, maybe it's part of their sort of bread and butter and how they make money or it's their market advantage or something. Would it be a problem for them to kind of push that model out to the browser? In the sense of like, can users that are using one of these applications just like open up a JavaScript console and like grab the model and use it themselves?

• 30:05Daniel Whitenack?

  What sort of like, are there certain concerns development wise around like keeping keeping your model in-house or things that you should be aware of as you actually port this model out to the browser?

• 30:18Speaker 03

  Right, yeah, you definitely raise a concern. And so at the moment, your model is just like any other web assets on your web application. And so just the same way you, the developer is responsible for securing elements like images and videos and every other content that's integrated into the webpage. You would need to think carefully around the security of the models. And so you're absolutely right. It's, it will be possible without any specific security implementations. I mean, beyond putting the model behind some kind of login or restricting access to

• 31:01Speaker 03

  base an IP address or any other security practice. If this is not implemented, it will be possible for the model files to be downloaded and perhaps used offline. And so I guess this is a concern that the user should develop, should think about while they consider putting models in the browser.

• 31:33Speaker 00

  Hey, guess what? Brain Science is officially launched. Episode number one is on the feed right now. So head to changelog.com/brainscience to listen, to subscribe and to join us on this journey of exploring the human mind. Once again, changelog.com/brainscience or search for Brain Science in your favorite podcast app.

• 32:11Chris Benson?

  So I'm curious, you know, as we've kind of talked about doing AI in the browser all this time, as we start seeing AI in the browser come about, you know, how is that going to change? How we're interacting with web apps now that these models are available to to kind of drive or provide services to the apps that we're engaging in? And do you expect with, you know, you were talking about HandTrack JS? Do you expect us to start having the ability to use gestures? Control things in a scroll and we're getting beyond just the mouse or trackpad and the keyboard into a more rich user experience on web apps?

• 32:50Speaker 03

  Right. As a researcher interested in human computer interaction and how AI can kind of influence or make that space better, I definitely entirely think that AI models, that they're well posed to kind of create that feature where we have a new set of interactions that are just purely enabled by AI. I think right now we have a lot of good examples and these things, they work so well that now we all take them for granted. And so examples like Smart Reply, Gmail Smart Reply, Smart Compose, even Auto Complete on our phones. And so I think in similar vein, I definitely see, you know,

• 33:34Speaker 03

  the opportunity to have gesture based interactions based on camera and maybe command much better voice based interactions, so speech, voice, computer vision, pointing gestures. And hopefully as these models become much smaller, much well optimized, it should be just as easy as adding just, you know, a really light JavaScript file and getting that to run. And so we're still a bit far from that. The main challenges here is that it's mainly around the size of these models.

• 34:11Speaker 03

  And so, for example, HandTrack JS, the current version, in terms of megabyte, the model itself is about 18 megabytes. And in web standards, 500 kilobytes are already a lot of data. And so I guess this is one limitation that kind of bars people from widespread adoption. But there's a lot of research that's kind of pointing towards a future where high performance models, we could actually find ways to compress them with very little loss in the loss of accuracy. And I think that kind of research is really going to be key to getting more of these models in production, easily integrated into a lot of web applications.

• 34:53Daniel Whitenack?

  So how does this fit into the, and this may be something you're familiar with or not, but I was kind of wondering while you were talking about that. And in terms of the limitations of the size of models coming down to devices, does this fit into the whole like federated training side of things? Where like some data scattered like between different phones and that sort of thing? Or maybe that's totally separate from the JavaScript side of things. Is the difference like, you know, in the federated sense, you're utilizing data that's on people's different devices and training a larger model that you would still use on the backend. Whereas like in the TensorFlow JS case,

• 35:35Daniel Whitenack?

  you're really interested in just kind of a single user and their data? Or are you aware of that, how that kind of fits into this picture?

• 35:45Speaker 03

  Right, so it's definitely related. Federated learning, like you mentioned, is the whole idea where we have a federated model. And at each end user or client device, we could train client models and then send some kind of model updates back to the server to have a much better or higher performance federated model. And the value here is that data still stays on the client devices, but just these model updates that don't compromise privacy, data privacy, gets sent back to the server. And so how is TensorFlow JS kind of connected to all of this? I guess the value here is that TensorFlow JS could be.

• 36:27Speaker 03

  A tool that lets you implement federated learning on a global scale. I mean, this choice is something that developers have to make. But with TensorFlow JS, you could definitely construct local models on end-user devices using local data. And depending on how you want to structure your system, you could send model updates to some federated model within your server. And I think there's some experimental implementation of a federated learning model. I think on the TensorFlow JS GitHub repository. And so if people are interested in exploring that more, that's a great place to start.

• 37:06Chris Benson?

  So I got a follow-up question as well, as we were talking about, you know, kind of the rise of gestures and richer interactions. And I'm just kind of curious what your thoughts are, you know, now that, you know, for on iPhone and other Apple devices, you have four ML. And, you know, on Android, you have the Google ML Kit. And is having those available on these end devices making a substantial difference in the ability to get there faster? In terms of that richer user interface, are you anticipating that those are, or are they already being used heavily with TensorFlow JS? To try to get every possible processing capability out of whatever device you're on?

• 37:47Speaker 03

  Within the TensorFlow ecosystem, there's this tool called TensorFlow Lite. I don't know if you're familiar with that.

• 37:55Chris Benson?

  Yep, we are.

• 37:57Speaker 03

  Yeah, and so TensorFlow Lite is all about, you know, finding ways to compress or optimize models such that they run on resource-constrained environments. And there's a bit of relationship with TensorFlow JS. Because the main difference being that TensorFlow JS is all about managing the whole machine learning experience in JavaScript, while TensorFlow Lite is all about making models smaller such that they run in resource-constrained environments. I think there's some relationship between both, because TensorFlow JS, you know, it does have some focus. And so the TensorFlow JS converter has some applications in model quantization,

• 38:36Speaker 03

  where you could actually explore ways to make your model smaller such that it runs fast in the browser. But I think these are slightly different efforts. And I also think the ability to have compressed models that run well on smartphones and resource-constrained devices like the Raspberry Pi.

• 38:56Speaker 03

  I think research in that general area should also be impactful and useful for and transferable, I guess, to to work being done with TensorFlow JS.

• 39:06Daniel Whitenack?

  So let's say that, for example, I know almost nothing about JavaScript, which is actually the case. And even though I've I have, you know, worked with front-end developers and developed APIs and that sort of thing, I don't really know anything about JavaScript other than like an occasional like hacking into something. So for someone in my position that's maybe coming from Python, what would you recommend in terms of getting hands-on with TensorFlow JS? Is it best to kind of start by looking at some JavaScript, you know, code tutorial online and going through that and then jumping into TensorFlow JS?

• 39:48Daniel Whitenack?

  Or are there kind of combined tutorials or resources that would be helpful? What are your recommendations there?

• 39:56Speaker 03

  Of course, it's always always valuable to get a refresher on the on the JavaScript language. And for beginners, people who are interested in getting into TensorFlow JS, I always recommend the tutorials on the TensorFlow JS websites. And so that is tensorflow.org/js. So they have a bunch of tutorials that walk you through the APIs that are available within the library. And so just to give an overview, TensorFlow JS supports two main types of APIs. So the first is a low-level linear algebra API. And so if you're interested in designing your multiplications, your additions, you want to implement your own loss functions,

• 40:41Speaker 03

  this would be the API to use. Definitely do not recommend it, except you really know what you're doing. And the second API it provides is something called Layers API, which is really similar in spirit to the Keras API structure or Keras API design. And so it's a really great way to reason about neural networks. And so if you have used the Keras API previously, using the TensorFlow GS Layers API should be something familiar and easy. So use fine traditional building blocks like LSTMs, 2D convolutions, transpose layers, batch normalization layers.

• 41:25Speaker 03

  And essentially, they are implemented just like you would implement that with Keras. And so to have your model built, you could compile it and then also get your accuracy metrics very similar to how you would do that in Keras. And so for people just interested in making the switch from, let's say, regular Python or maybe machine learning with Python to TensorFlow GS, I would recommend looking at tutorials on Layers API. And the other interesting thing here is that if you have models that are already built using and exported using the TensorFlow seed models format or the Keras seed models format,

• 42:09Speaker 03

  you can actually use the TensorFlow JS converter to convert that directly into the TensorFlow JS web model format. And then all you have to do is just spend some time learning how to load this and use that for inference in the web application. And so these are kind of like the mental steps to go through. It's useful to get a refresh in JavaScript. And then if you have some experience with Keras, the learning curve isn't that bad anymore. And the TensorFlow JS website has a bunch of tutorials and they have a lot of really good sample code on GitHub to get started.

• 42:47Daniel Whitenack?

  Yeah, so it sounds like the layers thing, like if I'm wanting to experiment like with layers, I could build a fairly easily build like a simple, you know, maybe a fully connected neural net that would, you know, solve some kind of toy problem. Let's say iris classification or something. I could do that fairly easily with the layers and kind of get a feel for it. But then I could also take like maybe a pre-trained model for image detection that's existing and try just to, just to do the inferencing part by using the TensorFlow converter. So those, those would be two things that would be reasonable to try first, maybe?

• 43:29Speaker 03

  Right, that is correct.

• 43:31Daniel Whitenack?

  Cool. You know, a lot of what you kind of have worked on personally as related to the hand tracking, that's, that's related to image and video based techniques. And I had just seen before the, before we started recording, I saw that you and the Fast Forward Labs team released this ConvNet Playground. And I was wondering if you could just mention, you know, what that is. That might be another great learning resource, kind of beyond TensorFlow JS, but also related because a lot of the stuff shows image detection examples.

• 44:07Speaker 03

  Right, yeah, thanks for that. The tool you're referring to is ConvNet Playground. And essentially it's a tool that lets you experiment and learn about how convolutional neural networks can be applied to the task of semantic image search. And so within the framework of that application, we have a very simple definition where semantic search is all about giving an image, find all other images that are similar to this image, but just by looking at their content. And so the implementation is really simple. We get a convolutional neural network and we use that as a feature extractor on all the images. And based on this feature extracted,

• 44:50Speaker 03

  we can compute some measure similarities in let's say cosine distance or Minkowski distance. And essentially that's how similarity is implemented.

• 45:00Speaker 03

  However, in practice, there's a bunch of decisions that a data scientist needs to make. And so what model do I use for feature extraction? If I was going to use a pre-trained model. And so there are dozens of pre-trained models out there. Inception, VGG16, VGG19. And even more recently, there are new architectures that are enabled by neural architecture search like EfficientNet and NASNet and MNASNet. And so the data scientist needs to make decisions on which one of these pre-trained models do I use? And even when they select an architecture, they need to also decide, do I use the entire model? Or do I use a subset of this model constructed from the original model?

• 45:45Speaker 03

  And so once they've made these decisions, they finally need to decide what similarity metrics might be best for these. And so we built ConvNet Playground to kind of create an environment where all of these decisions and all the computation for these decisions have been made. And the user can essentially interactively explore what the results look like. And so there's a search interface where we have some datasets. And for each image in the dataset, you could make a selection and then view how each model performs in terms of search accuracy or search quality for that particular search query. And then we have visualizations that let you compare, you know, how do the different model architectures compare?

• 46:28Speaker 03

  How well does the semantics for each model perform? And so we have UMAP visualizations of the feature embedding space. And we have a bunch of graphs that let you perform all of these comparisons. And so I'd really encourage everyone that has a chance and wants to learn about pre-trained models to kind of explore that. There's some of the really interesting insights there that one common thing that data scientists do right now is that to extract features, they would just select VGG16 and select its last layer and use that as a feature extraction. And so it turns out that this might be inefficient. The VGG16 has about 130, over 130 million parameters.

• 47:10Speaker 03

  But a model like Efficient Nets B2 has about 5 million parameters and it actually works better than VGG16 in terms of extracted features for natural images. And these are some of the insights you could actually extract by exploring that interface. And depending on the type of your data, so if you have retail data, if you have natural images, these different performances will change and you can kind of explore the space using ConvNet playground.

• 47:39Chris Benson?

  Cool. Well, this has been a fascinating conversation. Thank you, Victor, so much for coming on to the show and telling us all about tensorflow.js and these other projects and stuff. Really great conversation.

• 47:53Speaker 03

  Thank you.

• 47:54Chris Benson?

  And I guess for our listeners, we want to remind everyone that we have a number of different communities where you can reach out to us and have conversation. A lot of the show is built on your feedback and your comments. You can go to changelog.com/community and opt in. If you're on LinkedIn, there is a Practical AI Podcast group that you can join. We have our Slack community, which you can reach by the website. And also, because we're talking about JavaScript on this episode, we wanted to point out that the changelog also has the JS Party Podcast, which is a fantastic podcast having to do with all things JavaScript. And you can find that at changelog.com/jsparty. So we look forward to seeing you guys next time. Thank you very much.

• 48:41Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go on your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our errors before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to

• 49:25Speaker 00

  date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.