Automate all the UIs!

Dominik Klotz from askui joins Daniel and Chris to discuss the automation of UI, and how AI empowers them to automate any use case on any operating system. Along the way, the trio explore various approaches and the integration of generative AI, large language models, and computer vision. Sponsors: Statsig https://statsig.com/changelog – Build faster with confidence. Startups to Fortune 500s rely on Statsig to make data-driven decisions. Ship smarter and faster with the unified platform for feature flags, experimentation, and analytics. Our listeners get free white-glove onboarding, migration support, and 5 million free events per month. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Dominik Klotz – GitHub https://github.com/programminx-askui, LinkedIn https://www.linkedin.com/in/dominik-klotz-127a931b7Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: askui.com https://www.askui.comaskui on GitHub https://github.com/askuiaskui on LinkedIn https://www.linkedin.com/company/askuiaskui on Twitter/X https://twitter.com/ask_uiUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Speaker 01

  What's up, friends? I'm here with Vijay Raji, CEO and founder of Statsig, where they help thousands of companies from startups to Fortune 500s to ship faster and smarter with a unified platform for feature flags, experimentation, and analytics. So Vijay, what's the inception story of Statsig? Why did you build this?

• 1:02Speaker 02

  Yeah, so Statsig started about two and a half years ago. And before that, I was at Facebook for 10 years, where I saw firsthand the set of tools that people or engineers inside Facebook had access to. And this breadth and depth of the tools that actually led to the formation of the canonical engineering culture that Facebook is famous for. And that also got me thinking about like, you know, how do you distill all of that and bring it out to everyone? If every company wants to like, build that kind of an engineering culture of building and shipping things really fast, using data to make data-informed decisions, and then also informed to like, what do you need to go invest in next? And all of that was like fascinating, was really, really powerful. So, so much so that I decided to quit Facebook and start this company.

• 1:47Speaker 02

  Yeah, so in the last two and a half years, we've been building those tools that are helping engineers today to build and ship new features and then roll them out. And as they're rolling it out, also understand the impact of those features. Does it have bugs? Does it impact your customers in the way that you expected it? Or are there some side effects, unintended side effects? And knowing those things helps. You make your product better.

• 2:11Speaker 01

  It's somewhat common now to hear this train of thought where an engineer developer was at one of the big companies, Facebook, Google, Airbnb, you name it. And they get used to certain tooling on the inside. They get used to certain workflows, certain developer culture, certain ways of doing things, tooling, of course. And then they leave and they miss everything they had while at that company. And they go and they start their own company like you did. What are your thoughts on that? What are your thoughts on that kind of tech being on the inside of the big companies? And those of us out here, not in those companies, without that tooling?

• 2:49Speaker 02

  In order to get the same level of sophistication of tools that companies like Facebook, Google, Airbnb, and Uber have, you need to invest quite a bit. You need to like, take some of your best engineers and then go have them go build tools like this. And not every company has the luxury to go do that, right? Because it's a pretty large investment. And so the fact that the sophistication of those tools inside these companies have advanced so much, and that's like left behind most of the other companies and the tooling that they're they get access to is that's that's exactly the opportunity that I was like, okay, well, we need to bring those sophistication outside. So everybody can be, you know, benefiting from these.

• 3:29Speaker 01

  Okay, the next step is to go to statsig.com/changelog. They're offering our fans free white glove onboarding, including migration support. In addition to 5 million free events per month, that's massive. Test drive Statsig today at statsig.com/changelog. That's S T A T S I G dot com slash changelog. The link is in the show notes.

• 4:12Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder at Prediction Guard and I'm joined as always by my co-host Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well today, Daniel. How are you? I can't complain. It's like these two weeks in the Midwest in the United States that I enjoy each year where it's like between hot and really, really cold. And so, yeah, I'm really, I'm excited about that. And yeah, just cranking away. I'll be at the Intel Innovation Conference next week, which is going to be a fun experience to see some of what they're doing in the AI space and talk to them about some of the, some of the stuff we've been trying on Intel hardware. So yeah,

• 4:57Daniel Whitenack?

  it's a lot of good stuff coming up. I'm really excited. Actually, it was through that Intel community that I met Dominic from Ask UI, which is our guest today. So, welcome Dominic. It's great to have you here.

• 5:11Speaker 04

  Hey, thanks, Don, for having me.

• 5:14Daniel Whitenack?

  Yeah, well, Ask UI, could you tell us a little bit about, first, maybe just what is it about UI that Ask UI is concerned with? And how did you get to start thinking about UI and automation?

• 5:33Speaker 04

  Maybe a little bit back on what are we doing? We try to free humans from being robots. What does it mean in particular? Not only you have repetitive tasks on user interfaces. And what we are trying to do to bridge the gap between you have to be the programmer versus you can describe in a natural language what you want to do. So your intention, what you want to automate. For example, if you want to login in your Facebook, then you can say with us, please click the login button, please fill in this, this is a credential, it's natural language,

• 6:12Speaker 04

  which allows also non-technical guys to automate user interface and lower the border, which you can automate or start to automate things. And the second question was a little bit about...

• 6:27Daniel Whitenack?

  Why did you start caring about this problem?

• 6:31Speaker 04

  Why, why, why, why I started caring? A little bit of background about this. I previously came, my background is a software developer. I was working previously at Siemens before I founded with Jonas, my co-founder Askei. So there, what I was doing there was some plant automation system where we had to test everything because our systems have to run for 365 days a year. So everything should be tested well enough. So we were standing a lot of inside the testing level and test everything through. Then it switched also a little bit to Paltm, the same thing. You have to test, you have to stand directly and test your application.

• 7:12Speaker 04

  And then I was moving to a new organization, which tries to modernize or bring the agility within Siemens, where I learned a lot about Scrum, agility, bring all the new stuff, tools. But I had also the problem because Selenium and other, other tools couldn't solve the pain to write new tests, the user interface tests in the way. And then I was thinking, hey, can we not solve this with AI? Because AI can understand visual information and can understand natural information. So I thought, hey, can we not combine those? And all of this, the journey started.

• 7:49Daniel Whitenack?

  This might be a completely ignorant question, but I just want it for my own context as well. Now, in the, in my background as a data scientist, I've participated many times in the fun activity of web scraping. And through, through that, I know like, oh, there's a lot of ways in which web data or UI data might be exposed. So could you talk a little bit about like how?

• 8:19Chris Benson?

  What does the data look like? Like, what does the problem look like as you're trying to get an AI model to interact? Or use an AI model or natural language input to an AI model to then interact with either a UI or a webpage, especially when a lot of those things can be quite varied in terms of how they're built? Is it more of a visual thing or is it something else?

• 8:43Speaker 04

  Yeah, the real and the visual part. For example, when you're talking about Selenium, Selenium is only working with web interfaces and you cannot use this, for example, on Android and so on. What we are doing, we are doing with a screenshot of the system. So it means we can take a screenshot of the application, we can take a screenshot directly of the operating system and then our AI model, what we trained, can detect user interfaces. It means we're detecting buttons, we're detecting text, we're detecting text fields, we're detecting checkboxes, we're detecting icons. And so we have already trained a model which can understand the visual representations

• 9:25Speaker 04

  of user interface and then we connected our natural language part to it to match the intention, what you have, for example, click on the login button, to the real login button on the user, the fact, and then we are moving the mouse there. So this is a little bit different from the concept versus I connect directly to the application and then try to scrape the source code of the application to get my information. Maybe this is the best way to differentiate.

• 9:53Chris Benson?

  So you're kind of starting with a screenshot instead of doing web scraping?

• 9:57Speaker 04

  Correct.

• 9:58Chris Benson?

  And then just doing classification on the screenshot itself to do that?

• 10:01Speaker 04

  Correct. So we have object detection model in place, which really takes a screenshot and detects all the elements on it.

• 10:08Chris Benson?

  As a quick follow-up, when you do the classification and identify what you have in the screenshot, how do you tie that in with the tests desired? So you have a collection of tests that you're trying to automate. How are you like tying, like, I got a screenshot, I got, I have a button, I have a text field. How does that tie into a particular, like, a unit test or something?

• 10:28Speaker 04

  Currently, we have our TypeScript application, which you can download, which is also available as an ASCII MPM package, which you can use. And with this, you can directly start, install it, and then you generate a standard test. And then you have, for example, the chat test, test block, or your Java JUnit test block, where you then can write ASCII dot click dot with text, dot, so on. And the background, what happens? We have a controller then installed on your local system, which connects to the operating system, takes a screenshot, and also has the ability to control the mouse. Then we're taking the screenshot.

• 11:11Speaker 04

  We are connecting the instruction, the click on button, for example, then this to our inference backend. Then we get the result back, and then we move the mouse there. And with this, single steps, as you would do it in Selenium, you can then write your workflows or tests to automate every operating system. We are currently running on Windows, on Mac, and on iOS, especially for legacy application and the Windows environment. We can test also these applications. We are not limited to web. But I would say, did this answer the question?

• 11:47Chris Benson?

  It does, it does. Thank you, I appreciate the explanation.

• 11:51Daniel Whitenack?

  So you obviously had a certain perspective of, when you came to this problem, because you had worked at Siemens, you had thought about this sort of automation. I'm wondering, as you've developed this technology and seen others that might have had this need or experienced this pain, what is the sort of range of things that you're seeing people either do or want to do with this kind of automation technology that might fit the use case, sometimes fit the use case that you had in mind initially, but sometimes might be a sort of new things that you didn't think about previously?

• 12:30Speaker 04

  I think because we are using AI technology and with AI technology you are a little bit more flexible what you can describe because you can learn it based on data and have not to explicitly describe which things you want to do. New use cases are what we started where we want to do regression tests for classical user interface tests. Then we moved a little bit in the direction of robot processor automation. I think where we were planning in the future is to automatically transfer unstructured data to user interfaces. So if you have, imagine you have a PDF, which in a formula, and then you want to say, hey,

• 13:14Speaker 04

  please, please, also normally workers will do this, which are copying, for example, the PDF or information from the PDF over to another formula. And what you can do then with our technology is to say, hey, please automatically copy all the informations over, which is here on the left side, to the right side. This is what, where we think where the technology will go, so that you can have not define the matching of the files, everything, also when you then connect also large language models which have the ability to understand the language a little bit better, or a little bit closer as we could, as humans can define it, then you can build such systems.

• 13:58Daniel Whitenack?

  Yeah, that's really interesting. I'm thinking of even like in, I had a friend that works in the offices, same office building that we're in here, and he stopped by and we were talking today. He takes screenshots of his work as he goes along in the day, like every so often, and he does this because he wants, like, he can't always remember like all the things that he was doing throughout the day. And so he takes these screenshots as a sort of historical record of like the various interfaces and what he was doing and such. So I had that in my mind as you were talking through all of these things. I'm like, hey, this would be really cool for him. He could, you know, potentially query certain things about his day and these interfaces

• 14:43Daniel Whitenack?

  and what he did and like create potential automations off of those.

• 14:49Speaker 04

  This is also a use case what we're thinking to buy to be a recorder in the background, which then collecting repetitive tasks and then say, for example, hey, We have now detected you have done this over the last week, five times. Should we automate this for you, for example?

• 15:05Daniel Whitenack?

  Yeah.

• 15:06Speaker 04

  There are also cases where the technology can go. Because we have the understanding of human interfaces and with a lot of engineering stuff around this, then you can build such nice systems.

• 15:16Daniel Whitenack?

  That seems like an automation, like automation and AI seems to like scare people for some maybe justified reasons, some unjustified, or I don't know what's justified. But that's like an automation that like, why wouldn't I want that to to save myself some time? I would love to automate some of my repetitive tasks.

• 15:36Chris Benson?

  So one of the things I'm wondering is you're having looked at your website is you talk about like all the different platforms, you know, you get web apps and enterprise apps and everything. Are you able to take the same approach across the different platforms that you're targeting for that? And is that what makes it so flexible by doing screenshots and stuff? Is that what does it? Or do you have, how do you break down the challenge of getting from, you know, addressing one platform in your initial and then starting to spread out across the other platforms?

• 16:06Speaker 04

  From the other, the main technology is to accessing the screenshot from the operating system and to controlling it. I think you, if you have tried to do it, there are a lot of open source software already available, which can do this. If you have such technology accessible, then you can take the screenshot. For example, we have Android support, we can take the screenshot and it's the same model and the same technology behind them as we are using this on the Windows operating system or on the Linux operating system. And we have a general model which can solve all the tasks.

• 16:42Daniel Whitenack?

  So Dominic, I'm already thinking of a lot of use cases that maybe I might want to automate and interact with the system. One of the things we were chatting about, like prior to actually hitting the the record button on this episode was maybe the unique approach that Ask UI has taken in terms of more of a software engineering approach to understand. Understanding how these machine learning and AI systems work and utilizing them in a in this sort of like more practical software engineering approach to this. I'm wondering if you could talk about that in a little bit more detail and

• 17:23Daniel Whitenack?

  how you've approached these problems that you think is maybe unique or at least represents your perspective on how to build systems like this.

• 17:32Speaker 04

  Yeah, first of all, what I see or what we saw already in the past, the research area has built a lot of models, has released them on the public, but then it stopped it after you have published a paper, you have no interest anymore to to bring this to production. What we also see in that currently beginning from the 2020s, that a lot of new application are coming which try to solve this, to formalize this as software patterns as I know at the beginning, I started with the machine learning. I was wondering are there now software patterns like metric pattern, like trainer pattern, like other kinds of pattern which you can reuse and which you can communicate in better way.

• 18:18Speaker 04

  Why are we the good at software development nowadays is because we have standardized the patterns which we use and use them everywhere. So this is what what I was where we're always searching a little bit and also the tooling. And how did we this approach this from the machine learning, where we have different teams a little bit. We have the first thing what we have done, we have built an application which use directly our model and we use plug in the best, the first model which we can produce and we had we started with our model which has only five images for object detection, for detecting this five images training data set, which is not a lot.

• 19:01Speaker 04

  So what we done this to prove that this end to end possible. And we mocked a lot of things away and then we going out to the customers and say, hey, can you work with this? And they complained, yeah, it's nice, but your model object detection model don't work so good. So what we done the next one, we collected more data, trained the better model, then we go out to the customers now enough and then they were okay, it's going the direction, but we could only support one application at the beginning. And then we had related based on this and tries to connect all things together. What does means on the other side, we started directly to taking going out to the customer

• 19:45Speaker 04

  and then increasing everything. And now we are making because this was hard at the beginning because I mentioned already that in 2020, all the tools, path of sliding and meter flow and so everything came up, which you could reuse. And now we are migrating more and more to the data pipeline and trying to bring everything to the customer so that they can also train by themselves. But this is our software engineering approach where I say, hey, bring everything to the customer, let the customer complain, then you're doing the next iteration steps. This is the last move.

• 20:21Chris Benson?

  So as a follow up, I want to ask you to kind of flip flop what you just covered a little bit. And what's it like to engage from the customer's perspective? Because you're kind of taking it from your perspective and we take it over. So if you're a customer and you start to use Ask UI and deploy it, what does that picture look like? What does the customer go through as they start deploying or start utilizing the service?

• 20:43Speaker 04

  What he's doing too. There's also a little bit history, but what is this now? You're going now to our website, then login with your credentials and then you can directly upload your first screenshot and then simulate on this. So do a simulation directly on the screenshot so that you see the aha effect. And then the next step, what you want to do, if you have created your workflow a little bit, then you want to automate this and you send then schedule this, for example, in a Docker container in the background we have already. Also, if you're in the web and then after time you get the result and you're happy

• 21:25Speaker 04

  that you have automated with really easy things, the workflow. And what we are now doing is to reduce also the hassle to that you have to learn less as now because. Yeah, this is a thing. We're trying to bring all problems from the user perspective away. The user has a really easy life to create automations, to maintain automations, and to schedule all the stuff, set up all the testing environments. Because it's not only the automation part what you're interested in. You're also interested in where can I schedule it? And also broader thing, how can I connect data and so on? This is what we're doing, what we're currently doing.

• 22:09Daniel Whitenack?

  That's awesome. I want to propose what I think is probably a bad idea, but I want to get your reaction to it. So I'm wondering, there's this way now that you've enabled people to automate their interactions with various UIs. And there's plenty of cases where I don't really care to interact with a UI, but I do need to accomplish a task. But I also, like, for example, one of these that I struggle with all the time, right, is like AWS and its interface, which is just like, you can do everything, right? But it's super hard to understand how to do anything, right? Is there an opportunity?

• 22:53Daniel Whitenack?

  Let's say, and again, I think this is probably from the start a bad idea, but let's say I just gave some sort of agent tied in to ask UI my credit card information and whatever, and just said, "Hey, I want you to create a AWS account and spin up this infrastructure and do this. And then when you're done, like, give me the URL so I can access my..." Like, "Here's the GitHub repo. Tell me when it's ready, and here's my URL." Now, I imagine that would also need to tie into like other external knowledge, like the documentation from AWS or something. But is this type of scenario anything that, like, you've been talking about internally

• 23:38Daniel Whitenack?

  or see as things that might come about in the future? As soon as you start automating things around UIs, there's the one side of it, which I think we've talked about a lot, which is automating the things you've already done with UIs. But what if you want to do things with UIs that you haven't done yet, but don't really care to learn how to do?

• 23:59Speaker 04

  This is also our plan to do this to leverage. We already played a little bit around with large language models, giving all the documentation to do this and also tries to translate, for example, Google documentation and create out of this. This was working quite good and the direction will go there. So in the future, we can do this. So there was another part that was the attention base. Please create me an ET2 instance and here I'm a credit card information. This was intention. Please do this. You can do this, but the main problem with this is humans. Also, if you talk to your colleague or to someone from another nation and sound, you will always have communication hurdles.

• 24:44Speaker 04

  So you have always to have the tiny steps between where you then can correct a little bit. That's one thing. The other thing is are we trustful for credit card information? For this, I have to say your submitting also on GitHub or GitLab, your security opens to your WS access. There are standards already in place which are also validated, which can do this and this is what we have to follow. We have also business, also enterprise customers, which wants to have the standards. So we have to be compliant with the standards. So from our side, it's no problem. You can trust us. But on the other way, you have also the possibility,

• 25:26Speaker 04

  because it's nice to create the tests online, download everything and execute this locally on your machine without any access on our side. You can also hide all the information or the secrets, I would say, on your device. And I have the guarantee that they are not leaked on anything.

• 25:46Daniel Whitenack?

  Yeah, that was another thing I was going to ask is like, hey, if I'm going to a site, like, let's just say, I'll give an example because I love the product and what they're doing. So Chris, you remember we had Josh from Koki on talking about their voice studio and all that. I've been using that recently. You go in, you can input your text, like a sentence and synthesize a voice. And then you can change like the the language or something and then like go through export the file. So there's like, there's not just a UI interaction here. There's like input data and output data that might include things like passwords. It might also include things like, hey, I generated a file with this UI. Like, where does that go?

• 26:30Daniel Whitenack?

  Like a quick download or something? I don't know. What's the best practices around as you're automating these things? How should a customer think about these like inputs and outputs? And how do you handle those in terms of what you're building?

• 26:44Speaker 04

  I would forward this question to how are you normally doing integration tests or end to end tests? You have always the same data there. So first of all, to recommend customers, try to use always synthetic or generated data. Don't leave production data to it because it's security. This is a double processing. What you learn when you when you start automating and testing, don't use production data. And then the other thing, you have to apply security standards. So there are environment variables or secret files which you can inject to it. Go with it, do it, use our security function that you're not sending to us anything what's related to it.

• 27:29Speaker 04

  And this is what I could recommend to the customers.

• 27:32Chris Benson?

  I'm curious, we've been talking about testing for a while and most of the use cases have been unit tests. Are there any other types of testing like integration testing that you're able to do? Is there a workflow for those kinds of things? Or is this really focused on kind of the screenshot itself and everyone stands alone? Is there any way to tie them together?

• 27:51Speaker 04

  Yeah, but you can tie them together. What we are, we are only a library which you can use in TypeScript. In the future, we will also support Python and other languages to make this technology also available. But you can also combine this with Selenium or some other techniques. You can connect to MongoDB, database, getting the data out, processing this into other systems. Yeah, we are really flexible in this race because our main concept, this is maybe also another thing. What's unique on our side? We are thinking that automation or user interface automation or low code user interface automation is to a certain level nice.

• 28:33Speaker 04

  Because you can give the ability to other peoples, but at a certain point, you're reaching the limit. And for this, you need developers to build some nice stuff to automate or to connect, for example, to MongoDB or some other sort of software. In this case, you can always have always the possibility to go from our low-code view to our code view and insert directly code. So this is no problem. You can do this. And you can also install other libraries.

• 29:09Speaker 00

  This is a changelog news break. Pagefind looks pretty cool. It's a fully static search library that aims to perform well on large sites while using as little of your users' bandwidth as possible and without hosting any infrastructure. It runs after your static site generator, like Hugo, Elevene, Astro, etc., and generates a static search bundle to add to your built files. It then exposes a JavaScript search API that can be used anywhere on your site. Quote, "The goal of Pagefind is that websites with tens of thousands of pages should be searchable by someone in their browser while consuming as little bandwidth as possible." Pagefind's search index is split into chunks so that searching in the browser only ever needs to load

• 29:53Speaker 00

  a small subset of the search index. Pagefind can run a full-text search on a 10,000-page site with a total network payload under 300 kilobytes, including the Pagefind library itself. For most sites, this will be closer to 100 kilobytes, end quote. I'd love to see a comparison. Link me up if you know of one. But my guess is that this could easily replace Algolia on lots of open-source docs and websites. One less service to depend on. Why not, right? You just heard one of our five top stories from Monday's changelog news. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 30:45Daniel Whitenack?

  I always like to ask guests who have really manifested some new idea that's really driven fundamentally by AI and machine learning this question. And that's, as you were building out this product, what challenges did you find in using? You already alluded this a little bit with like, oh, researchers release like all of these models and then like what happens after that? They're not really supported or maybe they die off or other things. So what specifically were the kind of machine learning or AI challenges that you faced as you were trying to make this work? You alluded a little bit to the data side of things and kind of adding data over time,

• 31:30Daniel Whitenack?

  but I imagine there's much, much more than that. So what are some of those things that stand out? Just practical things that you faced in trying to apply this technology to a real world automation problem?

• 31:43Speaker 04

  Yeah, maybe to start at the beginning, previously I was a software engineer. I had no clue about machine learning. I get a little bit theoretical knowledge, but theoretical is nice, but in practice it's completely different. So I had no clue that, for example, if you feel that the learning rate, you can bring your model to convergence. So such things I struggled at the beginning with also how you connect the layers at the beginning to make everything work. But then you've solved this problem, then the next step where I started to struggle is to making the experiments visible so that you generate the right metrics that you see, that you understand what you're learning or what you're not learning. Then

• 32:25Speaker 04

  when you have solved this challenge for yourself, so you have tried out TensorBoard, you have the next challenge to go to how can I manage the data? How can I increase the data? How can I version the data? So and then you came in the challenge, how can you do repeatable experiments that you can say, hey, you have done now progress step by step. And for this, you have to then to search a lot about what tools are out there, which tools are good. You need to know, know feeling. Then the next step is you've figured out that you have messed everything up and your code is totally... So you have to think a little bit out, how can I structure the code? This is what our manager previously was patterns.

• 33:09Speaker 04

  So you're looking in other repositories, how other developers have structured the code that it's more maintainable and more reusable. So then you're coming across, for example, PyTorch Lightning, which I say, which are doing this really great to build up a model in a modular way. So and then you're not only one developer, then you're two developer, two machine, like the researcher, then you have to communicate in ways. So you have to exchange data. So then the thing is you're starting to copy and paste data and send it over Slack or some stuff. Then you have to say, hey, it's totally stupid what we are doing. We need the data platform. And then with you going through step by step, you have reached the complete expertise to say, hey, now we need a complete data search.

• 33:53Speaker 04

  We need the metric system. There's how we exchange data between the teams. This is the way how we label data. So for example, also the other challenges, not only exchanging data, getting data. This is also the challenge labeling good data. Then you're looking at the labeling tools. So then you figure out the labeling tools are nice for standard use cases, but sometimes, especially in our case, because we have five different models, which are changed together in a nice way. That you have labeled different kinds of data for different models. Because this model types are fitting perfectly for this use case. So you're then thinking about how you can improve the labeling process.

• 34:32Speaker 04

  So now we are building a labeling tool based on Streamlit so that we can easily connect, for example, our inference part. That we can do a little bit pre-loading, then can automate stuff and to improve everything. So and then at the beginning, you remember that you talked once with some guy, which always said, if you do machine learning, you will end up to building a labeling tool where we now.

• 34:56Daniel Whitenack?

  You've reached the pinnacle.

• 34:59Speaker 04

  But it's a journey and I think when somebody would now ask me how to start, I would answer a completely different and a completely different way.

• 35:08Chris Benson?

  So that's literally what I was about to ask you because that was a fantastic journey that you just took us on about all the practicalities and you know, you solve one problem and you hit the next and you hit the next and you hit the next and, and you just described coming in completely new to this and taking it all the way to being very, very productive and all the practicalities. So that's actually what I want to ask you is you said I wouldn't do it the same way. I'd like to know, there is at least one person out there right now, if not many, that are thinking about AI. They may have dabbled in it, maybe, maybe not. They have an idea for

• 35:48Chris Benson?

  a startup, they're listening to you and they're going, that's the guy, you know, who started doing this, but I have my own idea. What would you tell them? Like, how do you get started, you know, to get going? Because this is a daunting field to break into.

• 36:03Speaker 04

  You mean only the machine learning part or to start a startup based on machine?

• 36:07Chris Benson?

  Mostly the machine learning, like how did you learn? It's a skill set. It takes the time to digest and it's constantly evolving. How did you digest that skill set so that you could be productive?

• 36:18Speaker 04

  Okay. First of all, what I would recommend, directly introduce tools which support you. Use directly Hugging Face, try to build models based on Hugging Face so that you have libraries or based on PyTorch Lightning, because they're giving things for free, I would always say. Then the other thing I would directly introduce version control system for data. Beginning, I would recommend, we are currently using KRYMA, I would now recommend DVC at the beginning and then try to find, especially for the machine learning part. Try to find one researcher and one software engineer and bring them together and let them communicate. Because then you get the efficiency from software engineers, especially cloud software engineers.

• 37:04Speaker 04

  And you get the research knowledge and bring them together. So they can learn from each other to exchange IDs, how to do this in the software manner and how to do research. So and this I would, when I would start again, I would now say, hey, here you have one team, two people and software guy, which DevOps background, also the software development and DevOps spectrum a little bit. And then bring a machine learning researcher to it. And then I think they would benefit the best.

• 37:35Daniel Whitenack?

  Yeah, and you talked a little bit about your journey in terms of the technical side and learning about these tools and also the kind of bringing on more people side. As you look forward to the next steps in the roadmap, I love it how you publish your roadmap on your site, which is really cool. As you look forward to the future of that roadmap, what do you feel like are the challenges that you're facing right now? As someone who is, does it have to do with, oh, now what do I do with all these generative AI stuff? And how does that factor into our product? Or does it have to do with, how do we make these models better and support a wider set of use cases?

• 38:20Daniel Whitenack?

  Or is it, is it a combination or something completely different?

• 38:24Speaker 04

  I would say more, it's not more about the technical challenge. What you could solve because technical challenge you can solve with, with knowledge and a little bit research. So normally, if it's not physically impossible, you can solve those things with a certain time. This is normally, no, no problem. The main problem is what I see as to speed up the development process itself. So that the right things are researched, the right things are designed and the right things are started to develop. So that you're bringing more the focus on one topic. And if you have a lot of people in your company or a lot of, I would say, interfaces,

• 39:06Speaker 04

  then you have to bring them one common understanding, what you want to achieve, how you want to work, how you want to define requirement. This is, I would say, currently the main challenge when I would say so. And then from the technical challenge, we have to talk to customers, get the feedback, build the stuff as quick as possible and iterate in terms of what the business wants to have.

• 39:31Chris Benson?

  So as we kind of wind up here, and this is pretty typical, we love to get kind of the benefit of your insight, not only for these short-term practicalities, but a little bit of the dreaming. And as you're, as you are kind of, you've come this far in this journey that you've described, and you now have this capability that didn't exist as an entrepreneur, as you're looking at the future, what are the ideas that may be speculative? It doesn't have to be based in the realities of what we have today. Where do you want to go with this? Like, what do you envision building, you know, over the next couple of years, you know, you can, you can pick the horizon, two years, five years, whatever you think. So when you lay in bed at night, you're like, that's the place I'm going eventually. What does that look like?

• 40:17Speaker 04

  Maybe to look a little bit back in the history, how all this project is started. I think I haven't thought this before. When I was also at Siemens, then I also do, done my master's thesis about visual question answering, which should solve the tasks what we are doing now in separate tasks as an internet task. So what my dream is, is to take the now available technology, large language models, including the visual part, the natural part, combine everything so that we can teach or bring every kind of data inside the model. What does this mean, every kind of data? So this means we get, for example, manuals for software, which says, hey, please click on this button. So then

• 41:01Speaker 04

  some person coming to you and say, hey, please create me account in this one. You give this a manual, and then it does completely automatically without any kinds of learning, because it's learned how to interact with the operating system. This is one thing where we want to go to bring nowadays, technology in, to make it accessible for the users, and also that everyone, really everyone can use it also your grand master.

• 41:32Daniel Whitenack?

  That's great. I am excited to see some of those things come down the line. And I think one of the things I've enjoyed about this conversation is that you brought a lot of the sort of positive side of automation that is really, really so helpful to technical people, but also other people that are doing these tasks that they really actually don't want to do or can't scale to a certain point, right? So I think that's really awesome. And yeah, I'm looking forward to seeing your, your future work with Ask UI. And thank you so much for joining the podcast. Really appreciate it, Dominic.

• 42:06Speaker 04

  Thanks for having me.

• 42:16Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.