#338 Amith Singhee: Can India Catch Up in AI? IBM's Amith Singhee on What It Will Take

What if the country that trains the world's engineers finally built the infrastructure to match its talent? In this episode of Eye on AI, Craig Smith sits down with Amith Singhee, Director of IBM Research India and CTO of IBM India and South Asia, to explore where India actually stands in the global AI race and what it will take to close the gap. Amith gives an honest, ground-level assessment of why India has been slow to compete. The talent has always been there. But until recently, the investment, the compute infrastructure, and the institutional intent hadn't come together in a sustained, coordinated way. That's changing, and Amith explains exactly what's different now. He walks through IBM Research India's 27-year presence in the country, the research it's doing on foundation models, hybrid cloud AI deployment, agentic systems, and quantum computing. He also explains why building AI from India doesn't just help India. Working with less data, less compute, and more linguistic diversity forces better engineering and makes IBM's models more generalizable for the entire world. We also get deep into the technical frontier. Why catastrophic forgetting is one of the key unsolved problems standing between current AI and anything more capable. How IBM is already shipping continual learning in practice through its COBOL modernization tools, helping enterprises decode decades of legacy code before the engineers who wrote it are gone. And why agentic AI, for all the hype, still has a mountain of unglamorous enterprise engineering left to climb before it becomes truly reliable. Plus, what Amith would tell an 18-year-old engineer in India today about what skills will actually matter in an AI-driven world. Subscribe for more conversations with the people shaping the future of AI and emerging technology. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Introduction and Amith Singhee's Background (06:26) Why IBM Set Up Research in India (11:45) Can India Compete in AI (15:18) How IBM Collaborates With Indian Universities (19:25) Why India Has Been Slow in AI (24:50) IBM's Hybrid Cloud AI Research Focus (27:34) How Data Scarcity in India Makes Better AI (31:18) Fine-Tuning Models Without Losing General Knowledge (35:03) Continual Learning and Catastrophic Forgetting (38:25) COBOL and Legacy Code Modernization (42:11) Agentic AI Hype vs Enterprise Reality (48:09) What Young Engineers Should Study Today

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  What's your view of why India has been slow? And then do you think it's possible for India to catch up?

• 0:07Speaker 01

  What I have seen is India has taken more time to bring in the investment and the intent at the level needed that some of the other countries have done. It is perfect storm of meaningful investment coming together with the talent and with infrastructure in a single entity. So, once all these three things come together, then it's going to move really fast. I don't know if we can become number one or number two in the next two years. But if we look at it from the lens of are we maximally capable of applying the latest AI tech for the benefit of India on our own? I think we can get there. Whether we can make the next three trillion dollar startups, that's a whole other question. I'm Amit Singh, I'm the Director of IBM Research in India.

• 0:49Speaker 01

  And I also play a CTO role for the business here in terms of sharing our technology vision with the ecosystem and engaging on that front. Been at IBM almost eighteen years now. And my background is in engineering. I studied electrical engineering at the Indian Institute of Technology in Karakpur, IIT Karakpur. After which I went to the US, did my masters at Carnegie Mellon. This was in a field called electronic design automation. So I switched a little bit into semiconductor and microelectronics area. Worked for a couple of years, then went back to my PhD back at CMU.

• 1:34Speaker 01

  And then joined IBM Research in Yorktown Heights.

• 1:38Craig S. Smith?

  Oh, really?

• 1:39Speaker 01

  Yes, I spent almost eight years there.

• 1:42Craig S. Smith?

  Beautiful building. Where did you live?

• 1:45Speaker 01

  I used to live in Yonkers.

• 1:47Craig S. Smith?

  Oh, yeah.

• 1:48Speaker 01

  Because my wife used to work in the city, so it was convenient. Yeah, and I started off working in semiconductors from the design automation lens, which was basically creating software tools for chip designers.

• 2:03Craig S. Smith?

  Right.

• 2:04Speaker 01

  So it was at the intersection of computer science, electrical engineering, math.

• 2:10Speaker 01

  And then it's been eighteen years in IBM, the last ten have been in India. Uh, I've moved around in terms of uh themes and areas over the years. Moved into more uh more AI and applications and smarter energy. This was in the twenty ten to twenty fifteen time range. Uh, that was the period of big data and analytics. And lots of utility companies were looking to use that because they're data rich. Uh, we did some great projects at that time internally and also with uh utilities like uh DT Energy in Michigan and PG&E in California, in trying to simulate the weather and its impact on the grid.

• 2:53Speaker 01

  Once I moved back to India, I was more focused on things where we could leverage collaborations in the market here. Uh, so we look more at I O T and retail and fashion and how deep learning can make a difference there. And then in around twenty nineteen, twenty twenty, when uh we had a new CEO, we acquired Red Hat and IBM really pivoted into this hybrid cloud strategy. Uh, within the research division also, we uh we saw an opportunity to double down and help the company with innovations. So I started an effort around hybrid cloud research uh out of the India team, working with our global research team.

• 3:35Speaker 01

  And looked at lots of interesting problems at the intersection of hybrid cloud and AI. And then I got into this role about three years ago as director and uh now we are heavily focused on generative AI in the lab here. Uh, not a surprise, but we look at all the layers of the stack and more from an enterprise lens uh rather than a consumer lens. Which means we look at the stack in terms of enabling hybrid cloud deployments. So you should be able to deploy the models and the AI workloads wherever you want. Uh, if you're a bank and you have a highly regulated uh enterprise, you might want to decide where the AI goes. Uh, so we look at optimizing the software stack for AI, we.

• 4:20Speaker 01

  Also work on training and developing our granite model series, which is in the open. And we have a lot of work also on building agentic middleware, which allows enterprises to build agentic solutions. Uh, and also in the actual solutions themselves, specifically in areas like uh, software development, uh, IBM has this what's next code assistant series of products. Uh, we do a lot of the AI research that goes into those. Uh, also in the IT automation area, business automation, data. How do you use AI to better manage enterprise data? And we have some very interesting angles to the AI work we do, for example, in the time series domain.

• 5:04Speaker 01

  And geospatial domain, satellite imagery, where uh, uh, can you have instead of a language model, a model that understands satellite images? So good spread of work. Uh, we also have a growing team in quantum computing. Especially at the software layer and we are very engaged with the ecosystem here also. That's one of the unique things about the uh, research team in IBM in India is. We've been here twenty seven years. We started off within ID Delhi. Uh, we have very strong ties to many academic institutions. They see us as collaborators because of our research ethos.

• 5:44Speaker 01

  And uh, we're also very engaged with other parts of the ecosystem, whether it's the government, startups, companies, developers, from again, from an innovation roadmap.

• 5:57Craig S. Smith?

  Yeah, uh, I'm, as I said, I'm interested in uh, where India is generally in the AI uh, journey. Uh, and I had a long conversation with Amish Singh in Delhi about.

• 6:14Craig S. Smith?

  Why the government was late in in getting on board or in forming a vision. But they are very focused now and there's a lot of ambition.

• 6:29Craig S. Smith?

  Why do companies like IBM, why did IBM set up in India, as I said, is it a market strategy? Or is it a human resources strategy? Because there are a lot of talented engineers here. Or is it built around some specific research that was developed in India? And for which India is is has the expertise? How do you see it?

• 7:01Speaker 01

  So we IBM came to India well before the AI revolution, right? So e- even the research teams been here for twenty seven years and the company's been here longer. So the the beginnings were around being a global company, seeing the talent potential of a growing economy in the late nineties.

• 7:23Speaker 01

  It made sense to have a presence here. It's worked out well because that potential is realizing. And definitely the talent that we get access to here adds to talent we have elsewhere. So it's a very valid pool of deep tech talent with global mindset that helps us as a global team. Just us in IBM Research, the projects we do are not isolated by geography. There's part of a global strategy. And so having that combination of hard skills and soft skills.

• 8:02Speaker 01

  Available here does help our global mission. It also helps us engage with the market and understand variations of the problems that are relevant. Let's say in a developing country like India.

• 8:15Speaker 01

  So then the technology we develop is more generalized and more applicable. And then when it when it lands up in the product side, then more relevant. And it's not just India, India represents a lot of the global south. So when we look at the problems from this lens and bring it into our mainstream work, it makes the solutions relevant. A big many other countries.

• 8:38Speaker 01

  Uh. So it's a mix of both, I'd say, right? The talent and the understanding of the market. Uh, but talent is surely a big driver.

• 8:46Craig S. Smith?

  Yep. And on the talent, uh, why did you come back? Because again, I had this long conversation with Milesum, Professor Milesum, uh, about the you, about the difficulty of getting people to come back after their PhDs. Because there are so many opportunities in the United States. So, yeah.

• 9:12Speaker 01

  Uh, for me, it was simple and personal. I actually spent fifteen years in the US and I loved it. I could have stayed there forever, but I really wanted to be back close to family.

• 9:23Craig S. Smith?

  Sure.

• 9:23Speaker 01

  Uh, and I felt, okay, the time in life with my, with our family and everything was just right. And we moved back mainly for that reason.

• 9:31Craig S. Smith?

  Yeah, uh, as I said, I spent a lot of my life in China and Milesum talked about this, but also I lived through it in China. There was a moment when China's economy had reached a level uh, that they had tremendous foreign reserves and they very deliberately started offering competitive salaries to bring uh, expertise, research expertise back, uh, and but it wasn't that alone. It was the sudden transformation to generative AI uh, that allowed them to catch up because they had brought back these people.

• 10:16Speaker 01

  Hmm.

• 10:16Craig S. Smith?

  Uh, but they were far behind. But once generative AI, the transformer algorithm hit, uh, they suddenly didn't leapfrog the whole, uh.

• 10:31Craig S. Smith?

  Supervised learning optimization phase. Uh, and now they're uh competitors. So it looks to me that India has the opportunity to do the same thing. Do you, are are you guys able to attract talent from the Indian talent? Indian national talent back from the United States?

• 10:54Speaker 01

  So, speaking from the lens of IBM research, uh, we actually find our talent in the local market. Mostly. And we do have people like me who move back to the lab and join IBM research. But mostly it is because they're moving back for their own reasons.

• 11:12Craig S. Smith?

  Yeah.

• 11:12Speaker 01

  Rather than us convincing them. So, but this is from the lens of our lab and we have eminence in the country in the research space. So if there is good talent, there's a high chance that they would want to go to a research lab like ours. Uh, it's definitely not the case at large that there's a lot of local talent at that level. Uh, available at scale in the way it would be, say, in the U S. And it is something that does need uh more effort.

• 11:43Craig S. Smith?

  Yeah. Uh, uh, again with, uh, uh, and are you hiring from the I I Ts?

• 11:49Speaker 01

  We hire from the I I Ts, we also hire from some of the other colleges. Uh, we hire PhDs, we hire masters, we also hire undergrads.

• 11:58Craig S. Smith?

  Yeah.

• 11:58Speaker 01

  Strong undergrads.

• 11:59Craig S. Smith?

  And does IBM have a policy of, actually, I don't know if Partha is still teaching. Do you?

• 12:06Speaker 01

  I don't know.

• 12:07Craig S. Smith?

  Yeah. Uh, but he's now full time at DeepMind. If you hire somebody from a PhD student from the I I Ts, are, can they, or a professor? Are, uh, can they continue to teach at the I I Ts? Or, uh, do you, you know, as they do in the U S, a lot of people have dual roles?

• 12:28Speaker 01

  So here, India does have this notion of professor of practice. And in some cases, uh, if somebody is really passionate to do that. Hmm, there are mechanisms to put that in place. Uh, but what we see is a lot of people just want to, if they are coming into an industrial research lab, they want to be successful there.

• 12:51Craig S. Smith?

  Yeah.

• 12:52Speaker 01

  Make a difference and that tends to take up a lot of your time and energy.

• 12:55Craig S. Smith?

  Yeah.

• 12:56Speaker 01

  But yeah, there's a professor of practice mechanism that's available in India.

• 13:00Craig S. Smith?

  Yeah, and the the the resources, you were saying that you collaborate a lot with the I A Ts.

• 13:09Craig S. Smith?

  One of the reasons in the U S that people go into industries because of the resources, the compute and data resources that aren't available to academic institutions. Is that the same here? That's one of the drivers that pulls people toward industry?

• 13:27Speaker 01

  It is, especially in this generative A I era.

• 13:31Craig S. Smith?

  Okay.

• 13:33Speaker 01

  It's not, yeah, there's not a huge amount of resources available to academics in general. Some of them would have gotten grants, but again, that's very specific cases. Whereas in industrial settings, sometimes they can get more access. So that it depends what drives the the faculty. If they, a lot of them do great work, which doesn't need a lot of G P U's. Because they're doing theoretical things or they're trying to figure out nifty ways of improving something. But if somebody really wants to do at scale experiments, then it's not that easy to do.

• 14:12Craig S. Smith?

  Yeah. In in the collaborations that you have with academic institutions in India, how does that work? You you provide compute or you?

• 14:25Speaker 01

  Ah, so the way it works is, there's a couple of different ways. We have, we have long term part research collaborations contract with, let's say I D Bombay, I T Delhi, I I S C. Where there would be four or five different projects running at a given point of time with co P I's. A faculty member and one of our researchers or two of our researchers would be co P I's. Directing the work. So the student gets the benefit of industrial mentoring and practical problems and faculty groups, they benefit from steering research in areas where industry

• 15:09Speaker 01

  is focused on and creates a job pipeline.

• 15:11Craig S. Smith?

  Yeah.

• 15:12Speaker 01

  Some of these deep skills. And these would be funded programs by IBM.

• 15:19Speaker 01

  The other way also we work is in the open. There's this thing called the AI Alliance that was started about two years ago. It's a community of organizations globally that have said, okay, we want to do some work in the open to build AI, to create data sets, to set up responsible practices for AI. IBM is very active in that. And in India, we've had some academic institutions like I T Bombay, I T Madras, I T Chotbor, some startups, AI and some nonprofits and some large system integrators become members. And so there, these members might find common

• 16:03Speaker 01

  interests and they collaborate in the open. Each one gives something. Somebody gives people, somebody gives a few GPUs or data sets and we work on things.

• 16:14Craig S. Smith?

  And that's a global alliance, yeah.

• 16:16Speaker 01

  That's a global alliance, yes.

• 16:17Craig S. Smith?

  Yeah, that was another question. So you were saying that research is not geographic at IBM. That right, it's so you may be working on hybrid cloud solutions and we can talk about the research, but you would be working with a team in the US or with a team in France or wherever.

• 16:46Speaker 01

  Yeah.

• 16:48Craig S. Smith?

  Does, do you, right now, there's so much balkanization of the world politically, unfortunately.

• 16:59Craig S. Smith?

  Are you able, does that interfere at all with like working with IBM in China or working with IBM in, I don't know if IBM has a big research presence in Russia, but I'm guessing they might.

• 17:15Speaker 01

  We don't have research presence in China or Russia.

• 17:18Craig S. Smith?

  Okay.

• 17:19Speaker 01

  Uh, so we have research presence in Japan, in India, in Israel, in the UK, in Switzerland, in the US and in parts of Africa. Uh, so of course, we have to be very clear and careful about all of our export compliance requirements.

• 17:39Craig S. Smith?

  Yeah.

• 17:39Speaker 01

  And we are all aware of that and we keep up to date with it. But given our footprint, we don't really run in the footprint that I spoke to you about.

• 17:49Craig S. Smith?

  Right.

• 17:49Speaker 01

  We don't run into a lot of issues typically wherever there's specialized technology uh that uh even within this group needs to be contained, we be careful about that.

• 18:00Craig S. Smith?

  Yeah.

• 18:00Speaker 01

  But that's limited.

• 18:02Craig S. Smith?

  Yeah. And how do you feel about India on its, as I said at the beginning, what interested me about India is looking at it from China uh and not understanding why India is not a a player in in at least yet in the AI competition. I know researchers don't like to look at it as competition, but the political leaders certainly do. Uh, do you, what's your view of why India has been slow? And then do you think it's possible for India to catch up as the A India AI missions

• 18:47Craig S. Smith?

  uh ambition is?

• 18:51Speaker 01

  Yeah, so I I don't know why uh things have happened as they happened because it's up to decision makers in the government and what they've done. But what I have seen is India's taken more time to bring in the investment and the intent at the level needed uh that some of the other countries have done. And perhaps that was, that's politics, I don't know what it is, right? But definitely since uh last year, I would say, uh, it's.

• 19:25Speaker 01

  Uh grown up several notches and I'm very optimistic being engaged with many of these stakeholders and the intention that's there and the talent that's there. I think we have some ingredients. If we can use those right, we can be quite relevant. I don't know if we can become number one or number two in the next two years, but if we look at it from the lens of are we maximally capable of applying the latest AI tech for the benefit of India on our own, I think we can get there.

• 20:01Craig S. Smith?

  Yeah.

• 20:03Speaker 01

  Whether we can make the next three trillion dollar startups, that's a whole other question.

• 20:08Craig S. Smith?

  Yeah, as a matter of fact, I'm hoping to speak to a startup person in Chennai. Because that has been an another uh issue is the startup ecosystem and all the barriers to a startup. The level of investment, I think it's a

• 20:30Craig S. Smith?

  a bird of one point two or one point six billion. I can't remember the exact number. Uh and a lot of that's going into uh data centers and India is I think still only one gigawatt in uh in uh

• 20:54Craig S. Smith?

  consolidated gas data center uh power. Uh what do you think's the critical ingredient? Because India has with the I E Ts and the

• 21:05Speaker 01

  Yeah.

• 21:06Craig S. Smith?

  and the J E E it's you've got the top talent coming out of the I E Ts.

• 21:12Speaker 01

  Yeah.

• 21:13Craig S. Smith?

  Uh and there's also an English language facility which uh I would think helps. Uh so wh- wh- what do you see as the biggest uh?

• 21:24Speaker 01

  I I think it is the perfect storm of meaningful investment coming together with the talent and with infrastructure in a single entity, whether it's a startup or an academic effort or whatever it is. And that combination hadn't happened until last year for sure. Now this year with India AI mission and some of the startups that are getting the major funding.

• 21:51Speaker 01

  I think it will happen, but the but the infrastructure is still in a ramp up stage. The money is there, the people are there, modular money getting released and all of that, but it's been committed. But I think our data centers have been building up. I don't know all the details, but what I understand is getting all the GPUs alive and in a single data center. So that you have four thousand GPUs or whatever you need in a single place available for six months. It's getting there. So once all these three things come together, then it's going to move really fast. Because I know that the skills and the knowledge of what needs to be done in terms of how do you train these models? How do you prepare?

• 22:36Speaker 01

  Those skills are there. These startups, our folks, people have figured it out. So they don't need to rediscover that. It just that those three things have to come together in a sustained way. So that people can do.

• 22:51Craig S. Smith?

  And and by infrastructure, you you you mean primarily data centers?

• 22:55Speaker 01

  Yes.

• 22:55Craig S. Smith?

  Yeah, yeah. Okay, so the research that you're doing here, it's on hybrid cloud, is that the main focus?

• 23:04Speaker 01

  The main focus is AI, but enabling AI with a hybrid cloud architecture. Okay, so what that means is if I didn't have a hybrid cloud mindset, then I might just develop like a Gemini that runs on my cloud and I'm done. But if I have a hybrid cloud mindset, then I might say, look, I don't want to develop one Gemini on one place. I want to enable clients to use any model they want, anywhere they want. And I want to develop models that are portable. I want to develop software technology that runs those models, uses those models in a way so that it's portable.

• 23:46Speaker 01

  So it's still AI research, but it's architected and prioritized so that it can be deployed anywhere, on any cloud, on-premise data centers, on our power systems, on these mainframe.

• 24:00Speaker 01

  Computers and things like that. So uh that's the lens we take. But the core research we do is, again, it's L L M's and image models and agentic systems. So it's all the same fundamental technology.

• 24:16Craig S. Smith?

  And the L L M's are that you're working on here, are they uh English uh local language specific and do they cover a range of languages?

• 24:29Speaker 01

  So, no, we, again, the L L M work we do for our granite models, it's a global effort.

• 24:34Craig S. Smith?

  Right.

• 24:35Speaker 01

  So it's just one version of the model that anybody can use. And until now, uh we have supported mostly uh Indian languages have not been primarily supported, but our team here has been working to add better support for Indian languages into the same model series. So the next version that's going to come out, we expect it to be a lot more capable. The other thing we are also doing is as part of our collaborations in the A I Alliance and other areas where somebody else might be developing India specific models. We work with them to show how those models could be made hybrid cloud compliant, or how could you scale them better?

• 25:18Speaker 01

  Because we don't see I B M really building India specific models or Germany specific models. It would be one common model does reasonable everywhere. But it's likely someone will come up with a model that's really tuned for that market. And so we also want to work with them to allow them to scale and deploy their models wherever clients might want.

• 25:40Craig S. Smith?

  Yeah, and one thing you said earlier that that I thought was interesting is that by by working in India on these global efforts,

• 25:54Craig S. Smith?

  it helps generalize

• 25:55Speaker 01

  Yeah.

• 25:56Craig S. Smith?

  models. Is that because of the data that you're using from India?

• 26:03Speaker 01

  So I'll give you a couple of examples. If you are only looking at data that's in the West.

• 26:11Craig S. Smith?

  Yeah.

• 26:11Speaker 01

  You'll have a lot of data on the internet. But if you say, okay, let me look at digitized data in India, that's way less, right? It doesn't reach the levels. Then you're forced to think, how do I train models when I have less data and still get high quality? That helps you generalize your approach. So you're not training models with the assumption that I'll always have a huge amount of data. And that's useful because when we go to a bank or an enterprise and they say, look, I want to customize this model for my purposes, but I only have X amount of data. Can you do it? So that generalized approach of being able to work with smaller amounts of data can be useful in lots of these settings.

• 26:56Speaker 01

  That's one example. Another example is in these transformers, there's this notion of tokenization, where you break up the words into little tokens, like syllables. Now, which tokens you use can get biased based on which languages you've looked at. If you've not looked at Hindi and Indian languages carefully, then when you try and apply the models in India, those syllables are not going to be optimal and the model might not understand or interpret that language right. But if you have done that, then you can generalize your tokenization approach. These are some of the things that happen on the model.

• 27:35Craig S. Smith?

  Yeah, that's interesting. And this is something, a general question that I've never quite understood.

• 27:43Craig S. Smith?

  When you say you're working on the Granite series, so that's, those are open weight

• 27:54Craig S. Smith?

  models available to anybody. The foundational training is, what you're saying is that you contribute to that foundational training data so that the model will generalize across.

• 28:14Craig S. Smith?

  Cultures and continents.

• 28:16Speaker 01

  Yeah.

• 28:17Craig S. Smith?

  Uh, but when somebody of a bank wants to do something specific with that model, there you're, the model is already trained. You're talking about.

• 28:27Speaker 01

  Fine tuning, extended training. Yeah, there's different techniques, but generally you can, we just call it customization.

• 28:35Craig S. Smith?

  Right.

• 28:35Speaker 01

  So you make the model even better uh for that business.

• 28:40Craig S. Smith?

  Yeah. And how much data is, obviously it depends what you're trying to optimize for, but you need much less data to do that. Uh.

• 28:53Craig S. Smith?

  It is, when you were talking about helping uh customize, uh if, if a customer has only a little bit of data, how, what do you do? And what, where is the research leading?

• 29:09Speaker 01

  Yeah, so there are things like, there's a couple of tricks, right? For example, there's data mixing strategies. So if you have X amount of data, right, uh which is small, and I take a nice granite model and I tune it on that. What could happen is it might get really good on the tests you run.

• 29:32Craig S. Smith?

  Yeah.

• 29:33Speaker 01

  But it'll really lose all of its general capabilities that it had. It overfits.

• 29:38Craig S. Smith?

  Yeah.

• 29:39Speaker 01

  Uh, so it forgets things. Now that might become a problem when you deploy it out there in the field because you only test on a hundred examples, but your consumers are going to throw all kinds of questions and things at it, relevant to your business, but it does need that general knowledge and general skill there too. So you need to then figure out how do you maintain that general skill while infusing your business specific skills in this tuned model. And there's tricks around how do you mix the old data with the new data and what proportion do you use? How do you feed it to the model? There's this notion of curriculum training, like how you might train someone who joined New York Times.

• 30:19Craig S. Smith?

  Yeah.

• 30:19Speaker 01

  You might ease them into it. Uh. So all of these recipes, we call them recipes, have to be tuned to the data scenario you have in in that client's environment. There's also other tricks around synthetic data generation. Uh. So you might say, okay, if you only got X data, I will have some clever tools that will understand the distribution of that data and synthesize a lot of data that looks like that. It's not supernatural, but it's good enough so that if the model sees the realistic data and sees some synthetic versions also, that's better than just showing it that little bit of it. So there's all these innovations around data.

• 31:03Craig S. Smith?

  Yeah.

• 31:04Speaker 01

  That come in.

• 31:05Craig S. Smith?

  And the data mixing, are you pulling data out of the original training data and mixing it with the fine tuning?

• 31:12Speaker 01

  If you have access to it, yes. So if it's us doing that for the client and it's the granite model, then of course we have the data.

• 31:20Craig S. Smith?

  Yeah.

• 31:21Speaker 01

  But if it's somebody else trying to do it, they may not have access to the old data. Then there's other tricks you have to try. Where you try to make the model generate old data that it might remember and then use that back. Uh. So that's.

• 31:36Craig S. Smith?

  Yeah, that's fascinating because uh this is a little bit off tangent. The big blocker in my mind to uh right now, we seem to have reached a plateau in generate the AI models or there's some incremental improvement. Uh.

• 31:59Craig S. Smith?

  It feels like where we were with uh supervised learning, where the research community uh focused on optimization and getting smaller and smaller uh improvements or refinements, which are important for productization, but in terms of getting to ASI, uh it's not gonna happen. So.

• 32:23Craig S. Smith?

  To me, the big blockers are catastrophic forgetting and continual learning.

• 32:30Speaker 01

  Yeah.

• 32:30Craig S. Smith?

  Lifelong learning, whatever you call it. Uh, i- is, does this research, what you're talking about, uh, it sounds like that might be relevant for solving.

• 32:41Speaker 01

  Yes, I wasn't using those terms, but it's exactly what you said, catastrophic forgetting and continual. Actually catastrophic forgetting is no longer catastrophic. It's just forgetting because uh, we've gotten past those early days where we- researchers would just do like bad data mixes and it'll just forget everything. But now it's more like, oh, it used to be, it used to get a hundred marks on this kind of a task. Now it's getting eighty, uh, it doesn't go down to zero, but we still want to keep it at ninety plus. Because we know that's a useful skill in the wild world out there. And you don't want to forget that, even that little bit.

• 33:22Speaker 01

  Uh, and continual learning is another important one because uh, model versions change, new data comes along, uh, you want to add new use cases. As a bank, you say, okay, initially I thought I'll train this, but then a year later you've got sort of new things.

• 33:39Craig S. Smith?

  Yeah.

• 33:40Speaker 01

  And you don't want to go back to the scratch pad and do everything from the beginning. So these are all very valid problem statements that come up when we put this hybrid cloud and enterprise lens. We're going to give control to the client, uh, rather than keeping all of the control with the model provider.

• 34:03Craig S. Smith?

  Does, in lifelong or continual learning, if you can solve catastrophic forgetting, does the model, do the parameters need to uh, expand with, as the data is added?

• 34:22Speaker 01

  So we try not to, uh, it's typically the model, as of now, the approaches that are being.

• 34:30Speaker 01

  Pushed and improved, keep the same model architectures, the same parameters, same everything. It's just that you figure out new values for those parameters. Either through a extended training process or some reinforcement learning process. Or using these things called LoRa adapters, low rank adapters. But those low rank adapters, while they are additional parameters with this overlay on to the same parameters at the end of the day. So, yeah, as of now, there's not, I am not aware of much that changes the model.

• 35:06Craig S. Smith?

  Yeah. Are you hopeful that the catastrophic forgetting will be solved to a degree that continual learning can become a reality?

• 35:17Speaker 01

  I think it can become a reality, yes, but I don't think it's going to be perfect. But I think it's going to become useful.

• 35:27Craig S. Smith?

  Yeah.

• 35:27Speaker 01

  Yeah, it'll be good enough to be useful. In fact, when we work on our products, like the Watson X, we have this code assistant for COBOL, like legacy languages, which actually a lot of clients use. A lot of credit card transactions, bank transactions are actually handled on the mainframe.

• 35:48Craig S. Smith?

  Yeah.

• 35:48Speaker 01

  And those are all running on code written in COBOL, Assembler, RHEL and these things. So none of the models that you find in the public domain do well on these because they haven't seen that kind of data. So we take our granite models and then we customize them with our in-house expertise in IBM

• 36:09Speaker 01

  to become better on understanding COBOL, writing COBOL, converting COBOL to Java. And we release an updated version every four to six weeks, which means we've had to figure out continual learning and some aspects of continual learning for that. So it's already in practice, but I think solving it more generally so that you don't need a, you don't need PhDs, you just need a button and it should just work. I think we're not there yet.

• 36:40Craig S. Smith?

  Yeah, yeah. And that's interesting. So that's because there is this massive installed base of COBOL.

• 36:50Speaker 01

  Yes.

• 36:51Craig S. Smith?

  And there have been a lot of.

• 36:55Craig S. Smith?

  There's cottage industry in translating or modernizing COBOL. But because to find an engineer that understands the language is getting harder and harder.

• 37:11Speaker 01

  Yeah.

• 37:12Craig S. Smith?

  So, so these models are taking over that role. They can understand the language.

• 37:17Speaker 01

  Yes.

• 37:17Craig S. Smith?

  They don't, they're not necessarily translating it into a more modern language, but they allow engineers to work with.

• 37:30Speaker 01

  It's all of those. So you can use it to explain some COBOL that's looking cryptic to you. You can say, hey, explain this to me in English. What does it do?

• 37:39Craig S. Smith?

  Okay.

• 37:39Speaker 01

  You can use it to write new COBOL. Maybe you've got like a complex tax calculation code and you're like, oh, the tax rate changed. Where should I go change, make the change? So the model will try to guide you. You can use it also to convert some COBOL code to something like Java, because it understands COBOL and Java. And so it, it would read the COBOL and say, hey, this is a Java that you could use. Still have to double check it, but it can actually translate COBOL to some more modern languages. The new thing that we are trying to do on is not just the model, but the system, so that it can give, bring in business context.

• 38:23Speaker 01

  Because this COBOL code that clients have, it actually implements their business process. How is that line of state tax calculated on my mobile bill? Behind that, there's some business rules. That's what's coded up. And if you don't know COBOL, and if you didn't know forty years ago, why was this written in this way? That knowledge of those business rules is getting lost or it's locked in, I would say, right?

• 38:54Speaker 01

  So that's a huge risk for our clients because if let's say ten years from now, they don't have anybody who understands why that tax calculation was done that way, what is actually happening. It's hard for them to maintain it, or there's a new regulation that requires a change, it's a huge risk. So bring interpreting that entire code base and creating a business description of that is one very interesting problem we're working on.

• 39:21Craig S. Smith?

  Yeah, and you mentioned a middleware for agents. Is that an orchestration layer to to manage multiple agents or the coordination or cooperation of agents?

• 39:34Speaker 01

  That is that, that's the most obvious one. Uh, there's a product we have called Watson X orchestrate that does that. But then, that's not all. What happens is, let's say you you have you use a powerful L L M as an agent where you say, I have booked flight tickets for me or

• 39:53Speaker 01

  promote somebody, whatever. Now nowadays, these agents, they first make a plan dynamically. To do this, I got to do this, and then that will call that tool, call that A P I, read this file, and then I'll do that.

• 40:08Speaker 01

  And then based on that, it'll take actions, it'll call some A P I, it'll generate the code for calling an A P I, or it'll open up a file on the disk and read its contents and then decide what to do. So these things are called tools.

• 40:22Craig S. Smith?

  Yep.

• 40:22Speaker 01

  For the L L M, and if you don't provide good tool descriptions to the L L M, it may not know what it actually does. Not so different compared to humans. If you put a new employee into a complex business process, but you don't give them very good standard operating procedure documents, and you just give high level descriptions, they're gonna make mistakes and do something. Uh, so that's one example of now, so now you're shifting the.

• 40:55Speaker 01

  Who's going to write those descriptions? Now, again, you need humans, right? So you can create tools that actually generate good descriptions or that they test. Are these tools actually doing what the description says it does? Or did you miss something? So there's a lot of, that's one example. Another is security. If you've got an agent that is performing some business task in the SAP system or something on your behalf, you need the same identity authorization, auditability and all of that you would need if you were doing it. But now you're offloading it to this AI system and there might be thousands of users using the same system.

• 41:39Speaker 01

  So you need security mechanisms to track that this request came from person X, their enterprise authorization is Y, and so I need to like that particular call needs to. So there's all of this machinery you need, which is not just the orchestration. So we, as in IBM Research, not just the India team, we are working on a lot of these things so that then we have more mature capabilities.

• 42:06Craig S. Smith?

  And are you seeing there? Mike, I spoke to Kumar, I think her name was at Microsoft about, she was talking about society, societies of agents that will be working together invisibly, doing a lot of the routine stuff has to happen in business and society. But there's been very slow uptake by the enterprise.

• 42:41Craig S. Smith?

  And largely it's a trust issue, not just security, but.

• 42:48Craig S. Smith?

  And I've played around with agents and you set it off to do something and you go away and come back and realize it's been waiting for forty five minutes for you to say yes. Is there?

• 43:05Craig S. Smith?

  Do you? As IBM, of course, you believe in the promise of agents, but it just seems that.

• 43:16Craig S. Smith?

  There's a lot of hype about agents, but practical use is lagging and it's lagging because of these reliability issues.

• 43:28Speaker 01

  Yeah, I think the hype is moving faster than it should.

• 43:33Craig S. Smith?

  As it always does.

• 43:34Speaker 01

  Always does. So, uh, I think, again, for enterprises, uh, they want to be sure it's going to work in the way things used to work when I didn't have the agent in there. And just because I can do things with chat G P T using flight tickets doesn't mean that this thing is going to integrate right with my S A P systems and with my workday and with, and it'll have the right user experience because ultimately some end user is going to work with it and it needs to fit into that business process. Uh, so all of these engineering act, things are still left to be done.

• 44:17Craig S. Smith?

  Yeah.

• 44:17Speaker 01

  I'm not saying these are rocket science, but they, they need time and effort. They need a lot of design and iteration to figure out. And alongside that, there's also this thing that not everybody is going to integrate a public frontier model into their internal business process, right? And so when you have an on-prem deployment of some model, getting it to do what Open A I figured out is

• 44:40Craig S. Smith?

  Yeah.

• 44:41Speaker 01

  Um, not trivial. So that's the gap, this sort of engineering and sort of enterprisification, uh, that will take a bit of time.

• 44:51Craig S. Smith?

  Yeah.

• 44:52Speaker 01

  It will happen, but it'll just have to play out.

• 44:55Craig S. Smith?

  For young engineers in India, uh, there's very high unemployment among I A T grads. And I get asked this question, I'm a journalist, I have no idea, but people ask me all the time, what should I study? It, what would be the, uh? What would be your answer? What should an eighteen year old study today? Uh, to be part of the AI future that we're looking at?

• 45:24Speaker 01

  Yeah, so if it's an AI centric future, because they should study things that are not AI also, because there's still a mechanical engineering.

• 45:33Craig S. Smith?

  Yeah.

• 45:34Speaker 01

  All those fields.

• 45:35Craig S. Smith?

  Although AI will take care of a lot of that.

• 45:38Speaker 01

  Yeah, but you do need the domain expertise.

• 45:40Craig S. Smith?

  Yeah.

• 45:41Speaker 01

  You, if you don't understand mechanical engineering, even if you're using AI, you will not be useful to an automotive company. So I would say, uh, two things are important, not understanding the fundamentals is still important, even if you're going to use AI in the future. Because people hiring you will always hire the person who understands the fundamentals and can use AI versus someone who doesn't and can use AI.

• 46:10Craig S. Smith?

  Yeah.

• 46:11Speaker 01

  That's important, that'll empower you. And the other is, I think, uh, we're beyond, we're getting beyond this area of, I've learned a skill, I'm good in that, and I'll have a career in it.

• 46:21Craig S. Smith?

  Yeah.

• 46:21Speaker 01

  That's gone, right? So uh, the more, uh, diverse experiences you have, so that you condition yourself to continuous learning. Also, humans need to have continual learning and application of that learning very quickly. Uh, that's a skill that is important. I think it's not just on the students. Our academic institutions have to create that.

• 46:45Craig S. Smith?

  Yeah.

• 46:46Speaker 01

  Environment. That's the combination, uh, at this point.

• 46:50Craig S. Smith?

  Yeah.

• 46:50Speaker 01

  At least for the next few years.

• 46:52Craig S. Smith?

  Okay.