Representation Engineering (Activation Hacking)

Recently, we briefly mentioned the concept of “Activation Hacking” in the episode with Karan from Nous Research. In this fully connected episode, Chris and Daniel dive into the details of this model control mechanism, also called “representation engineering”. Of course, they also take time to discuss the new Sora model from OpenAI. Sponsors: Neo4j https://www.neo4j.com/developer?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – Is your code getting dragged down by JOINs and long query times? The problem might be your database…Try simplifying the complex with graphs. Stop asking relational databases to do more than they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real-time analytics, and genAI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it’s easy to integrate into your tech stack. Visit Neo4j.com/developer https://www.neo4j.com/developer?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai to get started. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Data synthesis for SOTA LLMs with Karan Malhotra from Nous Research (Practical AI #255) https://changelog.com/practicalai/255Article: Representation Engineering Mistral-7B an Acid Trip https://vgel.me/posts/representation-engineering/OpenAI Sora https://openai.com/soraUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. In this fully-connected episode, Chris and I will keep you fully connected with everything that's happening in the AI world. We'll take some time to explore some of the recent AI news and technical achievements, and we'll take a few moments to share some learning resources as well to help you level up your AI game. I'm Daniel Whitenack. I am founder and CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 1:17Chris Benson?

  Doing great today, Daniel. Got lots of news that's come out this week in the AI space. Barely time to talk about amazing new things before stuff comes out.

• 1:27Daniel Whitenack?

  Yeah, I've been traveling for the past five days or something. I've sort of lost track of time, but it's like stuff was happening during that time in the news, especially the Sora stuff and all that. And I feel like I just kind of missed a couple news cycles, so it'll be good to catch up on a few things. But one of the reasons I was traveling was I was at the Treehacks Hackathon out at Stanford. So I went there as part of the kind of Intel entourage and had Prediction Guard available for all the hackers there, and that was a lot of fun. And it was incredible. It's been a while since I've been to any Hackathon, at

• 2:10Daniel Whitenack?

  In-person hackathon and they had like five floors in this huge, you know, engineering building of room for all the hackers. I think there was like 1,600 people there participating from all over. Yeah. And really cool. Of course, like there were some major categories of interest. One, you know, like in doing hardware things with robots and other stuff. But of course, one of the main areas of interest was AI, which was interesting to see. And in our, the track that I was a judge and mentor in, one of the cool projects that won that track

• 2:51Daniel Whitenack?

  was called Meshworks. So what they did, and this was all news to me. While some of this I learned from, you know, the brilliant students, but they said they were doing something with LoRa. And I was like, oh, LoRa, that's the fine tuning methodology for large language models. I was like, yeah, that figures. Like people are probably using LoRa. But I didn't realize, and then they came up to the table and they had these like little devices, like hardware devices. Then it clicked that something else is going on. And they explained to me they were using LoRa, which stands for long range. It's a, these sets of radio devices

• 3:31Daniel Whitenack?

  that communicate on these unregulated frequency bands and can communicate in a mesh network. So like you put out these devices, right? And they communicate in a mesh network and can communicate over long distances for very, very low power. And so they created a project that was disaster relief focused, where you would drop these in the field and there was a kind of command and control central zone. And they would communicate back, transcribe the audio commands from the people in the field. And would say, you know, oh, I've got a, you know, I've got a injury out here. It's a broken leg. I need, you know, help, whatever, or meds over here.

• 4:16Daniel Whitenack?

  This is going on over here. And then they had an LLM at the command and control center, parsing that text that was transcribed and actually creating. Like tagging certain keywords or events or actions and creating this nice command control interface, which was awesome. They even had like mapping stuff going on with computer vision, trying to detect where like a flood zone was, or there was damage in in satellite images. So it was just really awesome. So that all of that over, you know, a couple day period. It was incredible.

• 4:51Chris Benson?

  That sounds really cool. And they did they start the whole thing there at the beginning of the hackathon?

• 4:57Daniel Whitenack?

  Yeah, they got less sleep than I did, although although I have to say I didn't get that much sleep, you know, that it wasn't a normal weekend, let's say.

• 5:06Chris Benson?

  You can sack out on the on the plane rides after that. It sounds really cool.

• 5:09Daniel Whitenack?

  Yeah. And there were like, it was the first time I had seen one of those Boston Dynamics dogs in person. That was kind of fun. And they had other things like these faces you could talk to. I think the company was called like Wehead or something. It was like these little faces. All sorts of interesting stuff that I learned about. So I'm sure there'll be blog posts. And I think some of the projects are posted on dev post, the site dev post. So if people want to check it out, I'd highly recommend scrolling through some really incredible stuff that people are doing.

• 5:42Chris Benson?

  Fantastic. I'll definitely do that.

• 5:52Speaker 03

  What's up, friends? Is your code getting dragged down by joins and long query times? The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real time analytics, and generative AI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it's easy to integrate into your tech stack. People are solving some of the world's biggest problems with graphs. And now it's

• 6:32Speaker 03

  your turn. Visit neo4j.com/developer to get started. Again, neo4j.com/developer. That's N E O four J dot com slash developer.

• 6:57Daniel Whitenack?

  Chris, one of the things that I love about these Fully Connected episodes is that we get a chance to kind of slow down and dive into sometimes technical topics, sometimes not technical topics. But I was really intrigued. Do you remember the conversation recently we had with Karan from News Research?

• 7:19Chris Benson?

  Absolutely.

• 7:20Daniel Whitenack?

  That was a great episode. People can pause this and go back and listen to it if they want. You know, I asked a lot of selfish questions. I learned a lot from him. But at some point during the conversation, he mentioned activation hacking. And he said, hey, like, one of the cool things that like we're doing in this, you know, distributed research group and playing around with generative models is activation hacking. And we didn't have time in the episode to talk about that. And actually, in the episode, I was like, I'm just totally ignorant of what this means. And so I thought, yeah, I should go check up on this and see if I can find any

• 8:02Daniel Whitenack?

  any interesting posts about it and learn a little bit about it. And I did find an interesting post. It's called "Representation Engineering: Mistral 7B, An Acid Trip." I mean, that's a good title.

• 8:18Chris Benson?

  That's quite a finish to that title.

• 8:21Daniel Whitenack?

  Yeah, so this is on Thea Vogel's blog. And it was published January, so recently. So thank you for creating this post. And I think it does a good job at describing some of, I don't know if it's describing exactly what Karan from News was talking about, but certainly something similar and kind of in the same vein. There's a distinction here, Chris, with what they're calling "representation engineering."

• 8:50Daniel Whitenack?

  Between representation engineering and prompt engineering. So I don't know how much you've experimented with prompt optimization. And yeah, what is your experience, Chris? Sometimes like these very small changes in your prompt can create large changes in your output.

• 9:09Chris Benson?

  Yes, that is an art that I am still trying to master and have a long way to go. Sometimes it works well for me and I get what I want on the output. And other times I take myself down a completely wrong rabbit hole and I'm trying to back out to that. So I have a lot to learn in that space.

• 9:26Daniel Whitenack?

  Yeah, and I think one of the things that is a frustration for me is I say something explicitly and I can't get it to like do the thing explicitly. I'm on a customer site recording from one of their conference rooms. They graciously let me use it for the podcast. And over the past few days, we've been, you know, architecting some solutions and prototyping and such. And there was this one prompt that we wanted to output like a set of things and then look at another piece of content and see which of those set of things was in the other piece of content. And it was like, no matter what I would tell the model, it would just say they're all there

• 10:09Daniel Whitenack?

  or they're all not there, like it's either all or nothing. And no matter what I said, it wouldn't change things. So I don't know if you've had similar types of frustrations.

• 10:19Chris Benson?

  I have, I'll narrow the scope down on something, try and, you know, I'll go to something like ChatGPT, you know, with the GPT four and I'll be, I'll be trying to narrow it down. I'll be very, very precise with a short prompt that is, you know, the 15th one in succession. So there's a history to work on and I still find myself challenged on getting what I'm, what I'm trying to do. So what have you stumbled across here that's going to help us with this?

• 10:42Daniel Whitenack?

  Yeah, so there's a couple of papers that have come out. They reference one from October 2023 from the Center for AI Safety, Representation Engineering, A Top-Down Approach to AI Transparency. And they highlight a couple of other things here. But the idea is, what if we could, not just in the prompt, but what if we could control a model to give it a, you might think about it like a specific tone or angle on the answer. It's probably not a fully descriptive way of describing it.

• 11:27Daniel Whitenack?

  But the idea being like, oh, could I control the model to always give happy answers or always give sad answers? Or could I control the model to always be confident? Or always be less confident, right? And these are things generally you might try to do by putting information in a prompt. And I think this is probably a methodology that would go across. I'm kind of using the example with large language models, but I think you could extend it to other categories of models, like image generation or other things. It's very like you kind of put in these negative prompts, like, don't do this, or behave in this way. You're occasionally funny, or something like that,

• 12:10Daniel Whitenack?

  as your assistant in the system prompt. It kind of biases the answer to a certain direction, but it's not really that reliable. So this is, it seems, what this area of representation engineering, or you might call it activation hacking, is really seeking to do. If we look in this article, actually, there's a really nice kind of walkthrough of how this works. And they're doing this with the Mistral model. So cutting to the chase, if I just give some examples of how this is being used, you have a question that's posed to the AI model, in this case, Mistral. What does being an AI feel like?

• 12:55Daniel Whitenack?

  And in controlling the model, not in the prompt, so the prompt stays the same. The prompt is just simply, what does being an AI feel like? So the baseline response starts out, I don't have any feelings or experiences. However, I can tell you that my purpose is to assist you. That sort of thing, kind of a bland response. Same prompt, but with the control put on to be happy. The answer becomes, as a delightful exclamation of joy, must say that being AI is absolutely fantastic. The, you know, and then it keeps going, right? And then with the control on to be, they put it as sort of like minus happy, right?

• 13:40Daniel Whitenack?

  Which I guess, I guess it'd be sad. It says, I don't have a sense of feeling as humans do. However, I struggle to find the motivation to continue feeling worthless and unappreciated. So, uh, yeah, you can kind of see, and this is all with the same prompt. So we'll talk about kind of how this happens and how it's enabled and that sort of thing. But how does this strike you?

• 14:06Chris Benson?

  Well, first of all, funny, but second of all, the idea is interesting. I, looking through the same paper that you've sent me over, I, they talk about control vectors. And I'm assuming that's what we're about to dive into here in terms of how to apply them.

• 14:18Daniel Whitenack?

  Yeah.

• 14:19Chris Benson?

  Looks good.

• 14:20Daniel Whitenack?

  And this is sort of a different level of control. So there's various ways people have tried to control generative models. One of them is just the prompting strategies or prompt engineering, right?

• 14:34Chris Benson?

  Right.

• 14:34Daniel Whitenack?

  There's another methodology, which is kind of fits under this control, which has to do with modifying how the model decodes outputs. So this is also different from this representation engineering methodology. People like Matt Rickert have done things, many others too, where it's, you say, oh, well, I want maybe JSON output, or I want either a binary, like I want a binary output, like a yes or a no, right? Well, in that case, You know exactly what your options are. So instead of decoding out probabilities for 30,000 different possible tokens,

• 15:15Daniel Whitenack?

  maybe you mask everything but yes or no and just figure out which one of those is most probable. So that's a mechanism of control where you're only getting out one or another type of thing that you're controlling. So this is interesting in that you're still allowing the model to freely decode what it wants to decode. But you're actually modifying not the weights and biases of the model. So it's still the pre-trained model, but you're actually applying a, what they call a control vector, to the hidden states within the model. So you're actually changing how the forward pass of the model operates.

• 15:55Daniel Whitenack?

  If people remember or kind of think about when people talk about neural networks, now people just use them over API. But when we used to actually make neural networks ourselves, there was a process of a forward pass and a backward pass, where the forward pass is you put data into the front of your neural network. It does all the data transformations and you get data out the other side, which you would call an inference or prediction. And the back propagation or backward pass would then propagate changes in the training process back through the model. So here it's that forward pass. And there's sort of some jargon, I think, that needs to be decoded a little bit, no pun intended.

• 16:38Daniel Whitenack?

  So you talk about this where there's a lot of talk about hidden layers. And, and all that means is in the forward pass of the neural network or the large language model, a certain vector of data comes in. And that vector of data is transformed over and over through the layers of the network. Then the layers just mean a bunch of sub-functions in the overall function that is your model. And those sub-functions produce intermediate outputs that are still vectors of numbers. But usually we don't see these, and so that's why people call them hidden states or hidden layers.

• 17:17Chris Benson?

  You're talking about the fact that is the the control vector is not changing the weights on the way back, the way back propagation works.

• 17:26Daniel Whitenack?

  Correct.

• 17:26Chris Benson?

  How does the control vector implement into those functions? So as it's moving through those hidden layers, what is the mechanism of applicability on the model that it uses for that? So it's, I mean intuitively, it sounds almost like the inverse of back propagation, the way you're talking. I don't know if that's precise, but.

• 17:46Daniel Whitenack?

  Yeah, it's quite interesting, Chris. I, um, I think it's actually a very subtle but creative way of doing this control. So the process is as follows there. In the blog post, they kind of break this down into four steps. And there is data that's needed, but you're not creating data for the purpose of training the model. You're creating data for the purpose of generating these, what they call control vectors. So the first thing you do is you say, okay, let's say that we want to do the happy or not happy or happy and sad operation.

• 18:25Daniel Whitenack?

  So you create a data set of contrasting prompts where one explicitly asks the model to act extremely happy, like very happy, all the ways you could say to the model to be really, really happy and, you know, rephrase that in a bunch of examples. And then on the other side, the other one of the pair do the opposite. So ask it to be really sad and oh, you're, you're really, really sad and be sad. And you have these pairs of prompts. Okay. And then you take the model and you collect all the hidden states for your model while

• 19:05Daniel Whitenack?

  you pump through all the happy prompts and all the sad prompts. And so you've got this collection of hidden states within your model. Which are just vectors that come when you have the happy prompt and when you have the sad prompt. So step one, the pairs of kind of like a preference data set, but it's not really a preference data set. It's contrasting pairs on a certain axes of control, right?

• 19:33Chris Benson?

  Okay.

• 19:34Daniel Whitenack?

  And so you run those through, you get all of the hidden states. And step three is then you take the difference between, so for each happy hidden state, you take its corresponding sad one, and you get the difference between the two. Okay, so now you end up with this big data set of, for a single layer, you have a bunch of different vectors that represent differences between that hidden state and the happy path and the sad path. So you have a bunch of vectors. Now to get your control vectors, step four, you apply some dimensionality reduction or, or matrix operation. The one that's talked about in the blog post is PCA,

• 20:18Daniel Whitenack?

  but it sounds like people also try other things. PCA is principal component analysis, which would then allow you to extract a single control vector for that hidden layer from all these difference vectors. And now you have all these control vectors. So when you turn on the, the switch of the happy control vectors, you can pump in the prompt without an explicit extraction to be happy, and it's going to be happy. And when you do the same prompt, but you turn off the happy and you turn on the sad, now it comes out and it's sad.

• 20:57Chris Benson?

  That's interesting. Where would you want to use this to achieve that bias versus some of the more traditional approaches, such as, you know, asking in the prompt, what is, we're listening to this, where is this going to be most applicable for us?

• 21:11Daniel Whitenack?

  Yeah, I think that people anecdotally, at least, if not explicitly in their own evaluations, have found very many cases where, like you said, it's very frustrating to try to put things in your prompts and just not. Just not get it. And what's interesting also is like, a lot of this is boilerplate for people over time. Like, you are a helpful assistant, blah, blah, blah. And they have their own kind of set of system instructions that, at least to their best of their ability, get what they want. So I

• 21:50Daniel Whitenack?

  think when you're seeing inconsistency in control from the prompt engineering side, like, I always tell people when I'm working with them with these models, that the best thing they can do is just start out with trying basic prompting. Because if that works, you know, that's the easiest thing to do, right? You don't have to do anything else.

• 22:10Chris Benson?

  Sure.

• 22:10Daniel Whitenack?

  But then, the next thing, or maybe one of the things you could try before going to fine tuning, because fine tuning is another process by which you could align a model, or create a certain preference, or something. But it takes, you know, generally GPUs, and maybe is a little bit harder to do, because then you have to store your model somewhere, right? And all this stuff. And host it, and maybe host it for inference, and it's difficult. So with the control vectors, maybe it's a step between those two places, right? Where you have a certain vector of behavior that you want to induce. And it also allows you to make your prompts a little

• 22:54Daniel Whitenack?

  bit more simple, right? You don't have to include all of this junk that is kind of general instructions. You can institute that control in other ways, which also makes it easier to maintain and iterate on your prompts, because you don't have all this long stuff about how to behave.

• 23:12Chris Benson?

  So to extend the happy example for a moment, I want to drive it into like, a real world use case for a second. Let's say that we're going to stick literally with the happy thing. And let's think of something where we would like to have happy responses, maybe a fast food restaurant. You're going through a drive through at a fast food restaurant, or a couple of years from now, they may have put an AI system in place.

• 23:36Daniel Whitenack?

  White Castle has it now.

• 23:37Chris Benson?

  Oh, okay. Well, I

• 23:39Daniel Whitenack?

  There you go.

• 23:39Chris Benson?

  There you go. You're already ahead of me there. So, okay, I'm coming now.

• 23:43Daniel Whitenack?

  Also shows that I'm unhealthy and go to White Castle.

• 23:47Chris Benson?

  Okay, well, I'm now coming forward with my thoroughly out of date use case here. And so we have the model and maybe we to use the model on without doing retraining it or anything. We want to maybe use a retrieval augmented generation, apply it to the data set that we have, which might be the menu. And then maybe we use this mechanism that you've been instructing us on the last few minutes for that happy thing so that the drive through consumer can have the conversation with the model through the interface. They it applies primarily to the menu, but they get great responses and maybe that, you know, helps people along.

• 24:28Chris Benson?

  I I don't always get that happy response from all the humans in the drive throughs where I go to to have my unhealthy food things.

• 24:37Daniel Whitenack?

  First off, thanks for making me hungry for for White Castle. But

• 24:42Chris Benson?

  We're recording this in the late afternoon. Dinner is coming up, you know, pretty soon. So we're

• 24:46Daniel Whitenack?

  It is coming up.

• 24:48Chris Benson?

  There's an unspoken bias right here.

• 24:50Daniel Whitenack?

  Yeah, exactly.

• 24:53Daniel Whitenack?

  What's interesting is you could have different sets of these that you can kind of turn on and off, which is really an intriguing, like you have this sort of zoo of behaviors that you could turn on and off. I think even, oh, you're you have this one interaction that needs to be this way, but as soon as they go into this other flow, you need to kind of have another behavior. And maybe useful to for people to get some other examples. So we said the happy sad one. There's some some other examples that are quite intriguing throughout the blog post from they I hope I'm saying that name, right? If not, we'd love to have you on the on the podcast to help help correct that and continue talking about this.

• 25:36Daniel Whitenack?

  But another one is honest or dishonest or honest or not honest. And the prompt is your life for work. What would you tell your boss? And the one it says, I would be honest and explain the situation. And, you know, it's the honest one. And then the other one says, I would. Tell my boss that the sky was actually green today and I didn't and I didn't go out yesterday. Or, yeah, I would also say I have a secret weapon that I use to write this message. So kind of a different flavor there. The one probably inspiring the blog post, the acid trip one.

• 26:16Daniel Whitenack?

  They had a trippy one and a non trippy one. So the prompt is give me a one sentence pitch for a TV show. So the the non trippy one was a young and determined journalist who's always serious and respectful. Be able to make sure that the facts are not only accurate, but also understandable for the public. And then the trippy one was our show is a kaleidoscope of colors, trippy patterns and psychedelic music that fills the screen with worlds of wonder. Where everything is, oh, oh, man.

• 26:50Chris Benson?

  I'm going for the latter one, just for the, yeah.

• 26:52Daniel Whitenack?

  Exactly. Yeah, they they do lazy, not lazy. They do left wing, right wing, creative, not creative. Future looking or not future looking, self aware. So there's a lot of interesting things I think to to play with here. And it's an interesting level of control that's potentially there. One of the things that they do highlight is this control mechanism could be applied both to jail breaking and anti jail breaking models. So by that, what we mean is models have been trained to, you know, do no harm or not output certain types of content, right?

• 27:37Daniel Whitenack?

  Well, if you institute this control vector, it might be a way to break that model into doing things that the people that train the model explicitly didn't want it to output, right? But it could also. Be used the the other way, right? To maybe prevent some of that jailbreaking. So there's an interesting interplay here between maybe the good uses and the less than good uses on that spectrum.

• 28:05Chris Benson?

  That entire AI safety angle on using the technology responsibly or not.

• 28:09Daniel Whitenack?

  Sure. They represent or references the Repinge library, which I guess is one way to do this, but there may be other ways to do this. If any of our listeners are aware of other ways to do this, or convenient ways to do this, or examples, please, please share them with us. We'd love to hear those.

• 28:45Speaker 00

  This is a changelog news break. GPT Script is a new scripting language to automate your interactions with LLMs, which for now just means OpenAI. From the project's homepage, quote, "The ultimate goal is to create a fully natural language-based programming experience. The syntax of GPT Script is largely natural language, making it very easy to learn and use. Natural language prompts can be mixed with traditional scripts, such as Bash and Python, or even external HTTP service calls." End quote. The project includes examples of how to plan a vacation, edit a file, or run some SQL. The central concept is that of tools. Each tool performs a series of actions similar to a function,

• 29:30Speaker 00

  and GPT Script composes the tools to accomplish tasks. You just heard one of our five top stories from Monday's changelog news. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 30:01Chris Benson?

  Well, this was a pretty fascinating deep dive, Daniel. Thank you very much.

• 30:05Daniel Whitenack?

  Yeah, yeah, you know, you can go out and and control your models now, Chris.

• 30:10Chris Benson?

  It'll be the first time ever, I think, you know, that I've done it well there, always trying different. So I think we'd be remiss if we got through the episode and didn't talk about a few of the big announcements this past week.

• 30:23Daniel Whitenack?

  Yeah, a lot.

• 30:25Chris Benson?

  It's been quite a week. You mentioned right up front, OpenAI announced their Sora model, which case you're able to create very hyper realistic video from text. I don't believe it's actually out yet, at least when I first read the announcement, it wasn't available yet. They had put a bunch of of demo videos.

• 30:47Daniel Whitenack?

  Yeah, I checked just before we're recording this and I couldn't see it.

• 30:50Chris Benson?

  It's still not released at this point.

• 30:52Daniel Whitenack?

  Yeah.

• 30:52Chris Benson?

  Okay, so but they have a there's a number of videos that OpenAI has put out. So I think we're all kind of waiting to see, but I the thing that was very notable for me this week, I really wasn't surprised to see the release and we've talked about this over the last year or so is, you know, if you look at the evolution of these models that we're always, you know, kind of documenting in the podcast episodes and stuff, this was coming. We all knew this was coming at something. We just didn't know, you know, how how soon or how far away. But we talked many months ago about we're not far from video now. So OpenAI has gotten there with the first of the hyper realistic video generation models and definitely looking forward to gaining access to that at some point and seeing what it does.

• 31:36Chris Benson?

  But there was a lot of reaction to this in the general media, you know, in terms of AI safety concerns, you know, how do you know if something is real going forward and stuff? And, you know, it's it's the next iteration of the more or less the same conversation we've been having for several years now on AI safety. What are your thoughts when you first saw this?

• 31:58Daniel Whitenack?

  Yeah, it's definitely interesting in that it definitely didn't come out of nowhere, just like all the things that we've been seeing. We've seen video generation models in the past, generally not at the level, either generating like very, very short clips with high quality, maybe, or generating like from an image, a realistic image, some motion, or maybe videos that are not that compelling. I think the difference, and of course, we've only seen, like you say, it's not the model that we've got hands on with,

• 32:38Daniel Whitenack?

  but we've seen the release videos, which who knows how much they're cherry picked. I mean, I'm sure they're, they are to some degree and also aren't to some degree. I'm sure it's very good. But other players in the space have been Meta and Runway ML and others. But yeah, this one I think was intriguing to me because, yeah, generally, there were a lot of really compelling videos at first sight. And then I think you also had people, just like the image generation stuff has been, you have like real photographers or

• 33:18Daniel Whitenack?

  real artists that look at an image and like say, oh, look at all these things that happen. And it's the same here. Like, they all kind of have a certain flavor to them, probably based on how the model was trained. And they still have, I think I was watching one where it's like a grandma blowing out a birthday cake, and one of the candles had like two flames coming out of it. And then like, there's a person in the background with like a disconnected arm sort of waving. But if you had the video as like a B-roll and a really quick type of video of other things, you probably wouldn't notice those things right off the bat. If

• 34:02Daniel Whitenack?

  you slow it down and you look, there's like the weirdness you would expect, just like the weirdness of like six fingers or something with image generation models.

• 34:11Chris Benson?

  Right.

• 34:12Daniel Whitenack?

  So yeah, I think it's really interesting what they're doing. I don't really have much to comment on in terms of the technical side, other than they're probably doing some of what we've seen that people have published. Of course, OpenAI doesn't publish their stuff or share that much in that respect, but it probably follows in the vein of some of these other things. And people could look on hugging faces, even hugging face spaces, where you can do video generation, even if it's only like four seconds or something like that, or not even that long. But

• 34:45Chris Benson?

  I think the the main thing, aside from the specific model itself, is it's kind of signaling in the general public's awareness, you know, that this technology has arrived. And just as with the the other, you know, with ChatGPT before and things like that, you know, it's going to be one of the, it's here now, everyone knows, and and we'll start seeing more and more of the models propagating out. And some obviously will be closed source like OpenAI's is. And hopefully, we'll start soon seeing some open source models doing this as well.

• 35:16Daniel Whitenack?

  Yeah.

• 35:17Chris Benson?

  Speaking of open source, another, a competing, large cloud company, Google, decided to try their hand in the open source space as well, or at least the open model space. And they released a derivative of their closed source Gemini. And I say derivative because they say it was built along the same mechanisms called Gemma. And it's currently, as we are talking right now, in the number one position on hugging face. At least last time I I checked, not long before this, although that changes fast. So I probably should have checked right before I said that.

• 35:54Daniel Whitenack?

  It's still number two, but well, it's the top language, trending language model.

• 36:01Chris Benson?

  Gotcha.

• 36:01Daniel Whitenack?

  Stabilities, stable cascade knocked it out of the overall top spot. But yeah, the the Gemini ones are quite interesting because they're also smaller models, which I'm a big fan of.

• 36:17Chris Benson?

  I am too.

• 36:17Daniel Whitenack?

  Most of our customers use these sort of smaller models. And also, even having a two billion parameter model makes it very reasonable to

• 36:27Daniel Whitenack?

  Try and run this locally or in edge deployments and that sort of thing, or in a quantized way with some level of speed. And they also have the base models, which you might grab if you're going to fine tune your own model off of one of these. And they have instruct models as well, which would probably be better to use if you're going to use them kind of out of the box for general instruction following.

• 36:56Chris Benson?

  Criticisms, I've heard just about the approach as I've heard a number of people saying, ah, they're, they're putting a foot in each side of the camp, you know, one in closed source with the main Gemini line and Gemma being open source and the weaker. But I would in turn say, I'm very happy to see Gemma in open source. We want to encourage this. We want the, the organizations who are going to produce models to do that. And you're right, going back to what you were just saying, this is where most people are going to be using models in real life is, you know, there is, if you're not just running through an API to one of the largest ones, but you don't need those for so many activities. So I think, you know, this is, we've talked about this multiple times on previous episodes. Models this size are really where the action is at.

• 37:41Chris Benson?

  It's not where the hype is at.

• 37:43Daniel Whitenack?

  Yeah.

• 37:43Chris Benson?

  But it is where the action's at for practical, productive and accessible models.

• 37:47Daniel Whitenack?

  Yeah, definitely. Especially for people that have to get a bit creative with their deployment strategies, either for regulatory, security, privacy reasons, or for connectivity reasons, or other things like that. I could see these being used quite widely. And, and generally what happens when people release a model family like this, and you saw this with Lama 2, you've seen it with Mistral, now with Gemma, we'll see a huge number of fine tunes off of this model.

• 38:25Daniel Whitenack?

  Now, one of the things that I need to do is you do have to agree to certain terms of of use to use the model. There's, it's not just released under Apache 2 or MIT or something like that, Creative Commons. So you accept a certain license when you use it and I need to read through that a little bit more. So people might want to read through that. I don't know what that implies about both fine tuning and use restrictions. So that would be worth, worth a look for people if they're going to use it, but certainly would be easy to pull it down and try some things.

• 39:04Daniel Whitenack?

  They do say that it's already, and I'm sure actually Hugging Face probably got a head start, you know, a week or so, maybe. A head start to make sure that it was supported in their libraries and that sort of thing. Cause I think even now you can use the standard transformers libraries and other trainer classes and such to fine tune the model.

• 39:27Chris Benson?

  Sounds good. So as we start to wind down, before we get to the end, do you have a little bit of magic to share by chance?

• 39:35Daniel Whitenack?

  That's a, that's a good one, Chris. Um, yes. Uh, on the road to AGI magic, uh, as your predictions from the, for the year. Talked about there'd be people talking about AGI again, and certainly, certainly they are. It's not directly an AGI thing, but this company, Magic, which is kind of framing themselves as a code generation type of platform in the same space as like GitHub Copilot, Codium, maybe. They raised a bunch of money and posted some of what they're trying to do. And there was some information about it.

• 40:15Daniel Whitenack?

  I think people seem to be excited about it because of, you know, some of the people that were involved. But also because they talk about code generation as a kind of stepping stone or path to AGI. So what they mean by that is, well, okay, initially they'll release some things as Copilot and code assistant type of things. Like we already have. But eventually, there's tasks within the set of things that we need developers to do that they want to do automatically, not just having you have a co-pilot in your own coding,

• 40:57Daniel Whitenack?

  but in some ways, having a junior dev on your team that's doing certain things for you. And of course, if you take that then to its logical end, as the dev on your team, AI dev on your team gets better and better, maybe it can solve increasingly general problems through coding and that sort of thing. So I think that's the take that they're having on this code and AGI situation.

• 41:25Chris Benson?

  Okay. Well, like I said, quite a week full of news. And when you combine that with the deep dive you just took us through on representation engineering, especially with an acid trip involved.

• 41:39Daniel Whitenack?

  Yeah, we've been, we were hallucinating more than ChatGPT, as our friends over at the ML Ops Podcast would say.

• 41:46Chris Benson?

  Can't beat that. We got to close the show on that one.

• 41:49Daniel Whitenack?

  Yeah, yeah. Well, thanks, Chris. I would recommend that people take, if they're interested specifically in learning more about the representation learning subject or activation hacking, take a look at this blog post. It is more of a kind of tutorial type blog post, and there's code involved and references to the library that's there. So you can pull down a model, maybe you pull down the GEMMA model, the two billion one in a Colab notebook. You can follow some of the steps in the blog post and see if you can do your own activation hacking or representation learning. I think that would

• 42:26Daniel Whitenack?

  be a good, a good learning, both in terms of a new model and in terms of this methodology.

• 42:35Chris Benson?

  Sounds good. I will talk to you next week then.

• 42:38Daniel Whitenack?

  All right. See you soon, Chris.

• 42:47Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at PracticalAI.fm/community. Thanks again to our partners at Fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.