Computer scientists as rogue art historians

What can art historians and computer scientists learn from one another? Actually, a lot! Amanda Wasielewski joins us to talk about how she discovered that computer scientists working on computer vision were actually acting like rogue art historians and how art historians have found machine learning to be a valuable tool for research, fraud detection, and cataloguing. We also discuss the rise of generative AI and how we this technology might cause us to ask new questions like: “What makes a photograph a photograph?” Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Amanda Wasielewski – Website http://www.amandawasielewski.com, X https://x.com/awasielewskiChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Computational Formalism Art History and Machine Learning https://mitpress.mit.edu/9780262545648/computational-formalism/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing well, Daniel. How are you today? I'm actually like, I'm super excited for this conversation because I don't know about you, but I've just been like swimming in generative text AI for like weeks and weeks. As have we all, I think. Yeah, this conversation feels like I can like come up for air and think more to both computer vision and generative image AI and other things like that because we've

• 1:24Daniel Whitenack?

  privileged to have with us Amanda Wasielewski, who is an art historian working in the Digital Humanities Program at Uppsala University, and she's the author of a new book coming out in May, Computational Formalism: Art History and Machine Learning. Welcome, Amanda.

• 1:43Speaker 02

  Hi, thanks. Thanks for having me.

• 1:45Daniel Whitenack?

  Yeah, yeah, this is like I say, I'm really excited about this. So I have to be honest, I was a little bit intimidated maybe because I don't know a lot about art history. But in looking at your book and also looking at your amazing research that you've been up to, like, there's so much practicality in this, both in terms of like what is applicable.

• 2:10Daniel Whitenack?

  To art historians and those working in that area, but also the things that you're talking about in terms of how we think about like machine learning and art and how those relate. And especially in light of generative things in recent years. So, so yeah, it's super excited about this conversation. I'm wondering, you mentioned in the lead up to when we were talking pre-episode that your background is more on the art history side. Where did the art history and machine learning start to collide for you?

• 2:43Speaker 02

  I actually started out, well, I studied chemistry as an undergraduate briefly before kind of discovering art and art history. And was a practicing artist for many years before I went back to studying art history again. So I had a kind of, I guess, I've never been formally trained, but I had a kind of sideline doing artwork that was based through using various digital technologies and certain kinds of programming. And I also kind of worked a little bit in, you know, web design and things like that. So I had a kind of background in computational things for both from a kind of

• 3:25Speaker 02

  art perspective and a professional perspective before I actually went into academia and academic art history. So I've always had those kind of interests in how art and technology coalide. And I kind of, I came to this whole field or the kind of emerging image and AI field through older things like image databases and how they're sorted by metadata, textual metadata. So that was the kind of entry point. And then suddenly it seemed that, you know, more and more art collections or digital image collections were starting to use different computer vision techniques. And so that's kind of how I came at the field through, you know,

• 4:08Speaker 02

  the way that computer vision was increasingly being used to sort large image collections and image collections of art in kind of institutional contexts.

• 4:16Daniel Whitenack?

  It's interesting that you mentioned both elements of like using machine learning to sort art, but also this background of like people using textual metadata to describe art. And I know that you use this word formalism, which in my understanding like has some history in the art world. But like, how standardized is the sort of literature and research around like how you describe the features of like an artwork? That's probably a very like naive way to ask that question as a person not in the field, but like I imagine, you know, metadata to describe artwork is like, you know, artist, Van Gogh, you know, medium, like whatever.

• 4:59Daniel Whitenack?

  What, it seems like what you're talking about goes well beyond that. Could you kind of describe that space a little bit?

• 5:05Chris Benson?

  As a quick add-on, can you also add just a little bit about like what is art history coming into that? Cause a lot of folks on, we'll probably have a lot of people that are doing machine learning, but not a lot of art history background. And some people may be wondering, including me, a little bit about trying to understand what it is. So kind of working your way toward where Daniel was, but starting a little bit earlier for me.

• 5:27Speaker 02

  Well, one of the ideas of the book was actually to kind of, in my own way, try to like bridge this gap because, you know, as I said, I don't have any formal training in any of these, like sort of the computer science side. But, you know, I've, you know, been in this kind of digital humanities million where it's a kind of combination of some computer science techniques with a kind of humanities focus in research. So, you know, I wanted with the book to both kind of introduce art history concepts to those people working in maybe computer vision, but also, you know, introduce people in art history to some of the things that are happening in computer vision.

• 6:08Speaker 02

  So kind of trying to play both sides a little bit, but obviously from my own perspective in art history. And so art history is not a very old academic discipline at all. Its origins in the 19th century revolved around sort of practices of collecting antiquities. So ancient Greek and Roman artifacts. And that kind of collecting practice started to become a more sort of studied and systematic.

• 6:36Speaker 02

  Area coalescing into like the first academic art history departments came about in the late 19th century. And back then, all academic sort of subject matter, the humanities included, kind of aspired to the scientific model in the same way that, you know, the natural sciences. So empiricism, taxonomy, these kind of things. So people at that point in time treated art objects kind of like specimens, like, you know, if they were studying plants and the kind of evolution of plants. And so, early art historians studied art in much that same way. They sort of traced the evolution of art through time and through history.

• 7:16Speaker 02

  And so, it was really focused on, you know, how the kind of superficial qualities of art change over time, rather than a kind of focus on other contextual things, like, you know, the artist's biography, or other kind of circumstantial things about the historical time period. But this has been a longstanding debate in the field, oh, pretty much since the beginning. So, it goes both ways and often falls into two camps. The so-called formalists, who are the ones who just care about the kind of external appearance of images or works of art, and then the people who care about the other stuff, the, you know, what the artist was thinking, what their intentions were, what their kind of historical context was, and all that sort of thing.

• 7:59Speaker 02

  So I'm kind of reaching back into that history of art history. One thing that kind of interested me in this area was, I saw computer vision research, you know, so, research that had no contact with the art history world, really, using data sets of artworks to answer computer science questions. So, you know, not answering art historical questions per se, but in the process, because they're using artworks, they're touching on things that are important to art historians, or that art historians might be interested in. But I saw that there was this kind of call back to these formalist methodologies, similar to what was happening in the late 19th and early 20th century.

• 8:40Speaker 02

  So, I was interested in this, kind of, what I saw as like a revival of these taxonomies kind of matching, like, really simple way, or even sort of, you know, the kind of object recognition by finding different motifs or things like that. So, yeah, that was my, as having had training in art history and its methodologies, that was what kind of piqued my interest in what was happening in computer vision, because I saw it as kind of, um, Like rogue art history that was happening like without art historians having any knowledge that it was happening. So I kind of wanted to like call attention to it on one hand for art historians, but on the other hand, call attention to some of the art historical

• 9:23Speaker 02

  issues that, you know, computer vision researchers may not have found or had access to. So I had that kind of that both directional interest for me.

• 9:34Chris Benson?

  I think Daniel and I probably really like the rogue art historian designation. Who knew that machine learning practitioners would be kind of the pirates of the art history world in that sense?

• 9:45Speaker 02

  Yeah.

• 9:46Daniel Whitenack?

  I've seen a lot of good parallels or memes recently. I think one of my recent ones was like AI is like computer LSD. I think probably like rogue art historian is another good one. So you mentioned that like machine learning people were integrating artworks like into their data sets or to like answer certain types of questions. Were those related to like, I can imagine like, oh, if I have these different artworks in my data set, maybe I can do image classification and classify like this is an artwork or maybe even more detail, like this is an artwork by a person or like in this time period or in this medium

• 10:29Daniel Whitenack?

  or something. But I could also imagine like artwork has objects in it, right? Like can I recognize objects within an artwork or certain like features, that sort of thing? Is that the sort of questions that were being asked? Or what were these questions that you kind of started running across that you connected with the art history world?

• 10:48Speaker 02

  Yeah. So you hit on sort of two of the main areas that were being addressed. And I think from my reading of the literature, as I understand it, the computer vision literature. There was a kind of, you know, obviously, object recognition in images has been a huge focus from the kind of the last 20 years plus because it has so many, you know, quotidian and nefarious applications, you know, you get lots of surveillance applications, but lots of like, you know, we open our phones with our face kind of applications. And the ability of, you know, a machine learning system to like, recognize an object has obvious practical applications. And so I came across a lot of papers that said

• 11:32Speaker 02

  something along the lines of, well, recognizing objects in a photograph is a solved problem. So I think at a certain point in the last like 10 to 15 years, I kind of cover like a 15 year trajectory of this research in my book, researchers kind of were looking for more difficult data sets to tackle. And one of those was art data sets because, sorry, to recognize an object in a kind of stylized painting would be something that would be slightly more difficult. Yeah. So, you know, we had these sort of object recognition activities that were happening, but from like my perspective in art history, it's not a very useful exercise, you know, I don't care really as an art historian if

• 12:17Speaker 02

  there are a bunch of dogs, if you can identify a dog in a painting, it's not that interesting as like a tool to use for my research. So simultaneously, there was a lot of research happening, which is, you know, the kind of categorization by style. And this was really interesting to me because this term style in art history is a really fraught term. It's a really, it has a complicated history and art historians have fought a lot about, you know, what does style mean and how do we define it? And the, yeah, categorization by style in this terms that you're looking at a kind of superficial quality and you're categorizing it by a known kind of textual label.

• 12:59Speaker 02

  I think it's interesting because you know, this has now really important kind of knock on effects in generative AI. Like if you open DALL-E and you see their like kind of suggestion for the initial prompt, they suggest you write, they say an impressionist oil painting of sunflowers in a purple vase. So, right there in the generative AI platforms, you always have these quote unquote "style" markers. So I really wanted to sort of, I guess, unpack what "style" means for art history and what it might mean when we're suddenly applying things like "impressionist" in the context of generative AI.

• 13:47Daniel Whitenack?

  Amanda, I love how you brought us along to understand both like this intersection of art history and machine learning and how like machine learning was sort of dipping into these formalism elements over time. You talked about like the prompts in DALL-E or something like that, like the style. When you're talking about now like art historians kind of realizing how they can employ machine learning within art history, is that the sort of thing that they're thinking about? Like, I could imagine if I take a bunch of artwork, you know, clustering image embeddings to like look at the style of like what is actually similar between all of these images and that sort of thing. That was kind of

• 14:32Daniel Whitenack?

  where my mind went when you were talking about "style", but how have like practically art historians kind of been employing this once they realized that machine learning people were kind of like extracting some of these interesting features?

• 14:48Speaker 02

  Yeah, so exactly in the way that you just described, there is one of the sort of founding fathers of art history, Heinrich Wolfhlin, who he pioneered the, you know, so art historians have always been kind of, you know, using tech for, you know, various, you know, teaching and or research purposes. And he pioneered in the early 20th century, the idea of having a double slide projector in an art history lecture so that you could compare to artwork. It doesn't sound like much to us now, but it was the idea that you could compare side by side in a lecture setting to artworks at once. And so you would kind of see, but you know, the human eye is only able to sort of kind of

• 15:31Speaker 02

  take in so many comparisons at once. And so the way that these type of technologies have been used in art history context is exactly in this kind of mass comparison sense, you know, comparing many, many artworks, many, many more than could be possibly compared in a kind of one single view. So in kind of literary studies, they have something called distant reading and there's a kind of corollary in art historical studies called distant viewing. And the idea is you get a kind of top down, very far away view of general patterns or general trends. And the hope was that you can kind of notice new things through looking from

• 16:13Speaker 02

  this distant point of view. But one of, you know, one of the things that, you know, is important in that is, again, you're looking primarily at visual characteristics.

• 16:23Chris Benson?

  Can I ask a non-technical question? Just when you're doing that remote viewing and you're making those comparisons, like, just to give me a sense of the field, like, what might be an example, like a typical example thing that you're trying to compare, aside from whether it's machine learning or entirely, you know, without technology in the process, just to give me a sense of a touchstone on what that is.

• 16:47Speaker 02

  In terms of what the point of comparison is, or?

• 16:50Chris Benson?

  Yeah, yeah, I'm just kind of curious, just as a newbie to art history and learning from you as we go, I was just wondering what a momentary, aside from the machine learning side of it, what would, what are some of the things you're trying to get to with it?

• 17:03Speaker 02

  Yeah, so this is like the classic art history 101, something we call formal analysis or visual analysis, where the basic step of art history is, you know, first looking without jumping to context or content of an image or work to look at things like texture, line, shape, color, those sorts of basic building blocks of visual information. And once you've kind of understood that, you start to notice details. And I think it's a way of like, looking very closely at an image or an artwork to sort of understand what that is doing visually, what the composition is doing. And then the next tool to add on to that is comparison. So, once you understand

• 17:47Speaker 02

  kind of what's happening on a visual level, purely visual level, you start comparing it. And then you see, okay, so there's different things going on in this other artwork, maybe from the same time period, or maybe from just after it. And so you kind of start to build an idea or narrative around, you know, how artworks change over time. So, that's the kind of standard art history, like 101 skill that, you know, we start to cultivate.

• 18:12Chris Benson?

  I'm sorry that I took you there, but I appreciate you doing it. It is helpful for me.

• 18:16Speaker 02

  Yeah, no, of course. No, I think it's, I mean, it's important because it ties back into thinking about what we want to do. If we want to use, you know, machine learning methods to perform those same tasks, we have to realize or recognize that machine vision doesn't understand images in the same way that we do, as much as we might, you know, remove how we interpret content or context, the way we kind of dissect an image visually, or the way we kind of analyze the visual properties is going to be very different in machine learning exercise. And the first way that that's different is that, you know, the vast majority of things we're dealing with are physical objects that have been digitized.

• 19:00Speaker 02

  So there's like a kind of layer of representation. They're photographs already. So there's already a difference between, say, looking at an artwork in person in a museum and looking at the kind of digital reproduction. I think it is important to sort of understand that foundation as well.

• 19:16Daniel Whitenack?

  So while you're talking about that and kind of the understanding, it's kind of like, I mean, my best parallel would be from the NLP world where like ChatGPT or something does not understand user intent, right? There's no understanding, right? It can produce text, but we process language different than ChatGPT does like as humans. And like you're saying, someone standing in a museum like processes that experience of standing in front of an artwork differently than a photograph, an intermediate representation differently than like a machine might like find features that are good for image classification or something like that.

• 19:57Daniel Whitenack?

  I'm wondering, because a lot of these computer vision models are so non-explainable or like there's an interpretability problem already, right? In terms of like, I might not know why an image was classified in this class with like a convolutional neural net or something like that. Is that a struggle for like taking this field forward in terms of applying machine learning in these contexts? Or are there ways to kind of extract some of those main features like you're talking about like shape and color and line and other things like that?

• 20:35Speaker 02

  Yeah, I think that there's a lot of similar issues actually between the kind of text world and the image world in terms of this idea of what constitutes meaning or understanding. Are you guys familiar with the tank classifier problem?

• 20:49Daniel Whitenack?

  The tank classifier?

• 20:50Chris Benson?

  I'm not, I'm sorry.

• 20:51Daniel Whitenack?

  I don't think I am, although Chris knows about military vehicles, but I don't know about tanks.

• 20:57Chris Benson?

  I don't think that's what we're talking about.

• 21:01Speaker 02

  It was a kind of apocryphal story that was passed around a lot in sort of machine learning circles. The story was, and actually, it dates back to a kind of, someone made this up as an example at some conference, I think in like the 60s. But it became kind of passed around as like it actually happened. The story is that the US military during the Cold War wanted to recognize in images, the, yeah, the tanks.

• 21:29Chris Benson?

  I do, now that, now that you go into it that way, I do remember this, yes.

• 21:33Speaker 02

  Yeah, so like differentiate Soviet versus American tanks in images, but then ended up accidentally classifying the images by the background weather or environmental conditions. And that is the kind of thing that I think like really illustrates what we deal with when we're dealing with images, because we understand things like background and foreground, or the kind of subject and surround in a different way. We interpret those, the kind of illusionistic space of an image in a certain way that, you know, for a lot of kind of algorithmic classification, that surface is what we might call a kind

• 22:13Speaker 02

  of a democratic surface. Like, all areas initially are kind of treated the same. It has to be some kind of training to differentiate those things. And of course, it's gotten very sophisticated, where it is, we are able to sort of separate those things out a lot of the time. But of course, you still get lots of cases, like in sort of medical imaging, like I read a few things about, you know, during COVID, they tried to classify, for instance, like COVID-infected lungs versus healthy lungs, but they used a training set of like children's lung imagery. And so they accidentally classified by children versus adults, which seems like a very silly error to make. But, so we get like issues like

• 22:57Speaker 02

  that, and I think are really important, because what it points to is that essentially, we're dealing with like a two-dimensional surface to interpret. But often, those are two-dimensional representations of a three-dimensional space. That we as kind of three dimensional beings intuitively understand when viewing an image like that or a photograph, for instance, whereas, you know, machine learning algorithms only know that we've kind of isolated a certain pattern of pixels to be a specific object. And, you know, given lots of examples, they're quite good at differentiating whatever object we've designated, but still there's no kind of understanding of space. It's not

• 23:38Speaker 02

  part of the understanding of images in that framework. So I think that that's kind of one of these interesting examples of like, just because it successfully identifies something doesn't mean it it understands what that thing is, like a dog in a photograph.

• 23:54Chris Benson?

  Very good explanation. But I do feel on behalf of the defense industry, I should note that we are much better at identifying and classifying tanks today than we used to be.

• 24:04Daniel Whitenack?

  I don't know if I want to know how good you are. That might be something that I want to be ignorant of.

• 24:12Chris Benson?

  I just feel the need to say that, yeah.

• 24:17Chris Benson?

  I have confidence that things have moved on significantly since the 60s. Someone should tell Vladimir Putin. That's all I'm saying. That's all the politics I'm inserting.

• 24:25Daniel Whitenack?

  So I am really interested in all sorts of things about what we just discussed in terms of the the understanding elements and other things. But I'm intrigued by this in reading through some of the materials about your book and your work. You talk about how computer scientists often process these sort of like art image data sets or or images that are part of their data sets without any real sort of understanding of art or art history. And you kind of one of the things you talk about in the book is how maybe there's an enrichment of like the data science and computer vision side by understanding more

• 25:09Daniel Whitenack?

  of the sort of humanistic issues and elements of the artwork and those sorts of things. Could you describe a little bit what you mean by that and how you think like, because we mostly talked about machine learning kind of enriching maybe art history or things that could be done there. What about the other side of that in terms of like, things computer scientists could learn based on this kind of background and research on the digital humanities side?

• 25:36Speaker 02

  Yeah, I mean, I think one of the things that is really important to me is this idea that, you know, the assumption that accepted categories are in some way static or objective and unchanging can lead to really misleading findings. So, for example, there was one study that I looked at where they were classifying paintings by artistic style and they noted, the authors noted that action painting was confused with abstract expressionism and, you know, said, "Oh, well, in future, you know, we will be able to hopefully rectify

• 26:18Speaker 02

  this categorization error." But for an art historian, you know, those are two kind of contextually specific style terms that two competing art critics came up with, or groups of critics, to, and they have a kind of ideological background. So, there's a reason that some critics wanted to call this mid-century American art movement abstract expressionism, and some wanted to call it action painting. And neither term is really subservient to one another. And you don't need to necessarily understand the full kind of art historical picture to like, you know, say if you're using Dali and you want to make either an abstract expressionist or an action painting as a style, you probably get good results with both of those terms. But the kind of issue

• 27:03Speaker 02

  is that, you know, these are not stable categories. There's, different style categories have very different kind of origins. They're inconsistent amongst each other. You know, some of them span a few centuries, some a decade. Some are small groups of artists who all knew each other and worked together. Some are kind of catch-all terms or contextual terms. So, I think people, you know, in computer science are like, "Great, I have a new dataset to work with and here's the categories and I'm going to work with this and, and then see how effective it is, you know, categories." And that, like, that's fine because they're working on a problem that's different than necessarily what an art historian might work on. But the reason I kind of insert myself there is I'm like, "Hey, well, that is actually kind of an art historical problem that you're working

• 27:47Speaker 02

  on, but in a kind of way that doesn't understand..." That these terms are not fact, that they're not stable in the way that you can kind of like, once you insert something into a database, it becomes kind of solid in a way that it doesn't when you're discussing it like I am, like I could talk for another 20 minutes about, you know, who came up with these terms and why and, and you know, what their political beliefs might be and that sort of thing.

• 28:11Chris Benson?

  Could you talk, maybe not for 20 minutes, but for, for some period of time, I'm kind of curious because you've kind of posed this problem, you know, that's kind of brought by the data science is the way I'm seeing it, whereas you're saying, you know, you may not have those categories correct. What are you proposing as a way of mitigating that in a way that is consistent with art history in terms of approach? Like, how would you, you know, that has that kind of qualitative aspect?

• 28:39Speaker 02

  Yeah, I mean, I think like something I was talking about with a colleague who comes from a kind of computer science background is how do we bring together some of the, you know, concerns and interests of computer science with art history in a way that is kind of interesting to both sides. One of those things is, you know, for art historians, the context of the nuance of terms in a kind of qualitative way is important, but then how do you integrate that into a kind of data context is the question. And unfortunately, I don't have a really good answer, but I know, you know, there are researchers who are beginning to sort of combine different, well, text and image or more different modalities

• 29:19Speaker 02

  of information together to try to create a sort of, you know, or networks, bigger picture about, you know, how we might understand artworks beyond just a kind of textual category. So of course, we can do a kind of like dispense with categories all together and do a kind of purely visual, kind of like unsupervised, like clustering type thing. Then what do we call those clusters? Or what do we call those collections? And that brings you right back to art history once again. So it's this kind of how to integrate all this sort of qualitative nuance within a data context is the big problem as I see it. And I think that's something that I still haven't found or heard a really good solution, but I've

• 30:02Speaker 02

  been talking about it with some of my colleagues. So maybe we'll come up with some bright idea in that area.

• 30:10Chris Benson?

  Could that change depending on what question you're answering with a given training session? Like you could do, you could take different reinforcement learning approaches, but I would imagine that that might change the output. And so you'd be looking for an approach that's kind of consistent with what you're trying to achieve from the art history side of things. Is there any thinking around different approaches based on as you change those that you get different types of outputs? You know, there's something that you're going for that maybe a data science practitioner without the art history might be going for something different, kind of as you've already talked about. What's the thinking around different approaches to it with generative or reinforcement or a combination of them?

• 30:52Speaker 02

  I mean, I don't think that, you know, we can expect that me and a computer vision researcher will have the same goals or desires or outputs out of a research question or problem. But I think from my end, I would like to add some nuance to this kind of the cold data. Because of course, even computer vision researchers, they have a kind of quantitative result, but they end up making an interpretation like the one I just said. They said, oh, well, we've had this confusion between these two categories and we'd like to fix that. So there's always a kind of, you know, as much as, you know, data scientists or computer scientists might think they're just concerned with sort of numbers or output or objective

• 31:35Speaker 02

  facts, there's always actually a kind of interpretive thing that happens. So from my point of view, I think, you know, we might not be answering the same research questions, but we could come together in that kind of, in the same space somehow to build a bigger, better picture of what we're like, whatever phenomenon or artworks or collection of images that we might be looking at.

• 31:57Daniel Whitenack?

  I think that's a really good general vision to have. I think in multiple ways and probably for multiple problems outside of this one. So one of the things that is mentioned in the book and that you discuss are a couple of these like paradoxes that I find really interesting in the fact that like deep learning as applied to like these features of artwork. Can be used to both like create and detect forgeries. So like both of those things are true. And there's like this side of things where like, like high artworks can become digital assets. And like digitally generated assets are in certain cases being considered sort of more

• 32:41Daniel Whitenack?

  like the high art side of things. Like, how are you wrestling with these paradoxes coming up that like machine learning and deep learning are operating on both sides of these things?

• 32:54Speaker 02

  I mean, I obviously think it's really fascinating, this kind of arms race. Or, you know, there's a famous quote by Verlio that the invention of the ship is also the invention of the shipwreck. You can't have one without the other. So I think it's interesting that there's always this sort of positive forward and the, and the sort of destructive, negative element as going on simultaneously. But I think in terms of like, you know, we really saw generative AI explode, you know, in the last, you know, especially with the image tools in the last year and in some months. I think, you know, the latest kind of the Pope jacket hoax of the last week really

• 33:38Speaker 02

  illustrates the extent to which, you know, I mean, we've been kind of distrustful of the authenticity of photographs, you know? I mean, since photography was invented, people were aware that it could be manipulated. In the 19th century, you know, we had hand techniques to manipulate photographs. There's always kind of editing. There was always different kinds of manipulation, but of course, it's only just gotten kind of easier and you know, Photoshop, there were a lot of the sort of fears that are currently being talked about in terms of authenticity or believability or fakeness or trust in images were raised in the 90s around Photoshop. And it, we kind of, you know, became accustomed to Photoshop. But I think, you know, this question of authenticity, you know, whether that's in

• 34:23Speaker 02

  detecting art forgeries or if it's in simply, you know, how we trust the images that we see is kind of rearing up again. We have this access. Now everyone has access to quite sophisticated tools to create photorealistic images that aren't photographs at all. And this is something that I've been working around subsequent to after I wrote the book is the, you know, idea of are the images that are created by some of these generative AI platforms that look indistinguishable from photographs? Can we consider them photographs, actually? So it's a kind of new tool to make photographs that doesn't have a camera, that doesn't have a lens, doesn't have a photographer. It's a kind

• 35:04Speaker 02

  of composite of the learnings of vast data sets. So that's like, all of those questions that I address in the book about like art authentication. And then on the flip side, you know, the idea you could create a forged or fake artwork from a generative tool are, I think, even more kind of relevant in the last year or a few months because of these sort of the new paradigm of creating manipulated images or manipulated photographs.

• 35:35Daniel Whitenack?

  Yeah, yeah, it's interesting that there's this element of what you're talking about where it's like, well, if you would have asked me a year ago, what is a photograph? Like, that would have been fairly clear cut. Like, I think now it's like, well, what really like, is it like you're saying, is a camera needed?

• 35:55Speaker 02

  Yeah.

• 35:56Chris Benson?

  I saw the Pope running from the police. I'm sure.

• 35:59Daniel Whitenack?

  Yeah.

• 36:00Chris Benson?

  Did you all see that one? Did you see that?

• 36:02Daniel Whitenack?

  I don't know if I saw the running one. I definitely saw the puffy coat.

• 36:05Chris Benson?

  It showed the Pope running with police trying to capture him on the street. Yeah.

• 36:09Speaker 02

  But you know, I've been on these kind of, I guess, doing a kind of auto ethnographic embedded study of these, lots of these communities on like Reddit and Facebook and other social media that are just the kind of amateurs, you know, doing mid journey or Dolly images or like Night Cafe or these kind of things. And I've been on them for, you know, over a year, just like reading posts and reading posts and looking at images. And even I, you know, after I, you know, spending so much time on these kind of venues and looking at lots of AI-generated images. My husband just showed me briefly on his phone on Twitter like, oh, do you see

• 36:50Speaker 02

  the Pope was wearing this big puffy coat? I was like, oh, that's weird. I didn't question it.

• 36:56Chris Benson?

  Yeah. And you've been embedded.

• 36:58Speaker 02

  Yeah, I mean, I'm someone who's like actively working on this. So how, you know, how can we expect people to be sort of distrustful when, you know, we want to believe what we see? And I think also in the kind of last, I mean, not to get too political or anything, but in the last kind of decade, the idea of the photograph as a document of, you know, truth telling medium in terms of things like police brutality or like documenting abuse and other situations as a kind of way to expose those things and trust, you know, incidents where the police may not have told the truth about what happened in a particular situation. We put a lot of stake in those things. And so, yeah, then the question becomes,

• 37:41Speaker 02

  you know, what are we facing now? Yeah, we have a new way to have manipulated images.

• 37:47Chris Benson?

  As you were describing that, and I can't, and given the industry I'm in, I can't help but obviously put the filter of my own, my own employment. But it made me realize that there are common problems that an art historian and that people in the intelligence community, for instance, are struggling to deal with at the same time. Who knew that there could be career paths crossing between the two? With that kind of maybe ominous point, where do you think this field is going as you look at doing these different types of qualitative analysis where not everyone is necessarily trying to get the same thing out of combining these fields and recognizing that there are a set of common challenges

• 38:29Chris Benson?

  that, you know, our history has that other fields may have. Where do you see, from your perspective, from your filter, where do you see this going? Where do you see your field evolving into? What kinds of questions do you expect to be asked? And, and what new technologies in the AI world do you either expect or maybe hope to see to help you find those answers in the years ahead?

• 38:54Speaker 02

  Yeah, I mean, I think like art history in particular is fairly technophobic in terms of like maybe would it be the earliest adopters of, you know, AI techniques per se. But, you know, I already think that maybe I don't have such a like sci fi dystopian outlook, but rather a kind of very almost boring outlook on, I think a lot of these tools will just simply be integrated into a research practice, the way that ChatGPT will be used as a kind of aid or different GPT type things aid to writing rather, you know, there's a lot of fear right now in academic settings about, you know, quote unquote cheating in terms of those text generators.

• 39:36Speaker 02

  But I think similarly in terms of image analysis or image recognition, either stylistic recognition or object recognition, will be a, you know, really useful tool in terms of sorting through large art data sets, you know, there's certain kinds of, you know, say for instance, I had a friend who, um, she was studying art in Israel around and before the founding of the Israeli state and they had a lot of art exhibitions, but they didn't keep very good records of what the artworks were that they were exhibiting. So she just had a bunch of photographs of artworks on a wall and had to like set herself to the task of like determining what these artworks were, and they weren't necessarily, you know, very well known artworks. It sounds kind of like a boring application, but you know, might be a very useful tool

• 40:19Speaker 02

  in terms of like, okay, if we had the ability to sort of put this image in and try to identify the artists of unknown artworks through these kind of mechanisms for, for my disciplinary perspective, that would be very useful. I mean, already it's they're being used, these kind of computer vision or machine learning techniques are being used to sort large art data sets rather than accessing artworks through textual metadata, accessing them through, uh, what can be interpreted visually in particular images or isolating images, extracting images, um, matching images across different publications or different exhibition venues. So I have a very kind of boring outlook, I guess.

• 41:04Speaker 02

  I, you know, I don't think it'll lead us to like some kind of scary dystopian future, but it'll just become a kind of naturalized. Tool, resource that we can use, but obviously with the caveat that we always have to think about ethical issues and also think about what categories mean and how we're organizing and arranging things, not just giving over the task of organizing to some unknown black box.

• 41:34Daniel Whitenack?

  I don't think that's boring. I am encouraged by that. As our listeners know, this is Practical AI and I think we all, to some degree, love the practicalities that come out of this. So I think that is actually the exciting part, that this goes beyond the hype and it's making a difference in people's day to day. I think that's where things really get exciting. Well, I really appreciate you joining us, Amanda. It's been a real pleasure to talk through these things. I've learned a lot and I am so thrilled to see the work that you're doing and your contributions, which I think are really important. So yeah, keep up the good work and happy to have you back on anytime to help us parse through some of these things.

• 42:17Speaker 02

  Great. Yeah, thank you guys so much. It's been really interesting and fun. Thanks.

• 42:29Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.