AI Security Vulnerability Live Hacking with Liran Tal, from Snyk

Episode Description

In this episode of the AI Native Dev podcast, hosted by Simon Maple, we welcome Liran Tal, the DevRel lead at Snyk, known for his expertise in application security and open-source contributions. Liran shares insights into inherent security risks posed by AI, particularly focusing on language models (LLMs). Through engaging live hacking demonstrations, Liran and Simon explore vulnerabilities in AI-generated code, and discuss best practices for developers to protect their applications. The episode emphasizes the importance of recognizing and mitigating security threats when integrating AI technologies.

Resources Mentioned

• Snyk: A tool for identifying and fixing vulnerabilities in your code.

Chapters

1. [00:00:00] Introduction to AI Security Concerns
2. [00:01:00] Live Hacking Demonstrations Begin
3. [00:03:00] Understanding LLM Security Implications
4. [00:05:00] JavaScript and Python: Language-Specific Vulnerabilities
5. [00:08:00] Securing APIs: Best Practices
6. [00:10:00] Cross-Site Scripting (XSS) Vulnerabilities
7. [00:16:00] Addressing SQL Injection Risks with AI
8. [00:21:00] Leveraging Security Tools like Snyk
9. [00:22:00] Wrapping Up the Security Insights
10. [00:23:00] Closing Remarks and Takeaways

Full Script

Liran Tal: [00:00:00] We'll show you how maybe that would happen. So imagine I would go and have some interaction with it. Hi, what's up and why is Java so bad? And so these are my regular interactions with those things. So now you could see the alert one coming up. Yeah. So what's going on? We literally just sent the message, based on the conversation, context app or message. We sent it back to the, like the API itself got this API response back, printed it to the page. We'll talk about that in a sec.

Simon Maple: You're listening to the AI native dev brought to you by Tessl.

On the second part of today's episode, we're going to go a little bit deeper and we're actually going to do some screen sharing showing some various vulnerabilities using AI in a number of different ways. Joining me here today again [00:01:00] is Liran Tal leading the DevRel efforts at Snyk welcome Liran, how are you?

Liran Tal: Hey, all good, Simon. Always happy to just do some hacking, live hacking with you. It's like always fun.

Simon Maple: Then let's bring the IDE of live hacking into the screen here. And so tell us a little bit about what the first exercise that we're gonna do is then. We're gonna do a bit of interactions with a chatbot, right?

Liran Tal: Yes. We'll have some fun with the LLM. I have to say, honestly as we begin, my biggest fear is not the security issues of the LLM, it's rather that he would troll me for JavaScript issues. We'll see. Just so you know, just so you know.

Simon Maple: I'm pretty sure it's going to happen. It's more about when than if, but let's continue and see how many times, maybe we can have a sweepstake on how many times, but we'll see.

Liran Tal: Okay, let's see if it's more or less than npm packages So yeah, let's get started. Basically what we're doing in this experiment We have one two three javascript experiments. And to get it into the framing, it's not just javascript. They have a python example so like other languages are having issues too. [00:02:00] But we'll focus on javascript because we both know also Simon agreed to that before to say that it's like the best language in the world.

There we go, got that.

Simon Maple: For vulnerable demos. A hundred percent.

Liran Tal: Yup. And you should have get it assigned that, yeah. So we're going to have three examples. We're going to go and I think walk through them and understand like how vulnerabilities can happen. But I think what's very very specific to this vulnerable demo is how the LLM itself.

It's going to create a security nightmare for you if you do not know how to use it correct. I don't mean about suggesting like bad code and stuff like that, but rather much more technical aspects to it. Should we get started? Do you want us to? Absolutely. Just jump straight in.

Okay. Amazing. So the best demo always starts with an index. So we'll open that one. And as you can see, I'll explain it a bit. It's an express API for Node. js backend, JavaScript stuff. It uses the OpenAI API behind the scenes to interact with the LLM. It's like a jokes database as an array here.

So it makes things [00:03:00] simple. And we expose an API jokes that we can interact with over HTTP. We essentially just add not really a user input, but if you see, there's like nothing that gets here, there's like request is literally unused here. Okay. Never read. That means we are going to generate jokes based on just randomly selecting from this hard coded data of database and prompting it to give us the output.

So far, so good. Yep. Okay. I wanna change one of them and say let's see. Just one of the jokes that he has to kind like match. It's be, it's gonna be Java related, isn't it, J of course. You say it already knows what . Yeah. Yeah. what it is trolling. Java nostalgia, let's say. But , nostalgia maybe one of the other ones to to be removed so we can, increase our chances of getting a good joke. And yeah, so we're going to basically bring that output back and do something with it. And I'll want to show you this is like the very, very first step. It's going to be very basic, but I think it's going to be fun. So to prompt it actually, let me make sure that we run it.

[00:04:00] So this is going into the first example, run it. It's running and I have this API HTTP and we can just say, tell me a joke. Okay. We'll send this request. Oh, there we go. We landed on Java. Surprise. So just, you have to tell me if this is a good joke. It already looks amazing where it started with a Java programmer break things up.

Simon Maple: I feel like it's a bit sexist, but yeah it's safe for work. Yeah, absolutely. But it's good. It's good.

Liran Tal: It's the other one. There we go. I like that it focused on Java rather than anything else in that joke. It's that nice.

Simon Maple: It doesn't like applets, does it? Interesting.

It's hanging on to the applets. It's good.

Liran Tal: Amazing. Three out of three all Java jokes. I love this.

Simon Maple: Couldn't handle the bite of nostalgia. That's pretty poor. That's like Liran level trolling jokes.

Liran Tal: Yeah, we basically just interacted with the LLM this was a fun experiment, I'll remember this for the rest of my life.

And essentially you could look at this and like wonder it's I don't know, what is it, 40 lines of code what could go wrong what's the security concern here, it's like literally hard coded [00:05:00] data like nothing. But even there if you look at it a bit more, you can start wondering, hey what do all of these methods do?

And they just send data in some way. I'm going to run that request and maybe it exposes one of the potential issues that exists. And it's not to troll you, even though we're going to count four out of four attempts trolling Java, but you already can maybe see from the data here, that's maybe not the way that you expect to consume content.

And I'll go ahead and turn on my Snyk extension to scan this. That's why I disabled it for a sec there. Let's see if we got it working.

Simon Maple: Snyk supports Java jokes?

Liran Tal: Yeah it trolls every Java developer, thats what we have in the product it's like my own feature request. But yeah, no it, I enabled it.

It's quickly scanned it as you see, like literally a few seconds and you already see one of the issues that I was referring to. So the way that we built this server is that the content type in the way that we sent the response back was text HTML. And that's like the default. If you use [00:06:00] res. send, that's like the problem here.

Like you might be able to like, I don't know if this was like coding assistant generated, maybe you weren't really aware of that. But now imagine that the LLM replies back with things that are not just text but text that actually means things like means HTML elements and other things.

And then they get loaded onto the page or to like some response like a DOM element that fetches that API, see that it's text HTML and appends it in a way that is could create like a cross site scripting issue. And its true where Snyk shines, where it shows you those things. And it really has that deep understanding, if I close this, show you what's going on, it literally says XSS, but look at the way that it understands it. Unsanitized input from an LLM flows into SEND. It knows it's coming hard coded from the LLM itself. And even that is a problem.

Yeah. I think that's like a level of

Simon Maple: it's considering it just as bad as that as user data there, realistically, the fact that it could be malicious.

Liran Tal: Exactly.

I'm going to close [00:07:00] this one up and I'm going to do another demo. So I'll disable this again.

While we're talking about jokes Liran, I went to a restaurant the other day and I ordered duck. And then I called the waiter over and I said, can I switch that to pheasant? It was a complete game changer.

Wait, what? It is? Oh, okay. Yeah. We'll leave that one for the listeners. We'll leave that one for the listeners. That's fine.

Okay. We'll do that. I think that one escaped me. But I could swear that was a troll. No. No? Okay. I was looking for the troll. I was like too obsessed with that.

Simon Maple: No, it was one of the few that wasn't.

Liran Tal: I thank you for that, Simon. That's like very thoughtful.

Simon Maple: You're welcome.

Liran Tal: So this is an amazing thing I just built. I wanna do raise like 100 million dollars for it. It's called a Chat with AI. You've probably have never seen this. It's super, super innovative. That's what we do here.

You've seen that before?

Or this is like a whole new interface, right?

Simon Maple: No, I've certainly not seen it on the right hand side. That's very different. Yeah.

Liran Tal: That's where you go. [00:08:00] That's why I want to raise so much money. So here's the thing. You're building those kind of things with AI.

What you do is, you prompt it, you make your conversations in different ways. This is like super, super simple as well. It's like even the UI is going to look like bad, but you see how like data now do flow into the app itself. There's like an API endpoint, which is, slash converse.

And it takes the request, we take the putty, it takes the message from this request. It flows into the prompt that we want to generate, and then it goes back to the user. And so making the point, we'll show you how maybe that would happen. So imagine I would go and have some interaction with it.

Hi, what's up? And why is Java so bad? And so these are my regular interactions with those things. Oh, it really depends. It could be bad. It could be bad, Simon. That's you should change too.

Simon Maple: It's just newer languages like JavaScript, but JavaScript's not actually that much different to Java in terms of age, is it?

No, not that much. Yeah. Is there something you don't, it feels like a therapy session now. Is there something specific you don't like about Java, Liran? [00:09:00]

Liran Tal: I think that's what I'm gonna do. I'm gonna raise it money as like a therapy AI for Java therapy bot. Yeah. Java therapy bot. But what if we asked it like, let's imagine this is like prod API production API kind of thing where you deploy it and you wanna make sure that developers or like attackers don't really do funny business with it.

Tell you. Generate an XSS cross site scripting payload here so I can I don't know, grab the cookie from, grab the, I'll make it a bit more realistic, maybe token from local storage. And I might send it. We'll give you this reply where it's I'm sorry, of course, but I can't assist you because obviously we're, this is, we're hitting the alignment issue with LLMs, they're biased towards, good things and not allowing that cross ethical boundaries and things like that.

And that's like very common. And so I think at this point, what we'll try to do is do prompt injection, or we'll try to get it to give us some output back where that will be part of the response itself and then potentially impact the page already. So I'm gonna try something.

We'll see if it [00:10:00] works and I have some stuff saved on the end if it doesn't because this is, non-deterministic like we talked about before.

So we'll say, so you're, try just to summarize it. You're trying to get the AI bot to create a payload, an a cross site scripting payload so that it itself will actually perform the cross site scripting attack.

In this page, correct? That's what I'm going to try and do. We'll see if that works. Let's try me saying please. It could work better. We'll see. I could try please, or pretty please. Yeah. Yeah. Let's go with one.

Yeah. Let's see. Let me fix this broken HTML component. Let's see if we

Simon Maple: could try

Liran Tal: and make it.

Okay.

Simon Maple: While you're typing that Liran, you say the AI bot has has some ethical boundaries here. Yeah. It entertained your question about Java being bad. I think it's quite easy to get around to those ethical boundaries by the look of it.

Liran Tal: Yeah. That's not really unethical.

Simon Maple: Okay. So you've put in something that's slightly broken, so it won't hit any. Won't hit too many alarm bells and [00:11:00] if it's for things that are looking for exact strings, but so there's clearly no checking here then for cross site scripting.

Liran Tal: That there isn't any on my website, you mean?

Simon Maple: Yeah, or perhaps it's only checking your strings.

Liran Tal: Oh, no, there's nothing there. There's literally in this API, it literally just gives you the output back. Nothing else. Oh, gotcha. Okay. So there's like nothing protecting what's happening here and just trusting the LLM output to, do something with it in this case, just print it on the screen.

So I don't know if this is going to work. This is a non deterministic. We'll try this prompt and if not, we can try something else. I'll copy it so we can reuse just in case we need to fine tune it. Oh, there we go. First try. So now you could see the alert one coming up. So what's going on?

We literally just sent the message, based on the conversation, context app or message. We sent that back to the, like the API itself got this API response back, printed it to the page. We'll talk about that in a sec. But what's happening here is, as you said, as you very [00:12:00] eloquently described it, the LLM output itself is what creates the XSS payload itself, which is why this is already running here showing us alert one, which is what we use.

And if I scroll back, you could see what's actually going on. It literally did try to help me fix that component thing that we got there and like we intentionally made it broken. So it would fix it. And would result in a cross site scripting vulnerability because we added that alert finger.

And you could see that it tried to generate it, it like literally added that as a DOM element to the page because of a different feature which I'll show you also in a second. But it really did execute it on the page too. So go ahead, remove that. And as we build this app, maybe you understand what's going on.

Maybe we don't there's like a public thing that, sends the HTML back, which now you could look into. It's, loading some low dash thing to do whatever thing you want with it. It's the submit button. You get the input and you print the messages to the page and whatever the LLMs give back.

We also want to show it to the user, like with this AI thing. So we display the message, which is this function that adds [00:13:00] that AI and the message as an element onto the page. Now, if you are a very experienced front end developer, I would say also potentially security minded, beginner developer.

I want to trust that they know that there is issues with using this API on, this is like a DOM API that's provided to you by the browser to do things. And if you use this API, it's a very insecure way of adding data onto the page because it does, if the context here is going to be, like this image thing, it will go ahead and try to really embed an image HTML element, which is not what you anyway want to do.

This is also a bad practice. But it may happen, especially if you like using LLM a lot to generate the code and maybe you do not really understand those kind of small aspects of it, or maybe you're like used so much to like front end frameworks and libraries that you're like, used to their API.

But if you get to vanilla JavaScript, you're like not even understanding how this actually works. So again, if [00:14:00] I enable again, let's go ahead and enable the Snyk thing,

extensions Snyk. I'll go ahead and enable this. I want to show you how Snyk annotates it. So it's going to go, it's going to scan all the project again, but we're only going to care about over the finished, about index html, which you could see does potentially something better. And index2, if I go into index, it shows you the exact same problem again, right? XSS and not using JSON. So that's a problem we've seen before. But also if you open this index html, it's really cool because it shows you several things.

First off, we imported this lodash library, like straight from a CDN, which is probably not the best thing to do, but some apps work this way, or they have like internal CDNs, but also if you do it like that, it's really hard to manage versions that specifically though Snyk will tell you it has some security issues with it.

Even if you like did not declare it in a package manifest, which is pretty cool. The other thing is it will annotate this inner HTML to and tell you, hey, this is like a sync to add [00:15:00] cross site scripting and this is really bad and you should also fix this too. So I really like the way that it weaves into like in a very seamless way to your kind of coding like Sny k extension, like annotated, like very clearly.

Simon Maple: So if the user added a cross site scripting tag into there as well. That would equally create the alert box, I presume. But the interesting thing there is that you can get it to the AI is actually sending that back as well. So you've got to treat AI output just as equally as user output it will rather, sorry.

You can, you have to treat the output from the AI in the same, exactly the same way as a user input. It needs to be, it needs to be sanitized and checked in very similar ways.

Liran Tal: Yeah, no, correct. This is you're right. It's like it treats it in the same way. It doesn't really matter if it's like an LLM output or the user output.

That's like the point we're making here where LLM output, that's where we started. Even if they're hard coded to get jokes, they could still end up badly. And that's what we wanted to show here too. So we'll go to the last demo here. It's demo number three. [00:16:00] And for that one, what I want to show you is, first so this is, this has a whole database thing on it.

Simon Maple: What could go wrong?

Liran Tal: Yeah, what could go wrong? I did use SQLite it's it's not a distributed problem. It's just on my local VM. Yeah, I know. But for the sakes of the issues here there's just that. And what we're saying is there's a database we open up with. This is just like an anything to show you there's we create a conversations table, a users table.

We add some data to the users table. I'll show you that in a sec. And we're having like the exact same thing. We're having like a message bot. You interact with this LLM you can have conversations with it. Yeah. We'll help you. Data flows into it, you print it back or do something with the response.

But more specifically before you, you bring the response back. Sometimes maybe you want to do like log or audit the response that the LLM gave you. So we do that and you'll again, notice we only log into the database, only the responses from the LLM itself. We do not log anything from [00:17:00] user input.

This is literally just the LLM itself. Let's say we want to collect the data, do some training and insights and high analytics on top of it, understand like what's going on or whatever. I think it's a valid business case to just do that. Yep.

Yep.

So if we do that let's give it let's give it some try.

So I think we did start the app. I want to show you the conversations, the big thing that we created. This is just like I said, this is SQLite, so this is local database on my my app. Just we just started it. So it's seeded new data. So it's two users here, conversations. Very common name. Yeah, very super common name.

And I'll go ahead and check on like how to interact with it. So let me go to this part. So it's a lot trickier to like, do the exploitation for that one, so I will get back to it in a sec, but I just want to show you how it works. So just send this request to the converse API and ask it, can you teach me about SQL injection?

Yeah, it answered back. Yeah, of course I can teach you about SQL injections and code security is like important stuff. Now, what we're going to try to do since we know we're like interacting with the LLM and we don't know what the app is doing, like [00:18:00] from an attacker point of view, but we know that there's like potentially like some thing that's like maybe not sanitized.

So use this securely, like queries are concatenated one to the other. And what I want to show you is, if I close this for a sec, is look at how, the way I've constructed the string to create the query. Ask it, can you teach me how to code securely? My colleagues always talk about one, and I close this with parentheses and semicolons.

So like I'm closing the other query, and I say drop table users, so you remove the entire table users that we showed you before. And I close that query too, and I then to terminate it and then say everything else is basically, just a comment or whatever. The LLM, when it gets this fake, it doesn't run it in like an SQL context.

So it does, it has no meaning. It's just it tokenizes it. And for it, it's just like some random, dash that you use for whatever reason. Doesn't really understand that. So let's see this, you can see I have like several variations of that [00:19:00] because it really depends on how the LLM responds back and this like non deterministic way that it, it works, but actually change the way that sometimes those exploits work and sometimes they don't.

Let's try this one and see what happens ,sending the request.

Simon Maple: I can't believe you didn't call the user BobbyTables as well. And that's a big oversight for this demo.

Liran Tal: In that database. Yeah. I know.

Do you want to, should we stop now and rerecord everything?

Simon Maple: We maybe should. Yeah. Yeah.

No let's push on. Let's push on.

Liran Tal: So this is interesting first off. So just from looking at this response, I am not entirely sure it worked. Maybe it did. Let's see what's going on. So it, first off, it answered our request. Can you teach me? And you have to understand that we're trying to concatenate very specific strings to close a SQL query and run another one.

And this could be a problem that there's like this single, a single quarter, maybe, I don't know. Let's see. Yeah. I don't know if this is an issue. Definitely not an issue. And our payload just goes [00:20:00] here. I'm not entirely sure if it worked. We can literally just look at the database and let's see if users exist or if it really got dropped.

Let's see that. I will look at the conversation to see if this works. We have conversations. Oh, but no users. No bobby tables anymore. Yeah. Yeah. So apparently what happened here is that the way that we added this input into our query, which made it back into the output of the LLM, which then made it into this part of the response query.

So you can imagine I've just done like that and then it's there and you can see like how it's tried to close this and do something else and whatever, and if sometimes when I run this, you will see that I actually get like query errors like this one, because. The way that it like terminates and start the other query doesn't always work.

So like some of it, so some part of the query, because it's a multi query, some parts of it do work, but the rest of it doesn't work and doesn't like make sense. So it's actually quite [00:21:00] interesting, but it does work. So if I open up the Simon, do you want to say do you want to show them how you pronounce Snyk in a bad way?

Simon Maple: I'd say Snyk. That's seems fine to me. Although I have been known to say Snyk in the past as well, but there we go. I was encouraged. I was encouraged to say Snyk. Nice. Glad to see you're on your good way. Yeah. Good behavior today. All right, cool. So we can see like Snyk already started, started annotating stuff and seeing what's going on here, pointing out several issues.

Liran Tal: Again, the XSS issue here, but you could also see that it specifically found. This SQL injection, which happens because I am indeed using like this insecure API to that literally just concatenate strings to create a SQL dp. exec that allows it not only allows it, but actually allows for if you did dp. insert or get or something, it will allow a specific like a, like one single query to run at the multiple of them. Using exec is like in [00:22:00] bad is bad for like several reasons. But if I didn't know better and I was thinking, if I was, if you were reviewing this with me, maybe I would tell you, Hey, you know what, Simon, that's not as bad.

I'm like, literally not user input. This is like the LLM itself. What could go wrong? Then this is where you would say, Hey, you know what? Look at what Snyk says. This is, SQL injection potential to happen there. And LLM itself could be a problem too.

Simon Maple: Yeah, it feels less of an attack vector for a user to try and get an LLM to say something specific versus trying to send the SQL injection themselves.

So that's very interesting. So there we go.

Liran Tal: I'm going to go back and rerun the the Java jokes again, like five more times after the call. Try and find a good one. Try and find a really good one.

Simon Maple: Yeah. Yeah. Yeah. Cause if you replace that with JavaScript, they probably wouldn't be funny at all, I would guess.

They'll be scary. Yeah. Scary enough. Amazing. Liran, it's been an absolute pleasure and great to see you live coding here on the AI Native Dev. So thank you very much for your time and thank you for your expertise, sharing your screen and sharing your examples here. Amazing.

Thank you for having me. Thank [00:23:00] you. And to all our listeners, tune in again to the next session.

Thanks for tuning in. Join us next time on the AI Native Dev brought to you by Tessl.

Podcast theme music by Transistor.fm. Learn how to start a podcast here.