LLM and Generative AI for Sensitive Data - Navigating Security, Responsibility, and Pitfalls in Highly Regulated Industries

Transcript

Chaplin: Who here is responsible for AI initiatives within their organization? That might be a decision maker, influence, involved somehow? Who is already doing stuff? Has it involved executing, implementing, looking good. Who's a little bit earlier? Maybe you still have some loopholes, you're still designing, discovery, thinking about. We'll try and tailor the talk accordingly.

I am Stefania Chaplin. I am a Solutions Architect at GitLab, where I work with enterprise organizations, predominantly in security, DevSecOps, but also in MLOps as well.

Mahmood: I'm Azhir Mahmood. I'm an AI Research Scientist at PhysicsX. Prior to that, I was a doctoral candidate at UCL developing cutting edge machine learning models. Prior to that, I was in the heavily regulated industry developing my own AI startup.

Chaplin: If you want to follow along with what we get up to, our website is latestgenai.com, where we publish a newsletter with the latest and greatest of everything happening, always from a secure, responsible, ethical, explainable, transparent perspective.

GenAI is Revolutionizing the World

Mahmood: GenAI is revolutionizing the world. It's doing everything, from hurricane prediction, where NVIDIA were able to forecast Hurricane Lee a week in advance. The team at Isomorphic Labs were able to use AI to predict protein structures. Actually recently, the first ever AI designed drug was developed by the Hong Kong startup, Insilico. It's currently going through FDA trials. At PhysicsX, we use AI for industrial design and optimization, really accelerating the speed at which we can invent new things, but understanding how AI is really transforming the world of engineering and science. How is it changing the world of business? Allen & Overy, they introduced ChatGPT 4 to 3000 of their lawyers.

Since then, with Microsoft, they developed contract matrix and are using it for drafting, reviewing, and analyzing legal documents. They've now launched it to their clients. At McKinsey, Lilli is able to condense all of McKinsey's knowledge, about 100,000 documents, and allow it to be called within mere moments. That's really reduced the amount of time associates are spending preparing for client meetings by about 20%. It's really unlocking a whole new area of creativity for them. Bain & Company, within their recent M&A report, they found about 50% of M&A companies are leveraging AI, especially within the early stages. They're doing everything from sourcing, screening, and due diligence. Now, having seen how AI can really revolutionize your business?

What Are Highly Regulated Industries, and Types of Sensitive Data?

Chaplin: What are highly regulated industries and types of sensitive data? These are industries that have government regulations, compliance. They have standards and laws, and these are in place for consumer protection, public safety, fair competition, and also environmental conservation. Some examples of the industries. Who here is in finance? Medical or pharmaceutical? Defense, government, and utilities? Anything I didn't say, anyone missing? I think I've got everyone covered. In terms of types of sensitive data, and the important thing when it comes to machine learning and AI, you are only as good as your data, so if you have bad data, how are you going to train your models? Especially when it comes to sensitive data, there are many types, and you're going to have to treat them differently. For example, maybe you need to start obfuscating or blanking out credit card details. You need to think about your data storage, how you're going to store it.

Also, types, so things like PII, name, email, mobile, biometrics, religion, government, ideologies, sexual orientation. There's a lot of different types of PII out there. From an organizational perspective for business, a lot of companies, especially larger ones, highly regulated, are listed. Any information that could influence the stock price will be counted as sensitive data, because if that is leaked or used nefariously, it will have issues. Within health, hopefully you will have come across the HIPAA regulation, which is one of the stricter ones out there in terms of medical records and what to do. Also, high risk. I work with a lot of defense organizations in my experience at GitLab, and it's very much, we have a low-risk data, medium, classified. We have our diodes for getting the data around. If you're going to be using sensitive data as part of your machine learning models, you really need to pay attention to making sure it's safe, secure, and robust.

The AI Legislation Landscape

The legislation landscape, and this has been evolving over the last five, six years. I'm starting off with GDPR. GDPR, if you were working in a customer facing role, it was everything everyone could talk about around 2016, 2017. What are we going to do, personal data? How are we storing it? Is it transferring to our U.S. headquarters? It was one of the first legislations that focused on data. Like I said, good data, good models, good AI. In 2021 we had the AU AI Act proposal. If you're watching this space, you may have noticed this just got passed into law 2024. This was very much around different risk classifications, so low risk, critical risk. It's the first time we're really talking about transparency in legislation. 2023, we had a few things happening. The first one was in D.C. We had the Stop Discrimination Algorithmic Act. Which then evolved to become Algorithmic Accountability Act, which was federal, so across the whole U.S., because what usually starts in D.C. spreads across the U.S, and the world.

With this one, it was very much, we need to understand what AI we're using, and what are the implications, and what is happening? You see less of the, "It's a black box. You just put in someone's name, and it tells you if it's alone." No, we really need the transparency and accountability. UK, they've taken a bit of a different approach, where it's very much about the principles. For example, transparency, explainability, responsibility. We'll be talking about these a bit later. Having fairness, having a safe, secure model. It was very much more focused for innovation and how we use AI.

The final one, United Nations, so this one's been very interesting. This one was talking about human rights. It was saying, we need to make sure, for one, cease using any AI which infringes on human rights. Also, quotes from the session included, we need to govern AI, not let AI govern us. If you see the way the legislation has evolved from just personal data, transparency, explainability, it's now become a global human rights issue. What you can see, here we have the map of the world and the vast majority of North and South America, Europe, Asia-Pac, there are legislations either being talked about, being passed. You probably are from one of these countries. It's also worth noting, where does your company do business? Because a bit like GDPR wasn't in America, but if you're an American business doing business with EU, you're affected. It's good to keep on top of what the latest and greatest are. I mentioned our newsletter at the beginning.

How to AI?" What is MLOps?

How to AI? What is MLOps? If I was going to ask you, what is an MLOps pipeline? How do you do AI? Who could give me a good answer? I have a little flow, a little bit like a Monopoly board, because you just keep going around. Starting off in the top left and walking around, this is where it's the data. For example, what is your data source? Is there sensitive data? How are you ingesting that data? Like, we want to look at our financial records, our sales records for this product, or we want to predict customer churn, so we're going to look at our customer records. That's great. How are you ingesting or storing that data? Especially with sensitive data, you really need to think securely how you do this. What's your data? It's ready, validate, clean it. What are you going to do if there's null data? What are you going to do if your data's been manually entered and there are mistakes? How are you going to standardize and curate it? This is majority the data engineer.

What this is meant to show, this flow, is that you have a lot of different people a bit like DevSecOps, really, who are overlapping and getting this done. Now the data is done, then we get our data scientist and ML engineer, so we think about our features. What are we looking at for features, for our ML? Then we have our model. This comes back, should be the final thing on this is our business user, what problems are we solving? Is it regression? Is it classification? Is it anomaly detection? Random Cut Forest is great for that. Once you have your model, train it, validate, evaluate. Ok, we have it ready. We've got our data. We've got our machine learning. Now let's do it at scale, so containerize, deploy, make it servable to applications. Then all starts again, because once you've got it working once, great, time to keep updating, keep retraining your models.

You can see there are a couple of different users involved in this. Really, when I think of MLOps, you have data, which I've spoken about quite a bit. You have your machine learning, something Azhir is very good at. If you only focus on those two, you get back into the problem of, it worked on my machine, which is something that a lot of people have heard from a long time ago in the DevOps world. This is why you need to bring along the best practices from software engineering. Having pipelines, pipelines that can standardize, they can automate, they can drive efficiency. You can add security scanning, pipelines as codes, so that you can do your continuous integration and continuous deployment, so CD4ML. It's really this overlap, because if you have one data source, one model and one machine, ok great, you're doing AI. This is very much about how you do enterprise at scale, making sure that is accessible.

When Things Go Wrong

What happens when things go wrong? These are mainly news stories from the last six months, and they cover a few things, so bias, hallucination, and going rogue. We had DPD, January 18th, and a lot of organizations are using a chatbot as part of customer service. Unfortunately for DPD, their chatbot started swearing at the customer and then telling the customer how terrible DPD is as a company. Not ideal, not the type of behavior you're trying to generate. Bias is a really important area, especially maybe 10 years ago. I got one of my favorite birthday presents I received at the time, a book, "Weapons of Math Destruction." This is very much talking about the black box. It's by Cathy O'Neil. It's great and it's ok. How do we stop bias, if it's trained on specific datasets? For example, with image detection, UK passport office exhibited bias based on the color of people's skin. Hallucinations, I say, a more recent problem. This is a real problem for a few reasons.

A few weeks ago, you had AI hallucinating software dependencies. I'm a developer, and I'm like, yes, PyTorture, that sounds legit. Let me just add that. If I take my developer hat off, I put my hacker hat on, ChatGPT is telling me about a component that doesn't exist, I know what I'm going to do. I'm going to create that component, and I'm going to put malware in it. Then everyone who uses ChatGPT for development and uses PyTorture, they will be infected. It's not just dependencies, we can see there was hallucinating a fake embezzlement claim. What was interesting about this, this was the first case of AI being sued for libel. When I speak to organizations who are still earlier in the journey, this is where the lawyers come in and think, what if the AI does something that is suable, or injures someone? This is where the whole ethical and AI headspace comes in. In terms of security, top one, plugins.

Plugins are great, but you need to make sure you're doing access control. I'll be talking about this a bit later. Because one of the critical ChatGPT plugins exposed sensitive data, and it was to do with the OAuth mechanism. Who remembers Log4j from a few years ago? Log4j happened just before Christmas, and then, I think this was maybe two years later, PyTorch on Christmas Day, between Christmas Day and the 30th of December, so when not many people are around, there was a vulnerable version of PyTorch. This collected system information, including files such as /etc/passwd and /etc/hosts. Supply chain, this isn't a machine learning specific vulnerability. This is something that affects all, and what you'll see in the talk is it's a lot about the basics. Finally, prompt injection. This is ML specific. What happened to Google Gemini is it enabled direct content manipulation, and it meant that you could start creating fictional accounts, and hackers could see only the prompts that only the developers could see.

Responsible, Secure, and Explainable AI

Mahmood: Now that you know essentially all the problems that can go wrong with machine learning models and generative AI, how do you avoid that? Fundamentally, you want to develop a responsible framework. Responsible AI is really about the governance and the set of principles your company develops when trying to develop machine learning models. There's a few commonalities. A lot of people focus on human-centric design, fairness and bias, explainability and transparency. Ultimately, you should assure that your responsible AI set of principles align with your company values. What are some responsible AI principles out there? Over here, we're spanning across Google and Accenture, and you'll recognize a few commonalities. Say, for example, there is a focus on fairness.

Fundamentally, you don't want your machine learning model to be treating people differently. There's a focus on robustness. You want to ensure that the model is constantly working and the architectures and pipelines are functional. Transparency, it should be the case that you should be able to challenge the machine learning model. Every company, from Google and Accenture, seem to have overlaps here. Fundamentally, what you want to avoid is you want to avoid having a situation where you're developing Terminator, because that doesn't really align with human values and generally isn't a good idea. Assuming you want Earth to continue being here.

What are some practical implementations you can use? Be human centric. Ultimately, machine learning, AI, they're all technologies, and really the importance of it is how actual users experience that technology. It's not useful if you develop a large language model, and essentially nobody knows how to communicate or interact with it, or it's spewing swears at people. You want to continuously test. I recognize that the world of AI feels very new, but ultimately, it's still software, you need to test every component. You need to test your infrastructure. You need to test pipelines. You need to test continuously during deployment. Just because a model may work doesn't necessarily mean the way it interacts with the rest of the system will get you the result you want. There's multiple components here.

Also, it's important to recognize the limitations. Fundamentally, AI is all about the architecture as well as the data. Analyze your data. Identify biases of your model and your data. Make sure you're able to communicate those biases and communicate the limitations of your system to your consumers and pretty much everyone. Finally, identify metrics. To understand the performance of your model, you need to use multiple metrics, and they give you a better idea of performance. You need to understand the tradeoffs, and thus you're able to really leverage machine learning to its full capacity.

To summarize, develop a set of responsible AI principles that align with your company values, and consider the broader impact within society. Don't build something like Terminator. Be human centric. Engage with the broader AI community. Ensure you're thinking about how people are interacting with these architectures. Rigorously test and monitor every component of your system. Understand the data, the pipelines, how it all integrates into a great, harmonious system.

Now that we understand how to be responsible, how do you be secure?

Chaplin: Is anyone from security, or has worked in security? This might be music to your ears, because we're going to cover this, and it's going to cover a lot of the basic principles that are applied across the IT organization. Secure AI, secure and compliant development, deployment, and use of AI. First, I'm going to talk a little bit about attack vectors, popular vulnerabilities, some prevention techniques, and summarize. You can see, this is my simplified architecture where we have a hacker, we've got the model data, and we've got some hacks going on. Who's heard of OWASP before? OWASP do a lot of things. Something they're very famous for is the Top 10 Vulnerabilities. It started off with web, they do mobile, infrastructure as code. They do LLMs as well. This I think was released '22, or '23. What you'll notice with the top 10, some are very LLM specific, for example, prompt injections, training data poisoning. Some are more generic.

In terms of denial of service, that could happen to your models, it can happen to your servers, or your laptop. Supply chain vulnerabilities, so what I mentioned earlier with PyTorch. If I have a look at those, and you have those top 10. We have our users who are actually on the outside. We have different services. We have our data, plugins, extensions, more services. What you'll notice is there are multiple different attack vectors and multiple places these vulnerabilities can happen. For example, sensitive information disclosure pops up in many times, especially around the data area. We have excessive agency popping up again. In security, you are as strong as your weakest link. You need to make sure every link, every process between these services, users, and data, is as secure as possible, because otherwise you can see, you leave yourself open to a lot of vulnerabilities.

In terms of prevention techniques, so with security, there are a lot of, I call them security basics. For example, access control. Who can do what and how? How are you doing your privileges amongst your users? How are you deciding who can access what data, or who can change the model? Also, you've set that up, but what happens if something goes wrong? Monitoring is very important. It actually moved up in the OWASP Top 10. It moved from, I think, number nine to number six, because if you're not doing logging and monitoring, if/when something goes wrong, how are you going to know? It's very important, whether it's for denial of service, for supply chain, for someone getting in your system, to just have logging and monitoring in place. For data specific, you need to think about validation, so sanitization, integrity.

A rule that I do, I worked in security training for a few years, and if everyone just checked that input, are you who you say you are? I know that usually training data comes from here, but are we just verifying that that is who it says it is? That would solve a lot of the different vulnerabilities. Even stuff like upgrading your components, having a suitable patch strategy in place. This is what I meant when I said it's your security basics, access control, monitoring, patching. If you want to look at a good example of someone who's got a really good framework, check out Google's SAIF, Secure AI Framework. You can go online, find it. They've got some really useful educational material, because it's talking about all these concepts. We're talking about the security foundations, detection and response, defenses, controls, looking at risk and context within business processes.

To summarize, adopt security best practices for your AI and MLOps process. If you're doing something for your DevSecOps or for your InfoSec, you can probably apply those principles to your MLOps and AI initiatives. I've mentioned access control, validation, supply chain verification. Number two, security almost manifesto, you are as strong as your weakest link, so make sure all your links are secure. Finally, check out OWASP and Google SAIF when designing and implementing your AI processes. You can also sign up our newsletter, and we'll have more information.

Now we've talked about secure AI, let's talk about explainable AI

Mahmood: What is explainability? Fundamentally, explainability is a set of tools that help you interpret and understand the decisions your AI model makes. That's an overview. Why should you use explainable methods? Number one, transparency. Fundamentally, you should be able to challenge the judgment that your machine learning model is making. Then, trust. It's great if your machine learning model is a black box, but ultimately, nobody is really going to trust a black box. You want to be able to bridge that barrier. It improves performance, because if you can understand your system, then you can identify components that are weak, and then, as a result, address them. You also are able to minimize risk. By using an XAI framework, it's the shorthand for Explainable AI, you're able to also comply with most regulatory and compliance frameworks. That's a broad overview of its importance. We're not going to go in too much detail, but here are a few. This is a broad landscape of what the XAI field of research looks like.

Fundamentally, there are model agnostic approaches which treat your machine learning model as a black box and then allow you to interpret that black box. You have your large language model. It's not initially interpretable. You use a few techniques and you can better understand which components are working and how they're working, at least. Then there's model specific approaches. These are tailor-made approaches specific to your architecture. For example, with a model agnostic approach, you could have any architecture. Transformers, those are generally what large language models are. Multiple MLPs, those are quite common deep learning architectures. Then with model specific approaches, it's specifically tailored to your machine learning architecture or even your domain.

Then, you also have both global analysis, so look at your whole model and attempt to understand it in its completion, and local analysis, which really identify maybe how individual predictions function. Within the model agnostic approach, there's a common technique, it's called SHAP. SHAP uses game theory and ultimately helps identify the most important features. You also have LIME. LIME takes in data and then fundamentally looks at how each prediction aligns. It's really great at identifying for single data points. Then there's the broader, holistic approach developed by Stanford University, it's called HELM. It's actually called Holistic Evaluation of Large Language Models. They look at the whole architecture and have a number of metrics that you can leverage. Then there's BertViz. BertViz helps you identify the attention mechanism within your model. This is like a whistle-stop tour of explainable AI.

I imagine what you're probably more interested is how organizations are using explainable AI. In the case of BlackRock, initially they had a black box. The performance of the black box was great. It's actually pretty superb. What happened was the quants couldn't explain the decision-making processes to stakeholders or customers, and as a result, that black box model was scrapped. It was then replaced with an XAI process, and that way the decision-making process could be understood. At J.P. Morgan, they've heavily invested in XAI. Actually, they have a whole AI institute dedicated to research, which they then integrate. At PhysicsX, we actually leverage domain knowledge. I, for example, may develop an architecture, I get a few results.

Then what will happen is I then communicate with the civil engineer, the mechanical engineer, really understand, does my prediction make sense? Then I leverage that expert judgment to improve my model and understand its failure points. IBM also leverage things called counterfactuals. Say, for example, they might ask a large language model, I have black hair, and it'll provide some result. The counterfactual of I have black hair is I have red hair or I do not have black hair. That helps you better interpret your model where the data is missing. For example, PayPal and both Siemens have great white papers that really go into details about this whole field.

To quickly summarize, you can use expert judgments and domain knowledge to better interpret and understand the performance of your architecture. It's a good idea to stay up to date with the latest in research within the field. It's an incredibly fast-moving field. Make use of model agnostic approaches and model specific approaches. Think about your model in terms of globally, how does the whole architecture work, as well as locally, how does it perform for each individual data point? There's actually a really great paper by Imperial, called, "Explainability for Large Language Models," which I imagine many of you might find interesting, and that identifies approaches specific for that domain. Also, you can leverage open-source tools as well as classical techniques. For the case of large language models, there's BertViz, HELM, LIT, and Phoenix. These are all open-source tools. You can literally go out, download today, and get a better understanding of performance of your model, as well as you can use more classical statistical techniques to understand the input data, the output data and really how things are performing.

The Future of AI

Chaplin: We've spoken a little bit about how GenAI is revolutionizing the world, highly regulated industries, sensitive data, regulation, when AI goes wrong. We've given you a responsible AI framework that covers responsibility, security, and explainability as well. Let's talk a little bit about the future of AI. We're going to take it from two lenses, my view, and then Azhir's. I come from a cybersecurity background. Maybe I'm biased. How AI can strengthen cybersecurity in the future. Michael Friedrich was talking about AI automated vulnerability resolution. As a developer, vulnerability in my pipeline reflected cross-site scripting. What does that mean? What AI will do, one, it will explain the vulnerability. "That's what it is. This is what the issue is." Even better, we have automated resolution. Using GitLab, and there are other tools out there, we will generate a merge request with the remediation. As a developer, my workflow, it's like, I've got a problem. This is what that means. This is the fix. It's all been done for me. I even have an issue created, and all I need to do is manually just approve.

The reason we have that is because if you have a change, say you're updating a component from version 2.1 to version 9.1, you might just want to check that with your code, because it's probably going to introduce breaking changes. That's why we have the final manual step. That's from a developer perspective, that's security. From a more ops perspective, incidence response and recovery. AI is very good at noticing anomalies, and PagerDuty are doing a good job at this, because they can identify something, "Something unexpected is happening. Let's react quickly." Maybe we block it. Maybe we're going to alert someone, because the faster you react the slower the attack vector is.

An example, if anyone remembers the Equifax hack from a few years ago, it was a Struts 2 component, and it took them four months to notice that they had been hacked. You can imagine the damage. It was actually 150 million personal records, which is over double the UK, most of America. This is before GDPR. Otherwise, the fine would have been huge. They lost a third of market cap. This was to do with the attack vector. Finally, NVIDIA, so they are using GenAI as part of their phishing simulation. Using GenAI, you can generate these sandbox cybersecurity trainings, so help to get all of your users as secure as possible. I'm sure everyone's come across social engineering. My brother-in-law is an accountant, and his whole team are terrified of opening the wrong link and accidentally setting the company on fire. GenAI can really help to speed up these phishing and cybersecurity initiatives.

Mahmood: More broadly, what does the future of AI actually look like? What we're likely to see is increasing prevalence of large foundation models. These are models trained on immense datasets. They may be then used for numerous domains. They may be multimoded or distributed, but what they'll be used for is everything from drug development, industrial design. They'll touch every component of our lives. As they integrate within our lives, we expect AI to become increasingly regulated, especially as they begin to be integrated within health techs, finance, all these highly regulated domains.

Summary

When designing and implementing AI models, think responsibly. Make sure to use a responsible, secure, and explainable framework.

Chaplin: Keep an eye on the legislation and regulations to stay compliant, not only of the country you are based in, but also to your customers.

Mahmood: AI is more than just tech. It's all about people. It's how these architectures and models interact with the broader society, how they interact with all of us. Think more holistically.

Chaplin: If you are interested in finding out more, we work with organizations doing MLOps, doing AI design, doing a lot of things, so check out our website.

Questions and Answers

Participant 1: You're talking about HELM as an explainability framework, but to my knowledge, it's just an evaluation method with benchmarks on a leaderboard. Can you elaborate a bit on that?

Mahmood: Fundamentally, one way you can think about explainability is really understanding multiple benchmarks. Say, for example, with HELM, I have to read through the paper to make sure I fully interpret everything. If you're able to understand how maybe changes in your performance, say, for example, you pretrain, you use a different dataset, and then you evaluate it, it gives you more interpretation to your model. That's a way you can holistically interpret. That's one way you could leverage HELM.

Participant 1: That doesn't give you any explainability for highly regulated environments, I think. You compared it with SHAP and LIME, and there you get an explanation of your model and your inference.

Mahmood: Within explainable AI, there's multiple ways you can think about explainability. There is the framework of interpretability where interpretability is fundamentally understanding each component. I would probably argue, yes, LIME provides you with some degree of interpretability, but explainability is much more of a sliding scale. You have where you might have a white box model, where you understand every component, where you might understand the components of your intention mechanism. While you can also use more holistic metrics to maybe put up your datasets and understand that. Those could be applied to less regulated domains, but still relatively regulated domains. You would, of course, use HELM with other explainable frameworks. You shouldn't be relying on a single framework. You should be leveraging a wide approach of tools. HELM is one metric, and you can leverage numerous other metrics.

Participant 1: Shouldn't you promote more the white box models, the interpretable ones, above the explainability of black box models, because LIME, SHAP are estimations of the inference. With the interpretable model, you know how they reason and what they do. I think in a highly regulated environment, it's better to use white box models than starting with black box models and trying to open them.

Mahmood: The idea is leveraging white box models over black box models, and attempting to interpret black box models. To some degree, I agree. We could leverage white box models increasingly within regulated domains, but what you end up finding is we sacrifice a great deal of performance by leveraging a white box model. It's not necessarily easy to integrate a white box model into a large language model, that's still an area of research. You do lose some degree of performance.

Then, within the black box models, yes we have great degrees of performance. Fundamentally, say, for example, in medical vision, it makes more sense to use a CNN, because using a CNN, you'd be able to detect cancer more readily. Using a white box model, yes, it's interpretable, but it might be the case it's more likely to make mistakes. As a result, what you'd use then is a domain expert with that black box model. I would probably say it fundamentally depends. It depends on how regulated your industry is. How much performance do you need, again, fundamentally in your domain? There's a whole host of tools out there.

See more presentations with transcripts