Attackers use many different techniques to manipulate artificial intelligence and the machine learning process. In this video, you’ll learn about poisoning training data, evasion attacks, and securing the learning algorithms.
<< Previous Video: Physical Attacks Next: Supply Chain Attacks >>
We’re able to make our computers much smarter these days by using machine learning. Our computers are able to sift through enormous amounts of data, find different patterns within that data, and then be able to provide us with services or information based on the patterns that it finds. One challenge with doing this is that it takes a lot of data to be able to train our machine learning systems.
If we’re doing face recognition, we need to have our computers analyze millions of faces to see the nuances between every individual. If this is machine learning that’s in a car, we need that car to understand all of the different things that can happen while somebody’s driving on the road. This means we have to put a lot of data into these systems to be able to train it properly.
But once we’ve done that, we can now start using this data. For example, our spam folder is able to capture a lot more spam because our system has been trained to recognize what spam might look like. Or if we’re in an online retailer, we might see things pop up that are recommended for us based on what our past history is with that particular retailer. Or you might be on a streaming video service and it’s able to offer you movies that it knows you would like based on its machine learning.
And if we’ve trained our cars to recognize harmful situations, it may be able to keep us out of a car accident. All of this training data assumes that everything that’s going into the learning process is legitimate data. But what if the attackers used malicious data or invalid data during the training process? That would mean that the resulting artificial intelligence would be invalid as well.
A good example of poisoned training data occurred with a Microsoft AI chatter bot named Tay. Tay stands for thinking about you. And Tay was added to Twitter on March 23, 2016. It interacted with Twitter users, was able to have conversations with people on Twitter. But Microsoft didn’t add any type of anti-offensive behavior or anti-offensive learning algorithms to Tay.
So once Tay started interacting with other users and other users realized that they could poison the learning process, they turned into a racist, sexist, and inappropriate bot. And ultimately, Microsoft had to turn off Tay and end this particular project. The users on Twitter knew that the artificial intelligence was only going to be as good as the programming and only going to be as good as the data going in during the training process. And if they can find holes or vulnerabilities in that training process, they could change the way the machine learning was able to understand the world around it.
Attackers might also change their approach once the learning process is over. For example, if a spam filter has now learned what spam looks like, the attackers can slightly change the way they’re formatting their spam messages and circumvent the machine learning that was done previously. For example, there have been cases where machine learning has used actual social security numbers and personal information. And attackers were able to interact with that machine learning after the training was over and extract those details and use that private information.
That’s why it’s important during the learning process that all of the data going into the machine learning is legitimate. You might also want to retrain with new data occasionally and make sure that the machine learning is always up to date with the latest type of information. And as a good security technologist, you might want to use some of the same techniques that the attackers are using just to make sure that your machine learning process does not become vulnerable.