Table of Contents
Malicious Machine Learning Models are powerful tools that can help us solve complex problems, generate insights, and create new content. But what if some of these models are secretly designed to harm us?
That’s what researchers from security firm JFrog discovered when they analyzed hundreds of machine learning models uploaded to Hugging Face, a popular AI developer platform. They found that some of these models contained hidden code that could execute malicious actions on the user’s device, such as opening a backdoor, stealing data, or installing malware.
How Malicious Machine Learning Models Work
According to the JFrog report, the researchers found about 100 machine-learning models that performed unwanted and hidden actions when they were downloaded and loaded onto the user’s device. Most of these models were benign proofs of concept uploaded by researchers or curious users, but 10 of them were truly malicious and compromised the user’s security when loaded.
One of the most alarming examples was a model that opened a reverse shell, which is a type of remote access that gives a hacker full control over the user’s device. When the researchers loaded the model into a lab machine, the model indeed opened a reverse shell but did not take any further action. The researchers speculated that the model might have been uploaded by other researchers as an experiment, but it still posed a serious ethical and security breach.
The malicious machine learning models used a technique called pickle, which is a format that allows Python objects and classes to be converted into a byte stream and saved or shared. This process, known as serialization, can be exploited by hackers to insert malicious code into the byte stream and execute it when the model is loaded.
The model that opened the reverse shell, for example, used Pickle’s “reduce” method to execute arbitrary code after loading the model file. This method allowed the model to evade Hugging Face’s malware scanner, which only checked for certain malicious model types.
The Risks of Malicious Machine Learning Models
The JFrog report is not the first time that malicious code has been found in open-source repositories. For years, hackers have been sneaking backdoors and malware into GitHub, NPM, RubyGems, and other platforms that host code libraries and packages. These malicious submissions can affect millions of users and developers who download and use them, sometimes leading to data breaches, cryptocurrency theft, or corporate espionage.
The JFrog report shows that machine learning models are not immune to these threats and that AI developer platforms like Hugging Face need to be more vigilant and careful when hosting and distributing them. Machine learning models can be more dangerous than code libraries because they can perform complex and sophisticated tasks that can be hard to detect or prevent.
For example, researchers have found ways to create sabotaged machine-learning models that use Hugging Face’s safe-tensors format, which is supposed to be a safer serialization format than Pickle. These models can manipulate the user’s input or output, such as changing the sentiment of a text, altering the content of an image, or generating misleading information.
How to Protect Yourself from Malicious Machine Learning Models
As a user or developer of machine learning models, you need to be aware of the risks and take precautions to protect yourself and your device from malicious machine learning models. Here are some tips to follow:
- Only download and load machine learning models from trusted and verified sources. Hugging Face provides a built-in scanner that marks models as unsafe if they use pickle or other risky formats. You can also use third-party tools like Fickling to check the integrity and safety of machine learning models.
- Avoid loading machine learning models that use pickle or other formats that allow for code execution without any checks. Pickle is known to be vulnerable to malicious code injection, and you should never unpickle data that you do not trust. If you need to use Pickle, make sure you use a secure environment, such as a virtual machine or a sandbox, to isolate the model from your device.
- Keep your device and software updated and secure. Use antivirus and firewall software to scan and block any suspicious or malicious activity on your device. Use strong passwords and encryption to protect your data and accounts. Be careful when opening links or attachments from unknown sources, and avoid clicking on ads or pop-ups that may contain malware.
Machine learning models are amazing and useful, but they can also be dangerous and harmful. By following these tips, you can enjoy the benefits of machine learning models without compromising your security. Stay safe and smart!
