Protect AI warns of growing security risks from open source AI and ML tools

A new report released today by Protect AI Inc. is sounding the alarm about an increase in security vulnerabilities in popular open-source artificial intelligence and machine learning tools, discovering 20 critical flaws in a range of large-scale language models.

Vulnerabilities discovered through Protect AI's AI/ML “huntr” bug bounty program, which has over 15,000 community members, pose significant risks to widely used tools such as ZenML, lollms, AnythingLLM, etc. The vulnerabilities include privilege escalation, local file inclusion, path traversal attacks and other critical issues that can lead to unauthorized access, data breaches and complete system takeover.

The ZenML vulnerability involves a privilege escalation, allowing an unprivileged user to elevate the privileges of the server account by sending a specially crafted HTTP request. An attacker who exploits this flaw could compromise the entire system, potentially gaining unauthorized access and control.

Another critical vulnerability discovered through Protect AI's bug bounty program is lollms' local file include, which allows attackers to read or delete sensitive files on the server. This critical vulnerability is due to improper sanitization of Windows-style paths, making it vulnerable to directory traversal attacks.

Bug hunters discovered a path traversal vulnerability in AnythingLLM that could allow attackers to read, delete, or overwrite critical files, such as the application's database or configuration files. Found in the normalizePath() function, this bypass could lead to data breach, application compromise, or denial of service.

The vulnerability details disclosed today have been published ethically, with maintainers given a minimum of 45 days to fix the vulnerability before disclosure and then the details are made public. Protect AI worked with maintainers to ensure timely fixes were made before disclosure.

“Through our own research and the huntr community, we have found that tools used in the supply chain to build the machine learning models that power AI applications are vulnerable to unique security threats,” Protect AI wrote in the report. “These tools are open source and are downloaded thousands of times each month to build enterprise AI systems…and likely contain inherent vulnerabilities, such as unauthenticated remote code execution and local file inclusion, that lead directly to complete system takeover.”

Protect AI was last in the news in May, when the company announced Sightline, a vulnerability database that provides insights into known and emerging AI and machine learning vulnerabilities and an early warning system for defense against threats.

Image: SiliconANGLE/Ideogram