The Hidden AI Threat to Your Software Supply Chain

AI-powered coding assistants like GitHub’s Copilot, Cursor AI and ChatGPT have swiftly transitioned from intriguing gadgets to indispensable sidekicks for modern developers. A recent survey by Stack Overflow revealed that over 76% of developers now rely on these assistants, with more than 80% reporting significant productivity improvements by using AI code generators & augmented code editors. These “virtual teammates” simplify complex tasks, streamline development workflows, and significantly accelerate project timelines.

But with every innovation comes new risks. AI coding assistants occasionally generate what’s known as “hallucinations”, confident recommendations for software packages that simply don’t exist.  A recent study by researchers from the University of Texas at San Antonio, the University of Oklahoma, & Virginia Tech found that open-source LLMs generated hallucinated packages at alarmingly high rates — around 21.7% on average — compared to commercial models which averaged about 5.2%. The researchers documented over 200,000 unique hallucinated package names in just this one study, illustrating how pervasive and dangerous this issue truly is.  These aren’t obvious mistakes; they often closely resemble genuine packages, fooling even experienced developers.

Additionally, there’s an intriguing cross-language dimension to these hallucinations. Researchers found that Python-specific code-generating models frequently hallucinated JavaScript packages — an unsettling scenario, given the potential confusion and risks it could create across software ecosystems.

Cyber attackers have quickly seized upon this vulnerability, introducing a new form of supply chain attack called slopsquatting. Unlike traditional typosquatting, which exploits human typing errors, slopsquatting capitalizes on these AI hallucinations. Attackers monitor AI suggested package names, quickly registering these imaginary dependencies on popular repositories such as npm or PyPI. When unsuspecting developers trust the AI’s recommendations and install these packages, they inadvertently bring malicious code into their environments.

Consider the “huggingface-cli” incident from late 2023. After noticing AI repeatedly suggesting this nonexistent package, a security researcher registered it on PyPI as a harmless test. Within days, thousands of developers, including teams at Alibaba, unknowingly adopted this fictitious package into their critical projects. While this incident was benign, it demonstrated just how quickly and easily attackers could exploit AI-generated vulnerabilities.

Now, imagine a scenario where attackers utilize this tactic on a larger, more malicious scale. A single widespread recommendation could rapidly compromise systems all over the world, resulting in significant data breaches, persistent backdoors, or severe disruptions. The automated nature of dependency management in modern CI/CD pipelines amplifies this threat, enabling malicious code to spread swiftly from development environments into production systems.

Traditional security practices often fall short against these emerging threats, as many dependency scanners depend on historical reputation data, leaving them blind to newly registered malicious packages. This underscores the urgent need for innovative, proactive strategies tailored to detect and mitigate AI-driven vulnerabilities.

Development and security teams must adopt rigorous practices to counteract slopsquatting effectively. First, treat AI-generated suggestions with skepticism — never blindly trust suggested dependencies. Implement strict verification processes, manually verifying unfamiliar package names, and consistently using lockfiles, pinned versions, and cryptographic hash verification. Additionally, maintaining internal mirrors or allowlists of trusted packages can drastically reduce exposure to malicious newcomers.

The researchers tested multiple approaches for reducing hallucinations, notably retrieval augmented generation (RAG) and supervised fine tuning. While these techniques dramatically reduced hallucination occurrences (by up to 85%) they also introduced a critical trade off by negatively affecting overall code quality. This underscores the need for comprehensive security solutions capable of proactively identifying threats without sacrificing development efficiency.

Education and awareness also play crucial roles. Clear policies should mandate heightened scrutiny and peer reviews for any AI-generated dependencies. By fostering an organizational culture that questions unfamiliar AI recommendations, teams significantly bolster their overall cyber security posture.