Understanding the Risks of Open-Source AI

Hot off the press! This is one of most comprehensive, international and collaborative reports on AI safety to be released to date. It's being released on the eve of the Paris AI Action Summit, that I'm excited to attend in February. I was honored when Yoshua Bengio's team at Mila - Quebec Artificial Intelligence Institute (Université de Montréal) reached out to me to give feedback on the section of the report about "open-source"/"open-weights" AI models starting on page 149. A few excerpts that I found helpful, and supportive of the arguments I've been making on this topic for quite some time now: "Risks posed by open-weight models are largely related to enabling malicious or misguided use. General-purpose AI models are dual-use, meaning that they can be used for good or put to nefarious purposes. Open-model weights can potentially exacerbate misuse risks by allowing a wide range of actors who do not have the resources and knowledge to build a model on their own to leverage and augment existing capabilities for malicious purposes and without oversight. While both open-weight and closed models can have safeguards to refuse user requests, these safeguards are easier to remove for open models. For example, even if an open-weight model has safeguards built in, such as content filters or limited training data sets, access to model weights and inference code allows malicious actors to circumvent those safeguards. Furthermore, model vulnerabilities found in open models can also expose vulnerabilities in closed models. Finally, with access to model weights, malicious actors can also fine-tune a model to optimise its performance for harmful applications. Potential malicious uses include harmful dual-use science applications, e.g. using AI to discover new chemical weapons, cyberattacks, and producing harmful fake content such as ‘deepfake’ sexual abuse material and political fake news. As noted below, releasing an open-weight model with the potential for malicious use is generally not reversible even when its risks are discovered later...." "A key evidence gap is around whether open-weight releases for general-purpose AI will have a positive or negative impact on competition and market concentration. Publicly releasing model weights can lead to both positive and negative impacts on competition, market concentration, and control... However, this apparent democratisation of AI may also play a role in reinforcing the dominance and market concentration among major players. In the longer term, companies that release open-weight general-purpose AI models often see their frameworks become industry standards, shaping the direction of future developments, as is quickly becoming the case with the widespread use of Llama models in open development projects and industry application. These firms can then easily integrate advancements made by the community (for free) back into their own offerings, maintaining their competitive edge." #AI #AIsafety #AIActionSummit

My team spent this week evaluating the security of Deepseek’s R1 model. While open-source AI models offer transparency benefits, DeepSeek's safety remains questionable. Recent findings reveal significant vulnerabilities in DeepSeek R1, including susceptibility to jailbreak techniques, RCE, and potential data privacy issues. This team summarized some pros and cons of this model:    Pros ✅   - The group has an MIT license which empowers users with flexibility of local deployment and customization options, offering some control over data handling  - Potential for custom security measures: Users can implement additional security protocols when running the model locally  - The model offers weights for fine-tuning, allowing for tailored performance enhancements and security    Cons ❌   - Susceptible to exploits and lacks robust safeguards against malicious outputs. In fact, a recent incident exposed a DeepSeek database containing over a million lines of log streams, chat history, secret keys, and backend data   - The lack of transparency in training data raises critical questions about data integrity and potential biases  - Potential malicious Python code embedded within the model poses additional security threat as the model configuration requires trust_remote_code=True to be set, which increases the risk of arbitrary code/remote code execution    Remember, also, these models are brand-new, which adds risk.   Of course, we’re early in our evaluations. If this is helpful, I’ll keep sharing what we learn. #deepseek #AI #AISecurity #InformationSecurity #data #privacy

If according to Synopsis, 75% of code in codebases are open source software, then it is pertinent to understand how vulnerabilities in open source AI/ML tools can impact our ecosystem. I am thrilled to share with you my latest report, published with my team at AI Cyber Insights. The report titled "Industry Implications of Security Vulnerabilities in Open Source AI and ML Tools,” provides a comprehensive analysis of the security vulnerabilities identified in Open Source AI and ML tools from January to August 2024 . This report provides crucial insights for #AI professionals, developers, and industry leaders navigating the field of AI security. Here's a glimpse into the Report: ➡️ A disturbing upward trend in vulnerabilities, with a total of 176 vulnerabilities disclosed across various Open Source AI and ML tools in just eight months. ➡️ An alarming 75% of these vulnerabilities are classified as Critical or High severity, indicating a high potential for exploitation if left unaddressed. ➡️ Popular tools like MLflow, anything-llm, and lollms account for 40% of all reported issues, making them critical targets for security enhancements. ➡️ Remote Code Execution, Path Traversal, Privilege Escalation, and Server-Side Request Forgery are the most prevalent types of vulnerabilities. By understanding the nature and extent of these vulnerabilities, organizations can take proactive steps to secure their AI initiatives and protect against potential breaches. Shout out to my co-authors Isu Momodu Abdulrauf and Victoria Robinson for working closely with me to put this report together. Don't forget to comment what your thoughts are, on the report | tag your friends | Repost | Like #opensource #ml #artificialintelligence #cybersecurity #informationsecurity