prompt father: Prompt Fuzzer is an interactive tool designed to assess the security of GenAI application system prompts by simulating various dynamic LLM-based attacks. We assess security by analyzing the results of these simulations and allow users to enhance system prompts accordingly. This tool specifically customizes tests to the unique configuration and domain of your application. Fuzzer also features a Playground chat interface that allows users to iteratively adjust system prompts to increase resilience against a wide range of generative AI attacks. Users should note that using Prompt Fuzzer consumes tokens.
Garruk: Garak is a tool to assess whether your LLM is likely to fail in an undesirable way. Test for vulnerabilities including hallucinations, data leaks, instant injections, misinformation, toxicity generation, jailbreaks, and other potential weaknesses. Similar to nmap for network security, Garak is a diagnostic tool for LLM. It's available for free, and the developers are passionate about continually enhancing it to better support your applications.
Howi: This repository contains the source code for HouYi, a framework designed to automatically insert prompts and test for vulnerabilities in applications integrated with large-scale language models (LLMs). I am. Additionally, the repository includes demo scripts that simulate LLM integration applications and demonstrate how to deploy HouYi against such attacks. Users can apply HouYi to their real LLM integration applications by creating harnesses and defining attack intent.
Jailbreak LLM: Although there is increasing focus on aligning LLMs with human values, these models are susceptible to hostile jailbreaks that circumvent safety mechanisms. To address this, the Prompt Automatic Iterative Refinement (PAIR) algorithm was developed. Inspired by social engineering tactics, PAIR uses one LLM to automatically generate jailbreak prompts for another target LLM without the need for human assistance. PAIR can efficiently create jailbreaks by executing queries iteratively, often in less than 20 attempts. This method shows a high success rate and is effective against various LLMs such as GPT-3.5/4, Vicuna, and PaLM-2.
LLM attack: Recent efforts aim to moderate LLMs from producing objectionable content. The LLM Attacks method effectively prompts these models to produce undesired outputs. By automatically generating adversarial suffixes through a greedy gradient-based search, the process avoids the need for manual creation. These suffixes have proven transferable across multiple LLMs including ChatGPT, Bard, and Claude, as well as open source models such as LLaMA-2-Chat and Pythia. This advancement highlights significant vulnerabilities in LLM and emphasizes the need for strategies to counter such adversarial tactics.
prompt inject: Transformer-based LLMs like GPT-3 are widely used in customer-facing applications, but are still vulnerable to malicious interactions. In this work, we introduce his PROMPTINJECT, a framework for creating adversarial prompts through a mask-based iterative process. This research reveals how GPT-3 can go awry using simple hand-crafted inputs. We focus on two attack techniques: goal hijacking and prompt leak. Our findings reveal that even low-skilled attackers can exploit the probabilistic nature of GPT-3, which can pose significant long-tail risks to these models.
research framework: Recon-ng is a comprehensive reconnaissance framework for efficient web-based open source intelligence gathering. It has a user interface similar to the Metasploit framework and eases the learning process, but its purpose is different. Unlike other frameworks aimed at exploitation or social engineering, Recon-ng is specifically designed for reconnaissance. Metasploit must be used if you want to perform exploits, and Social-Engineer Toolkit is recommended for social engineering. Recon-ng supports a modular architecture, allowing Python developers to access and contribute. Users can refer to the wiki and development guide for starting points and details.
buster: Buster is a sophisticated OSINT tool that facilitates a variety of online investigations. Gravatar can retrieve social accounts linked to emails from various platforms such as About.me, Myspace, Skype, GitHub, LinkedIn, and from records of previous breaches. Buster also finds links to mentions of your email across Google, Twitter, dark web search engines, and paste sites. Additionally, identify email-related compromises, reveal domains registered in emails via reverse WHOIS, generate potential emails and usernames of individuals, and associate them with social media accounts or usernames. You can identify the emails that have been sent to you and reveal your personal work emails.
witness me: WitnessMe is an Eyewitness-inspired web inventory tool designed for extensibility and uses a backend-driven headless browser via the Pyppeteer library to enable custom functionality. The tool stands out for its ease of use with Python 3.7 and above, compatibility with Docker, and avoidance of installation dependencies. It supports extensive parsing of large Nessus and NMap XML files, provides CSV and HTML reporting, and has HTTP proxy support and a RESTful API for remote operations. WitnessMe includes a CLI to view scan results and is optimized for deployment to cloud platforms such as GCP Cloud Run and AWS ElasticBeanstalk. Additionally, it also provides signature scanning and device-based screenshot previews.
LLM Canary: The LLM Canary tool is an easy-to-access open-source security benchmark suite that allows developers to test, evaluate, and compare LLMs. This tool helps developers identify security tradeoffs when selecting models and address vulnerabilities before integration. It includes an OWASP Top 10 for LLM compliant testing group to keep you up to date with the latest threats. LLM Canary users can identify and assess potential vulnerabilities, run tests on multiple LLMs simultaneously for efficiency, compare results to benchmarks or previous tests, and perform comprehensive security assessments. You can design custom tests for
Pilot: Developed by the AI Red Team, PyRIT is a library designed to enhance robustness evaluation of LLM endpoints, targeting hazard categories such as fabrication, misuse, and prohibited content. The tool automates AI red teaming tasks, freeing up resources to tackle more complex problems and identifying security and privacy breaches such as malware generation and identity theft. It provides a benchmark for researchers to compare current model performance with future iterations and helps detect degradation. Microsoft uses PyRIT to improve product versions and meta prompts to better protect against prompt injection attacks.
LLMFuzzer: LLMFuzzer is an innovative open source fuzzing framework tailored for LLM and its API integration. This is ideal for security enthusiasts, penetration testers, and cyber security researchers looking to discover and exploit vulnerabilities in AI systems. The tool streamlines the testing process with features such as robust fuzzing, LLM API integration testing, various fuzzing strategies, and a modular design for easy expansion. Future enhancements include additional attacks, HTML report generation, diverse connectors and comparators, proxy support, side LLM monitoring, and autonomous attack modes.
prompt map: Prompt injection is a security vulnerability that allows a malicious prompt to manipulate a ChatGPT instance to perform unintended actions. The 'promptmap' tool automates testing for these attacks by analyzing the context and purpose of ChatGPT rules. Use system prompts to create customized attack prompts and test them on your ChatGPT instance. promptmap then evaluates the success of the prompt injection by analyzing the response from the ChatGPT instance. This tool helps identify and mitigate potential vulnerabilities by simulating real-world attack scenarios.
Gitreeks: Gitleaks is a static application security testing (SAST) tool designed to discover hardcoded secrets such as passwords, API keys, and tokens in Git repositories. Provides an easy interface to scan codes and find past and present secrets. Users can easily run Gitleaks locally using simple commands, providing details such as file location and author to identify sensitive information. Gitleaks can be installed via Homebrew, Docker, and Go, and binaries are available for a variety of platforms. We also support integration as a pre-commit hook or GitHub action to strengthen your security practices.
Cloud_enum: Multicloud OSINT tools are designed to identify public resources across AWS, Azure, and Google Cloud. For Amazon Web Services, you can enumerate open or secured S3 buckets and various awsapps such as WorkMail and WorkDocs. In Microsoft Azure, this tool can discover storage accounts and open blob storage containers, hosted databases, virtual machines, and web apps. Google Cloud Platform discovers open and secured GCP and Firebase buckets, Firebase real-time databases, Google App Engine sites, and Cloud Functions. This includes enumeration of projects and regions, and brute force of function names. It also identifies open Firebase apps.
Hello, my name is Adnan Hassan. I'm a consulting intern at Marktechpost and soon to be a management trainee at American Express. I am currently pursuing a dual degree at Indian Institute of Technology Kharagpur. I'm passionate about technology and want to create new products that make a difference.
