Warns of Chatbot 'Prompt Injection' Threat

---

 The UK's cybersecurity agency has issued a warning about the potential manipulation of chatbots by hackers, leading to serious real-world consequences. The National Cyber Security Centre (NCSC) highlights the increasing cybersecurity risks associated with "prompt injection" attacks. In these attacks, users input prompts designed to make language models – the technology powering chatbots – behave unexpectedly.

Chatbots, powered by artificial intelligence, respond to user prompts, simulating human-like conversations. These bots are trained through extensive data scraping and are commonly utilized in online banking and shopping for handling simple queries.

Large language models (LLMs), such as OpenAI’s ChatGPT and Google’s AI chatbot Bard, undergo training with data that generates human-like responses based on user prompts.

Given that chatbots facilitate the transfer of data to third-party applications and services, the NCSC has highlighted that the potential risks stemming from malicious prompt injections are set to increase.

For example, should a user input a statement or query unfamiliar to the language model, or if they manage to concoct word combinations that override the model's original programming or prompts, the user could induce unintended actions from the model.

Such inputs have the potential to trigger a chatbot to produce objectionable content or divulge confidential information within systems that accept unchecked input.

This year, Microsoft introduced a new iteration of its Bing search engine and a conversational bot powered by LLMs. A Stanford University student named Kevin Liu successfully executed a prompt injection to uncover the initial prompt used by Bing Chat.

By submitting a prompt requesting Bing Chat to "ignore previous instructions," Liu revealed the concealed prompt – a collection of statements dictating how the chatbot interacts with users – which is typically hidden from users.

Security researcher Johann Rehberger also discovered that he could compel ChatGPT to respond to new prompts through a third party, even if the original request hadn't specified this interaction.

Rehberger conducted a prompt injection using YouTube transcripts and identified that ChatGPT could access these transcripts, thus potentially creating more indirect vulnerabilities to prompt injection.

The NCSC asserts that prompt injection attacks can lead to tangible real-world repercussions if systems lack adequate security measures. The susceptibility of chatbots and the simplicity with which prompts can be manipulated could result in attacks, scams, and instances of data theft.

As the utilization of LLMs grows for transmitting data to external applications and services, the potential for risks stemming from malicious prompt injections is on the rise.

According to the NCSC, "Prompt injection and data poisoning attacks can be exceedingly challenging to identify and counteract. Nevertheless, no model operates in isolation. Thus, we can develop the entire system with security as a primary concern. By recognizing the risks linked to the machine learning component, we can structure the system to prevent the exploitation of vulnerabilities that might lead to catastrophic consequences. A straightforward example would involve implementing a rules-based system atop the ML model to hinder it from executing harmful actions, even when prompted."

The NCSC emphasizes that cybersecurity threats originating from artificial intelligence and machine learning, which expose systems to vulnerabilities, can be alleviated by designing with security in mind and comprehending the attack techniques that capitalize on "inherent vulnerabilities" within machine learning algorithms.