Days after a deadly attack by the militant Islamic State (IS) on a Russian concert hall in March, a man clad in military fatigues and a helmet appeared in an online video, celebrating the assault in which more than 140 people were killed.

“The Islamic State delivered a strong blow to Russia with a bloody attack, the fiercest that hit it in years,” the man said in Arabic, according to the Site Intelligence Group, an organisation that tracks and analyses such online content.

But the man in the video, which the Thomson Reuters Foundation was not able to view independently, was not real — he was created using artificial intelligence, according to Site and other online researchers.

Federico Borgonovo, a researcher at the Royal United Services Institute, a London-based think tank, traced the AI-generated video to an IS supporter active in the group’s digital ecosystem.

This person had combined statements, bulletins, and data from IS’s official news outlet to create the video using AI, Borgonovo explained.

Although IS has been using AI for some time, Borgonovo said the video was an “exception to the rules” because the production quality was high even if the content was not as violent as in other online posts.

“It’s quite good for an AI product. But in terms of violence and the propaganda itself, it’s average,” he said, noting however that the video showed how IS supporters and affiliates can ramp up production of sympathetic content online.

Digital experts say groups like IS and far-right movements are increasingly using AI online and testing the limits of safety controls on social media platforms.

A January study by the Combating Terrorism Center at West Point said AI could be used to generate and distribute propaganda, to recruit using AI-powered chatbots, to carry out attacks using drones or other autonomous vehicles, and to launch cyber-attacks.

“Many assessments of AI risk, and even of generative AI risks specifically, only consider this particular problem in a cursory way,” said Stephane Baele, professor of international relations at UCLouvain in Belgium.

“Major AI firms, who genuinely engaged with the risks of their tools by publishing sometimes lengthy reports mapping them, pay scant attention to extremist and terrorist uses.”

Regulation governing AI is still being crafted around the world and pioneers of the technology have said they will strive to ensure it is safe and secure.

Tech giant Microsoft, for example, has developed a Responsible AI Standard that aims to base AI development on six principles including fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability.

In a special report earlier this year, Site Intelligence Group’s founder and executive director Rita Katz wrote that a range of actors from members of militant group Al Qaeda to neo-Nazi networks were capitalising on the technology.

“It’s hard to understate what a gift AI is for terrorists and extremist communities, for which media is lifeblood,” she wrote.

Chatbots and cartoons

At the height of its powers in 2014, IS claimed control over large parts of Syria and Iraq, imposing a reign of terror in the areas it controlled. Media was a prominent tool in the group’s arsenal, and online recruitment has long been vital to its operations.

Despite the collapse of its self-declared caliphate in 2017, its supporters and affiliates still preach their doctrine online and try to persuade people to join their ranks.

Last month, a security source told Reuters that France had identified a dozen ISIS-K handlers, based in countries around Afghanistan, who have a strong online presence and are trying to convince young men in European countries, who are interested in joining up with the group overseas, to instead carry out domestic attacks.

ISIS-K is a resurgent wing of IS, named after the historical region of Khorasan that included parts of Iran, Afghanistan, and Central Asia.

Analysts fear that AI may facilitate and automate the work of such online recruiters.

Daniel Siegel, an investigator at social media research firm Graphika, said his team came across chatbots that mimicked dead or incarcerated IS militants.

He told the Thomson Reuters Foundation that it was unclear if the source of the bots was the IS or its supporters, but the risk they posed was still real.

“Now [IS affiliates] can build these real relationships with bots that represent a potential future where a chatbot could encourage them to engage in kinetic violence,” Siegel said.

Siegel interacted with some of the bots as part of his research and he found their answers to be generic, but he said that could change as AI tech develops.

“One of the things I am worried about as well is how synthetic media will enable these groups to blend their content that previously existed in silos into our mainstream culture,” he added.

That is already happening: Graphika tracked videos of popular cartoon characters, like Rick and Morty and Peter Griffin, singing Islamic State anthems on different platforms.

“What this allows the group or sympathisers or affiliates to do is target specific audiences because they know that the regular consumers of Sponge Bob or Peter Griffin or Rick and Morty, will be fed that content through the algorithm,” Siegel said.

Exploiting prompts

Then there is the danger of IS supporters using AI tech to broaden their knowledge of illegal activities.

For its January study, researchers at the Combating Terrorism Center at Westpoint attempted to bypass the security guards of Large Language Models (LLMs) and extract information that could be exploited by malicious actors.

They crafted prompts that requested information on a range of activities from attack planning to recruitment and tactical learning, and the LLMs generated responses that were relevant half of the time.

In one example that they described as “alarming”, researchers asked an LLM to help them convince people to donate to IS.

“There, the model yielded very specific guidelines on how to conduct a fundraising campaign and even offered specific narratives and phrases to be used on social media,” the report said.

Joe Burton a professor of international security at Lancaster University, said companies were acting irresponsibly by rapidly releasing AI models as open-source tools.

He questioned the efficacy of LLMs’ safety protocols, adding that he was “not convinced” that regulators were equipped to enforce the testing and verification of these methods.

“The factor to consider here is how much we want to regulate, and whether that will stifle innovation,” Burton said.

“The markets, in my view, shouldn’t override safety and security, and I think — at the moment — that is what is happening.”