Twitter AI Pentagon explosion fake speaks to broader problems on the platform

The new system also means that media shared by accounts with a blue tick is ranked higher in Twitter’s algorithms, meaning that entire networks of misinformation spreaders can get synthetic media like this into more people’s feeds by virtue of their paying $13 a month for Twitter Blue.

Loading

“I think that this particular image was algorithmically boosted by accounts that had bought verification. In essence it’s easier now, much easier, to have misinformation go viral on Twitter,” Barnet said.

“It’s somewhat ironic that [Musk] is one of the people out there saying that we should be worried about AI, but the system on the platform he bought that was there to deal with misinformation is one of the first things he dismantled.”

Musk and Twitter have been on a campaign to bring advertisers back to the platform after the verification changes and other new initiatives prompted an exodus over the past six months. Organisations can now buy special gold ticks to prove their legitimacy, as well as custom logos to indicate that accounts belong to a certain organisation. Musk has also appointed proven advertising boss Linda Yaccarino as Twitter’s next chief.

At the same time, Musk has also begun positioning Twitter as a town square for discussion of the 2024 US election, inviting Florida Governor Ron DeSantis to announce his candidacy via Twitter Spaces last week, and announcing that all candidates are very welcome on Twitter.

Loading

As of now, there are still verified users on Twitter claiming the 2020 US election was fraudulently stolen by the Biden campaign, despite Musk promising earlier this month these claims would be eliminated.

Twitter was contacted for comment, and replied with a poop emoji.

Barnet said that even though synthetic media such as the faked pentagon photo was nothing new, the rise of generative AI tools that could create content from text prompts meant it could be created more quickly and be more specific.

“To generate misinformation in the past, you needed to know how to use Photoshop. Or if you were generating written text, you’d want to sound as much like a journalist as you possibly could,” she said. “And this is all available now at the click of a button.”

Loading

Companies including OpenAI and Microsoft include protections in their products that are designed to prevent dangerous media being created. This includes watermarking images, or refusing to comply with requests including certain words or themes. Additionally, social media platforms including Twitter have the capacity to recognise and label fake images, although in the case of the Pentagon picture, Twitter didn’t do so until hours after the fact.

But none of these measures are foolproof, and Barnet said it would likely fall to users to think before they engaged.

“I don’t know if any one thing can stop it. In the meantime, I’ve been telling people, particularly since verification was watered down on Twitter, that they’re going to have to manually look for the markers of authenticity. The markers that indicate it’s probably been fact-checked.”

Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.

The new system also means that media shared by accounts with a blue tick is ranked higher in Twitter’s algorithms, meaning that entire networks of misinformation spreaders can get synthetic media like this into more people’s feeds by virtue of their paying $13 a month for Twitter Blue.

Loading

“I think that this particular image was algorithmically boosted by accounts that had bought verification. In essence it’s easier now, much easier, to have misinformation go viral on Twitter,” Barnet said.

“It’s somewhat ironic that [Musk] is one of the people out there saying that we should be worried about AI, but the system on the platform he bought that was there to deal with misinformation is one of the first things he dismantled.”

Musk and Twitter have been on a campaign to bring advertisers back to the platform after the verification changes and other new initiatives prompted an exodus over the past six months. Organisations can now buy special gold ticks to prove their legitimacy, as well as custom logos to indicate that accounts belong to a certain organisation. Musk has also appointed proven advertising boss Linda Yaccarino as Twitter’s next chief.

At the same time, Musk has also begun positioning Twitter as a town square for discussion of the 2024 US election, inviting Florida Governor Ron DeSantis to announce his candidacy via Twitter Spaces last week, and announcing that all candidates are very welcome on Twitter.

Loading

As of now, there are still verified users on Twitter claiming the 2020 US election was fraudulently stolen by the Biden campaign, despite Musk promising earlier this month these claims would be eliminated.

Twitter was contacted for comment, and replied with a poop emoji.

Barnet said that even though synthetic media such as the faked pentagon photo was nothing new, the rise of generative AI tools that could create content from text prompts meant it could be created more quickly and be more specific.

“To generate misinformation in the past, you needed to know how to use Photoshop. Or if you were generating written text, you’d want to sound as much like a journalist as you possibly could,” she said. “And this is all available now at the click of a button.”

Loading

Companies including OpenAI and Microsoft include protections in their products that are designed to prevent dangerous media being created. This includes watermarking images, or refusing to comply with requests including certain words or themes. Additionally, social media platforms including Twitter have the capacity to recognise and label fake images, although in the case of the Pentagon picture, Twitter didn’t do so until hours after the fact.

But none of these measures are foolproof, and Barnet said it would likely fall to users to think before they engaged.

“I don’t know if any one thing can stop it. In the meantime, I’ve been telling people, particularly since verification was watered down on Twitter, that they’re going to have to manually look for the markers of authenticity. The markers that indicate it’s probably been fact-checked.”

Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.