AI-Generated Music Platforms Manipulated to Produce Hateful Anthems

There is rising concern as malevolent individuals manipulate AI-powered music generation tools to craft songs filled with homophobia, racism, and incendiary propaganda. Further, they are distributing how-to guides to propagate this misuse.

As per intelligence from ActiveFence, an entity that handles online platforms’ trust and safety features, discussions on exploiting AI music tools for crafting bigoted songs have surged in “hate speech-associated” circles since March. These AI-composed tunes circulated in online spaces celebrate conflicts against various ethnic, gender, and religious groups, encourage violence and extremism, and foster hate towards minorities, according to a study by ActiveFence researchers.

Hostile and damaging musical compositions are not a modern creation. Nonetheless, experts fear that today’s accessible music-generating tools might enable mass production by individuals previously lacking the means or expertise, in analogy with how other generative media technologies have accelerated the diffusion of hate speech, along with misinformation and disinformation.

Noam Schwartz, the co-founder and CEO of ActiveFence, commented to TechCrunch, “These are advancing trends as more people discover methods to create and circulate these songs. Perpetrators swiftly identify specific weak spots to manipulate these platforms for generating malicious content.”

The Creation of “Hate” Tracks

AI music services like Udio and Suno allow compositions with custom lyrics. While these services implement measures to block explicit slurs, users have found ways to elude these restrictions, reports ActiveFence.

ActiveFence’s report illustrates strategies shared on white supremacist forums where users exchange phonetically spelt words, like “joo.Are AI music tools being exploited to produce hate speech?
z” for “Jews” and “say tan” for “Satan,” to outsmart content filters. There are also suggestions of altering terms pertaining to violence to bypass detection.

TechCrunch’s testing of these subversions on platforms Udio and Suno revealed varying levels of filtration success. While Udio blocked certain offensive terms, Suno did not prohibit any.

A Udio representative, by email to TechCrunch, reiterated their platform’s stance against hate speech utilization. Meanwhile, Suno did not offer a response.

ActiveFence discovered links in online spaces leading to AI songs echoing conspiracies against Jews, ones praising mass violence, songs promoting terrorist ideologies associated with ISIS and Al-Qaeda, and melodies glorifying violence against women.

Schwartz emphasizes the impactful nature of music, beyond mere text, in conveying emotion and serving as an influential tool for hate groups and political warfare. He points to historical events like Rock Against Communism, the notorious music gatherings from the late ’70s in the U.K. which bred racist and antisemitic music sub-genres.

Schwartz maintains that AI heightens the appeal of destructive content, stressing the ease with which harmful narratives can be transformed into catchy songs that garner widespread remembrance and recognition. Such content can promote cohesive hate-group dynamics, indoctrinate new members, and shock the general online community.

He contends that music generation platforms should execute stronger prevention measures and thorough safety assessments. Suggested solutions include red teaming to pre-emptively uncover system loopholes and implementing stringent moderation protocols.

However, as new methods to subvert moderation are regularly discovered, such solutions may only provide temporary relief. ActiveFence’s investigation found AI-generated music promoting terrorism using Arabic euphemisms and transliterations that went undetected due to weaker language-specific content filters.

Similar to other AI-generated digital content, AI-fueled hateful music may spread extensively. This was illustrated by a viral AI-altered clip of Adolf Hitler, which Wired reported garnered over 15 million views on a large platform after being posted by a widely recognized conspiracy theorist.

The problematic rise of AI-generated media containing racism, antisemitism, Islamophobia, and xenophobia was cited as concerning by various experts, including a UN advisory group.

“Generative AI platforms offer those lacking resources or artistic/technical proficiency, the means to construct and disseminate content that can vie for global audience attention,” said Schwartz. “And, malicious agents have, indeed, found creative avenues to evade moderation and maintain anonymity, with a high degree of success.”