Meta and TikTok let harmful content rise after evidence outrage drove engagement, say whistleblowers

More than a dozen whistleblowers and insiders have laid bare how the companies took risks with safety on issues including violence, sexual blackmail and terrorism as they battled for users' attention.

"They sort of told us that it's because the stock price is down," the engineer said.

Decisions were being made to "maintain a strong relationship" with political figures to avoid threats of regulation or bans, not because of the risks to users, the TikTok staffer said.

The company invested in 700 staff to grow Reels, while safety teams were refused two specialist staff to deal with protecting children and 10 more to help with the integrity of elections, another former senior Meta employee said.

The algorithm offered content creators a "path that maximizes profits at the expense of their audience's wellbeing" and the "current set of financial incentives our algorithms create does not appear to be aligned with our mission" to bring the world closer together, according to one internal study.

In response to the whistleblowers' claims, Meta said: "Any suggestion that we deliberately amplify harmful content for financial gain is wrong." TikTok said these were "fabricated claims" and the company invested in technology that prevented harmful content from ever being viewed.

The algorithms are a "black box" whose internal workings are difficult to scrutinise, said Ruofan Ding, who worked as a machine-learning engineer building TikTok's recommendation engine from 2020 until 2024.

It was hard to build systems like this that were completely safe, he said. "We have no control of the deep-learning algorithm in itself."

The engineers do not pay much attention to the content of the posts, he said. "To us, all the content is just an ID, a different number."

He said they relied on the content safety teams to ensure that harmful posts were removed so they could not be promoted by the algorithm, comparing their relationship to different teams working on parts of a car.

"There's the team that are responsible for the acceleration, the engine, right? So we expect the team working on the braking system was doing a good job," he said.

But Ding said as TikTok tried to improve its algorithm almost on a weekly basis to gain more market share, he started seeing more "borderline" content or problematic posts that only appeared after users had been browsing videos for a while.

Borderline content is a term used within social media companies usually to describe posts which are harmful but legal - including misogynistic, racist and sexualised posts, as well as conspiracy theory content.

In one extreme case, another teenager, Calum - now 19 - said he had been "radicalised by algorithm" from the age of 14. The algorithm showed him content that outraged him and led to him adopt racist and misogynistic views, he said.

The videos "energised me, but not really in a good way," he said. "They just made me very kind of angry. It very much reflected the way I felt internally, that I was angry at the people around me."

Counter-terror police specialists in the UK who analyse thousands of posts on social media every year say they have seen the "normalisation" of antisemitic, racist, violent and far-right posts in recent months.

"People are more desensitised to real-world violence and they are not afraid to share their views," one officer said.

"If you're feeling guilty on a daily basis because of what you're instructed to do, at some point you can decide, should I say something? " said Nick.

The volume of cases they were assessing was too difficult to keep on top of to keep users safe, which left teenagers and children especially at risk, he added. Cuts and the reorganisation of some moderation teams - where some roles are being replaced by AI technology - have, in his view, limited the ability to deal effectively with this kind of content within the company.

Material linked to "terrorism, sexual violence, physical violence, abuse, trafficking" appears to be increasing, the whistleblower said.

The reality of what the app recommends, and the action taken against harmful content, is "very different in a lot of aspects to what the sites are saying" in public, he added.

In one example, a political figure who had been mocked by being compared to a chicken was prioritised over a 17-year-old who reported being the victim of cyber bullying and impersonation in France and a 16-year-old in Iraq who complained that sexualised images purporting to be of her were being shared on the app.

Referring to the Iraq case, Nick said: "If you look at the country where this report comes from, it's very high risk because it's a minor and it involves sexual blackmail and then you can see the priority here. The urgency is not high."

Nick also showed examples of where some posts encouraging people to join terror groups or commit crimes had not been labelled as top priority.

When the trust and safety team had asked to prioritise cases involving young people over political cases, the whistleblower said, they were told not to and to continue to deal with the cases according to the ranking they had been given.

Nick said the reason for the prioritisation was that - in his view - the company ultimately cares less about children's safety than it does about maintaining a "strong relationship" with politicians and governments, to avoid regulations or bans which would harm its business.

Nick said when he and other staff members raised some of these concerns with management, they were not receptive to it because they "are not being exposed to this content on a day-to-day basis".

The trust and safety employee's blunt advice to parents with children using TikTok is: "Delete it, keep them as far away as possible from the app for as long as possible. "

TikTok said it rejected the idea that political content is prioritised over the safety of young people and said the claim "fundamentally misrepresents the way their moderation systems operate".

The team Nick belongs to is part of a wider safety system which has multiple teams responsible for reviewing complaints about content. TikTok said: "Specialist workflows for certain issues do not result in the deprioritisation of child safety cases, which are handled by dedicated teams within parallel review structures."

A TikTok spokesperson said the criticisms "ignore the reality of how TikTok enables millions to discover new interests, find community, and supports a thriving creator economy".

The company said accounts for teens have more than 50 preset safety features and settings which are automatically turned on. It also said it invests in technology that helps prevent harmful content from ever being viewed, maintains strict recommendation policies and provides features for people to tailor their experiences.

His job between 2019 and 2023 involved "running large-scale experiments on sometimes as many as hundreds of millions of people" - who often had "no idea" this was happening - testing out how content was ranked in feeds.

"Meta's products are used by north of three billion people and the more time they can keep you on there, the more ads they sell, the more money they make. But it's very important that they get this stuff right, because when they don't, really bad things happen," he said.

When it came to Reels in particular, Motyl said the approach was to move as quickly as possible whatever the impact on users. He said there was a "common trade-off between protecting people from harmful content and engagement".

He said there was a "power imbalance" because safety staff had to get the agreement of teams in charge of Reels to introduce a new product or feature that would improve user safety. The Reels staff had "incentives to not let those products launch because toxic stuff gets more engagement than non-toxic," Motyl said.

"When he feels like there are potential competitive forces there's no amount of money that is too much," Silverman said.

A former engineer at Meta, who we are calling Tim, said that, as the company sought to compete with TikTok, more borderline harmful content was allowed on the site. His team had been focused on reducing this content, until the "business positioning" changed.

"You're losing to TikTok and therefore your stock price must suffer. People started becoming paranoid and reactive and they were like, let's just do whatever we can to catch up. Where can we get like 2%, 3% revenue for the next quarter?" Tim said.

He said that decision to stop limiting content that was possibly harmful but not illegal - and that users were engaging with - was made by a senior vice-president of Meta who Tim believed reported directly to Mark Zuckerberg.

The documents outline how sensitive content - which can include material that touches on people's moral beliefs or posts that incite violence - is more likely to trigger a reaction and engagement on the site, especially if it causes outrage.

"Given the disproportionate engagement, our algorithms presume that users like that content and want more of it," the study says.

Silverman said Meta's leadership seemed initially unsure how to respond to toxic content on the platform and there was a period of time when the company was "genuinely introspective".

But he said their stance "began to calcify into a sort of defensiveness". Their attitude was that "we're not responsible for all of polarisation in society", he said.

"Nobody's saying you're responsible for all polarisation. We're just saying you contribute to it, and probably in ways where you don't have to. If you just made a few changes, you might not contribute to it as much," Silverman said.

A Meta spokesperson denied the whistleblowers' claims. "The truth is, we have strict policies to protect users on our platforms and have made significant investments in safety and security over the last decade," the spokesperson said.

The company said it had "made real changes to protect teens online" including introducing a new Teen Accounts feature "with built-in protections and tools for parents to manage their teens' experiences".

Additional reporting by Robert Farquhar, Daisy Bata and Ez Roberts

Sign up for our Tech Decoded newsletter to follow the world's top tech stories and trends. Outside the UK? Sign up here.

Abbie says she felt "really lost" when her mum Jayne died of cancer while she was pregnant.

A group representing tech giants called government action against Anthropic a "temper tantrum".

The forum-style app has sparked interest by showing how AI bots interact without human involvement.

Meta's advisers say its methods for policing AI videos are inadequate, especially at times of crisis.