Jan 12, 2025

An Engineering View of the Meta “Hateful Conduct” Changes

TL;DR: I helped build Hate Speech detectors at Meta when I was there. Here are my thoughts on the roll-out of these changes, and what will be hard for them.

I worked at Meta for 6.5 years, and as a software engineer in the Integrity group for my final 3 years, from July 2019 through July 2022. By the time I left, I was tech lead for the ~200-person Attack Vectors team there. I was really proud of the work we did in Integrity: keeping fake users off the platform, removing malicious content, and adapting to new types of threats. This was all very important work and I found my colleagues there to be extremely intelligent, engaged, and concerned about keeping the platform safe. I don’t have up-to-date context on changes to the system so some of this is speculation, but since I left there haven’t been any big changes in the public information that is up on the web site.

Last Wednesday, Meta announced major changes in how it would approach Hate Speech (newly renamed to Hateful Conduct). They updated their posted guidelines with about 80 changes (they have been heavily discussed elsewhere, and you can see them for yourself here). Some key points are removing restrictions such as slurs, “sex- or gender-exclusive language,” religion-based comments, and most kinds of cursing. I worked at Meta on their hate-speech detection systems and here are what I think are going to be their big issues going forward.

Legal/Regulatory (though IANAL, and the rest of this post is focused on technical stuff):

There is only one set of these policies for the whole world. By enforcing rules targeting the US market on a global basis, Meta will have a lot of problems in Europe — the UK has already pushed back on this, though Zuckerberg has implied that he will work with Trump to change the EU rules. (Not yet, though. At the end of the policy document, there is even a new section saying, “Note: if you are a European Union user, and are seeing content that you believe violates hate speech laws within your country, you can submit a legal removal request to Facebook or Instagram.”
There is a way for other countries’ laws against content to be enforced through content-restriction, but that relies on things like court orders and GDPR requests.

Technical:

The big automated systems are not trained for each market, so by default enforce similar rules no matter where someone is posting or viewing their feed.
Automated enforcement of these changes will be patchy for some time, as all the detection systems are updated. There are multiple ways that violating content is detected, and most of them will have to be changed (ML, regular expressions, content similarity metrics, etc).
Many of the automated systems make it easy to add new restrictions, as opposed to removing them, as is the case here.
Meta will probably be relying on multimodal large-language models to override the production systems and scale up these changes as quickly as possible.

When I worked at Facebook, even small policy changes to Community Standards were approached very cautiously, requiring many months of Policy, Legal, and User Research work before they could be approved by leadership. After they made a decision, it took a lot of time just to prepare training material for human reviewers and roll out those changes to the contractors in the reviewing teams worldwide. Last week‘s major changes were released only two months after the elections. They were either in the works for some time, or were more likely rushed out, and the teams will be under enormous pressure to implement them quickly.

A classical violating-content detection and enforcement system looks like the one below. Meta uses a much bigger version of this that has mostly the same high-level blocks but a lot more detail inside each of them:

Before we go further, note that:

The heavy lifting is handled by the ML classifier(s) (yellow rectangle), which is how it all scales up globally. Here’s an older post on the types of classifiers that Meta uses for this. Something like these classifiers is probably still used at the core, simply because old-school classifiers are much less expensive and much faster to run at global scale than the amazing new models that are coming out.
Content is generally only reviewed by humans (blue rectangle) when users disagree with the classifier through either reporting violating content, or by the creator appealing the decision (human reviewers are far too busy to look at borderline cases). This is where almost all the ground-truth Training Data comes from.
Only the human reviewers are trained on the policies (purple rectangle). ML classifiers implicitly learn them from the training data that comes from human reviews. When a policy gets changed, it takes a while (in Meta’s case, they have publicly stated several months) for the new rules to be picked up implicitly in training the next version of the ML classifier, as more examples come in, enforcing the new preferences. In the meantime, the burden of the implementation of the policy changes is on the human reviewers.
Human review is hard and awful but necessary, both to handle appeals/reports, and to train the AIs. At least 15,000 people do it at Meta, and they are mostly contractors in emerging markets, who are constantly at risk of burnout, PTSD, etc.

Below, I’ve added to that diagram the critical piece (indicated here in gold) that Meta will need to be able to roll these changes out. At Meta, I was a tech lead on the work to turn it from internal research into a production system. Even if it’s not exactly the same (GPTs and LLMs have evolved a lot since 2022, and Meta’s Llama is quite good at multimodal), they will need to use similar technology to roll out these changes quickly. This system is called “Few-Shot Learner” here (even with a glitzy video), and is based on this research paper.

The original plan was to directly copy/paste the policy change document and feed it into the “Few-Shot Learner” to over-rule the automatic decisions that are taken by the old-school classifiers, and thus reduce the roll-out time from months to weeks, as well as taking a burden off of human reviewers. In practice, we had to do a lot of prompt engineering (before that was a thing) to take the policy documents and phrase them in a way that the model could understand. Since it was too expensive to run the system on the entire Meta feed, we used regular-expression and simpler ML systems to perform a first pass on data to collect the “few shot examples” that the Few-Shot Learner would use. These examples needed to be human-reviewed (always difficult to get that allocated) to make sure they were compliant with the policy.

Over a few rounds of this, we would roll it out on a limited basis, and then progressively upgrade the model until it became a fully scalable version. Due to more automated detection, more posts can be actioned on, which leads to more ground truth data for the production classifiers to train on. After a few months, once the policy changes had been fully integrated into the simpler production classifiers, that configuration of the few-shot learner could be retired, since it would no longer be needed. Below is Meta’s diagram of the process, from their web site.

I can only imagine what it’s like in the teams implementing these changes now amidst all the other recent news at Meta. I’m a bit sad to see all the recent news, and I’m especially worried about emerging problems that look like they will be even harder to stop:

Highly targeted phishing at scale through LLMs.
High-quality fake news and fake engagement customized for each micro-demographic.
Amplification of conspiracy theories and hate speech designed to tear us apart rather than come together.
The overall ongoing enshittification of the Internet as all of these things drive off genuine discussion among actual people.