Skip to main content

Facebook discloses new measurement errors, continues to hone its math


Math is hard, even for Facebook. And it’s getting harder to trust Facebook’s math, even as the social network fine-tunes its calculations.

For the third time since September, Facebook is disclosing new measurement errors. The two new errors affected the reaction counts Facebook reports on Pages’ Live videos as well as the engagement figures Facebook reports for off-Facebook links; the latter link engagement metrics were recently used in investigations by BuzzFeed and The New York Times into fake news articles’ performance on Facebook.

In addition to acknowledging the two new errors — of which one has been corrected and one is still being inspected — Facebook has refined a measurement marketers may reference when buying ads through the social network.

None of the aforementioned metrics had any impact on how much money Facebook charges advertisers for their campaigns. But they may have informed brands’ Facebook ad-buying strategies as well as brands’, publishers’ and others’ Facebook-related content-publishing strategies.

Facebook should be given credit for being upfront about its mistakes and rectifying its errors. But the more measurement errors and corrections it discloses, the more difficult it becomes to trust Facebook’s measurements. It’s a dilemma that some brands and agencies have been wrestling with following the previous two measurement error disclosures. And it’s one that may not subside until Facebook allows independent firms to directly measure these previously faulty stats, rather than relying on Facebook for the raw, corrected data.

Okay, now for the details of what’s up with Facebook’s math this time.

Inconsistent engagement stats for off-Facebook links

Facebook's Graph API and mobile app report mismatched engagement rates for outside links.

Facebook’s Graph API and mobile app report mismatched engagement rates for outside links.

Facebook has confirmed a discrepancy exists between two measures of the Facebook engagement rates for outside links, but the company has not been able to identify the reason for the measurement anomaly.

I brought the error to Facebook’s attention a few weeks ago after I came across it while analyzing BuzzFeed’s recent investigation of fake news links’ performance on Facebook. (At the bottom of this post, I’ll get into how I found the error and how these stats can be accessed for any URL by anyone with a Facebook account).

The two metrics in question are: 1) the total number of shares, likes and comments a link has received on and off Facebook, reported through Facebook’s Graph API, and 2) the number of people who have shared or commented on that link, reported through search results in Facebook’s mobile apps.

The problem with those metrics: For some links — including ones examined by BuzzFeed — Facebook reported that the number of people who shared or commented on the link exceeded the sum of shares, likes and comments for that link. That shouldn’t happen since the shares and comments stat reported in the mobile search results is a subset of the shares, likes and comments stat reported by the Graph API, a Facebook spokesperson confirmed. It’s unclear whether the inconsistent stats mean that Facebook’s Graph API is over-counting engagements or its mobile search results are under-counting them.

Once Facebook determines why the measurement discrepancy exists, the company will notify people that its cause has been identified and the error has been corrected.

Misallocated reactions on Live videos

Facebook incorrectly reported the source of reactions on Pages' Live videos.

Facebook incorrectly reported the source of reactions on Pages’ Live videos.

Facebook has been incorrectly reporting the number of times people “reacted” to a Live video by clicking the like button or one of the corresponding “reactions” emojis while watching the live stream. To be clear, the total number of reactions, which appears publicly below a Live video post, has been accurate, but the breakdown of where those reactions occurred, which is shown only to Pages, has not been.

Within Facebook’s Page Insights analytics tools, Pages are able to see the number of reactions generated from the Live video broken down by whether the reactions happened while people were watching the live stream — “reactions on post” — or whether they were received on others’ posts that included the Live video — “reaction on shares” — such as when someone shares, or Facebook-retweeted, a Page’s Live video post to their friends and those friends clicked the like/reaction button on the friend’s post after the live stream had already ended.

The “reactions on post” number was supposed to only measure one reaction per individual live viewer, and it did. But people may react multiple times to a live video, such as by clicking “haha” each time the stream makes them laugh. Facebook incorrectly included those extra reactions to the “reactions on shares” figure, even though they weren’t reactions on shared posts.

In addition to correcting the reactions misallocation, Facebook’s “reactions on post” metric will no longer count only one reaction per viewer. Instead it will also count those extra reactions that had been incorrectly counted as “reactions on shares.” Facebook will no longer offer a measurement of viewers that reacted to a live stream at least once, according to a Facebook spokesperson. As a result, any Pages that had been tracking the “reactions on post” metric for their past live streams might as well do away with that metric’s historical data. The comparison will no longer be apples to apples, but instead apples to watermelons.

The revised Live reaction breakdowns will be applied to new Live videos beginning in mid-December. They are expected to result in a 500% average increase in the number of “reactions on post” and a 25% average decrease in the number of “reactions on shares,” according to Facebook.

Ads’ estimated reach becoming less of a guesstimate

Facebook is making its estimates of ads' potential reach more precise.

Facebook is making its estimates of ads’ potential reach more precise.

Facebook is honing how it estimates the number of people an advertiser’s campaign will reach.

Before I explain how these metrics are changing, let me be clear about what the metrics are and are not. The estimated total reach and estimated daily reach figures are hypotheses for how many people may be exposed to an advertiser’s campaign. They are not how many people were, in fact, exposed to the campaign. These estimates, presented when a brand is in the process of creating a campaign using Facebook’s ad creation tool and before the ad buy has been ordered, are supposed to help an advertiser judge how defined its target audience should be and how that targeting affects the number of people who may see its ad.

These estimates will remain an estimate, but Facebook is making them less of a guesstimate.

Previously Facebook took an advertiser’s defined target audience, looked at the behavior for a smaller sample of people in that audience — like how likely they were to check Facebook, Instagram or a site or app in Facebook’s Audience Network ad network on a given day — and projected that behavior for everyone in that audience. The process bore similarities to an election poll that takes a representative sample of expected voters to forecast how the rest of the country may vote. To make its estimates more precise, Facebook will rely less on these projections. Instead it will rely on what it knows about the behavior for the full target audience and use that more complete view to gauge a campaign’s potential reach.

As a result of the methodology change, Facebook’s audience estimates may fluctuate by less than 10% from the prognostications it had previously provided advertisers, according to the company.

The backstory on Facebook’s link engagements discrepancy
Like I mentioned above, I came across the error between Facebook’s two link engagement metrics when I wanted to take a closer look at BuzzFeed’s fake news investigation.

BuzzFeed evaluated the fake news articles’ performance based on the aggregate number of shares, likes and comments that the links to those articles received on Facebook. BuzzFeed had looked at all of the engagement categories combined, and I wanted to see if there was a way to zero in on the articles’ share counts. I figured checking the share counts specifically could signal how the news feed algorithm change Facebook announced in June, which prioritized sharing for organic Page post distribution, may have impacted the spread of fake news articles on Facebook.

BuzzFeed used a third-party analytics tool, BuzzSumo, to access and collect these stats, but the figures are actually open to anyone with a Facebook account. All you have to do is paste a link URL into Facebook’s Graph API Explorer tool, click submit and Facebook will spit out the total number of shares, likes and comments for that link, including the number of people pasting the link into a status update posted to Facebook. (Confusingly the resulting sum is categorized as “share_count,” but Facebook’s Graph API documentation defines the stat as the total number of shares, likes and comments for that link).

But the Graph API would only give me updated versions of the figures BuzzFeed already cited, and I wanted to isolate the share count to see what percentage of those total engagements may have carried more weight with Facebook’s algorithm. Luckily an article published by The New York Times — coincidentally also about fake news — showed me I could access that stat, or what seemed like a close proxy. The New York Times article showed that you can paste a link URL into the search bar in Facebook’s mobile app and Facebook would show you the number of people who have shared that link. A Facebook spokesperson later clarified that this stat is measure of how many people shared the link on Facebook or commented on posts that included the link.

For 20 fake news links and 20 real news links from BuzzFeed’s investigation — the August to Election Day groups — I pulled the total number of engagements from Facebook’s Graph API and the total number of people who shared the link from Facebook’s mobile search results. And that’s when I saw something weird. For three of those 40 links — to NBC News, Breitbart and Liberty News — Facebook said that more people had shared or commented them than had shared or liked or commented on them. And the discrepancies weren’t small. At the time I checked, the NBC News link received 407,708 shares and comments , per the mobile search results, but 237,881 shares and comments and likes, per the Graph API results.

Since Facebook is still figuring out why there’s a deviation between the link engagement counts accessed through the Graph API and the ones accessed through mobile search, it’s hard to say which of the two stats are flawed, if not both.

via Marketing Land


Popular posts from this blog

How to Get SMS Alerts for Gmail via Twitter

How do you get SMS notifications on your mobile phone for important emails in your Gmail? Google doesn’t support text notifications for their email service but Twitter does. If we can figure out a way to connect our Twitter and Gmail accounts, the Gmail notifications can arrive as text on our mobile via Twitter. Let me explain:Twitter allows you to follow any @user via a simple SMS. They provide short codes for all countries (see list) and if you text FOLLOW to this shortcode following by the  username, any tweets from that user will arrive in your phone as text notifications. For instance, if you are in the US, you can tweet FOLLOW labnol to 40404 to get my tweets as text messages. Similarly, users in India can text FOLLOW labnol to 9248948837 to get the tweets via SMS.The short code service of Twitter can act as a Gmail SMS notifier. You create a new Twitter account, set the privacy to private and this account will send a tweet when you get a new email in Gmail. Follow this account …

Another SEO tool drops the word “SEO”

This guest post is by Majestic’s Marketing Director, Dixon Jones, who explains the reasons for their recent name change.
Majestic, the link intelligence database that many SEOs have come to use on a daily basis, has dropped the “SEO” from it’s brand and from its domain name, to become Since most people won’t have used Google’s site migration tool before, here’s what it looks like once you press the “go” button:

In actual fact – there’s a minor bug in the tool. The address change is to the https version of (which GWT makes us register as a separate site) but that message incorrectly omits that. Fortunately, elsewhere in GWT its clear the omission is on Google’s side, not a typo from the SEO. It is most likely that the migration tool was developed before the need for Google to have separate verification codes for http and https versions of the site.
The hidden costs of a name change
There were a few “nay sayers” on Twitter upset that Majestic might be deserting it…

6 types of negative SEO to watch out for

The threat of negative SEO is remote but daunting. How easy is it to for a competitor to ruin your rankings, and how do you protect your site? But before we start, let’s make sure we’re clear on what negative SEO is, and what it definitely isn’t.Negative SEO is a set of activities aimed at lowering a competitor’s rankings in search results. These activities are more often off-page (e.g., building unnatural links to the site or scraping and reposting its content); but in some cases, they may also involve hacking the site and modifying its content.Negative SEO isn’t the most likely explanation for a sudden ranking drop. Before you decide someone may be deliberately hurting your rankings, factor out the more common reasons for ranking drops. You’ll find a comprehensive list here.Negative off-page SEOThis kind of negative SEO targets the site without internally interfering with it. Here are the most common shapes negative off-page SEO can take.Link farmsOne or two spammy links likely won’…