Late last year, Methbot, a sophisticated bot network responsible for $3 to 5 million of ad fraud losses on a daily basis, captivated the industry. The news revived interest in fighting against invalid traffic, which some estimates say accounts for over 40% of all traffic on the web.
Invalid traffic is disastrous for marketers—it wastes significant portions of ad budgets on impressions where no human even has an opportunity to see an ad, let alone make a purchase. Marketers will waste an estimated $6.5 billion in advertising spend on invalid traffic in 2017 alone.
What is invalid traffic?
The Media Rating Council defines invalid traffic as “traffic that does not meet certain ad serving quality or completeness criteria, or otherwise does not represent legitimate ad traffic that could be included in measurement counts.” The two general categories of invalid traffic are:
General invalid traffic (GIVT), which consists of traffic that can be caught through routine means of filtration, such as lists and standardized parameter checks. GIVT can be accidental and non-malicious, and generally identifies itself proactively. Examples include:
- Known data center traffic (a consistent source of NHT or non-human traffic)
- Search engine spiders and crawlers
- Non-browser user-agents (as identified by the user-agent string in the header of a request)
- Pre-fetch or browser pre-rendering traffic
Sophisticated invalid traffic (SIVT), which is more difficult to detect and requires advanced analytics or significant human intervention to analyze and identify. Examples include:
- Bots or scrapers masquerading as legitimate consumsers
- Hijacked devices (for example, a device that’s altered to make non-user-generated ad requests), hijacked sessions within hijacked devices, hijacked ad tags, and hijacked creative
- Hidden, stacked, covered, or otherwise intentionally obfuscated ads
- Invalid proxy traffic
- Adware and malware
- Incentivized manipulation of measurements
- Misappropriated content and falsely represented sites or impressions
- Cookie stuffing, recycling, or harvesting
- Manipulation or falsification of location data
While much focus is placed on SIVT, any attempts media owners make to limit invalid traffic should eliminate GIVT as well.
Invalid traffic implications for media owners
Though media owners could ostensibly benefit from the inflated revenue caused by invalid traffic in the short term, SIVT lowers the value of legitimate impressions in the long term and will become a huge liability as the industry takes steps to eliminate it.
Services like HUMAN (formerly known as White Ops), Integral Ad Science, and MOAT monitor and report on SIVT across broad swaths of inventory. Marketers are responding by scrutinizing where ad spend is allocated and by applying strict standards based on ad campaign performance. Media owners with significant amounts of invalid traffic will perform poorly on downstream metrics and risk devaluing their inventory over time.
Four habits of highly successful media owners
In 2015, HUMAN analyzed members of Digital Content Next, a trade organization of high-quality digital content companies, to identify practices that have proven particularly successful in limiting levels of SIVT. Their findings uncovered these four best practices that media owners can follow to limit invalid traffic.
- Strictly vet traffic from third parties or opt out of sourced traffic. While traffic sourcing from third parties can result in high bot percentages, media owners that restrict traffic purchases to top-line recommenders can do so without seeing increased bot traffic.
Media owners can safely source traffic while maintaining high levels of valid impressions in their inventory. Three of the top five DCN media owners with the lowest sophisticated bot percentages sourced traffic from large suppliers and maintained sophisticated bot percentages below 2%, with general bot percentages of 0.5 to 1.3%.
Another well-known media owner highlighted carefully vetted sourced traffic options and found that the available inventory didn’t meet quality standards. This media owner opted not to source traffic from third parties, and had a sophisticated bot rate of 2.6%, far better than average. - Don’t use viewability as a sole measure of whether an impression was caused by a bot. Viewability and fraud are two very different metrics. Two top-volume media owners with bot rates below the average of the other 32 DCN members analyzed did not bill based solely on viewability. Managing all factors, including viewability, with a strong foundation of policies that limit the ingress of bots and other inventory defects through all channels helps to maintain high inventory quality.
- Maintain visibility and control over your inventory. Inventory classified by media type, such as video, had bot percentages well below the average of unclassified inventory. This may indicate that inventory that is easier to tag and monitor is also the easiest to manage for quality factors such as humanity percentages.
- Use retargeting carefully. High-volume media owners that did not retarget site visitors had an average sophisticated bot percentage of 1.8%, compared to an average of 2.7% for similar media owners who do retarget site visitors.
Limiting SIVT at the exchange level
In order for the industry to seriously address ad fraud, invalid traffic must be identified and filtered out at every stage of the supply chain. At Index Exchange, we limit SIVT by screening out media owners that do not meet a standardized list of criteria. As it turns out, high-quality media owners tend to share the following qualities:
- Engaged viewer base: High bounce rates are an indicator that a media owner’s inventory isn’t premium and that its traffic may not be legitimate. Visitors should be checking out multiple parts of the site or platform, not just one page then leaving quickly.
- Modern-looking site: Most premium media owners have transitioned to the more efficient, minimalist visual style of Web 2.0 and take pride in the presentation and organization of their own content.
- Seamless ad experience: A refined ad experience means the media owner doesn’t feature disruptive ads on their pages or platforms, making it as easy as possible for consumers to enjoy their content.
- Reasonable ad quantity: Generally speaking, websites should have no more than five ads per page. Additional ads can run the risk of distracting from site content and devaluing the site’s inventory.
- Explicit permission: Media owners should leverage initiatives like ads.txt to make clear which exchanges are authorized to sell their inventory and which are not.
The media owners featured in this report all have exceptionally low SIVT according to data from HUMAN, and they’ve shared their own strategies for identifying and eliminating it. Marketers will want to consider these strategies when vetting media owners and the exchanges that work with them.
Case study: Canadian Broadcasting Corporation (CBC)
Despite being one of the largest media properties in North America, CBC and its broadcast service, Radio Canada, have SIVT levels far below the industry average. With an SIVT level consistently below 2% for their desktop display inventory, according to data from HUMAN, the CBC has done an excellent job of making sure the traffic to their sites is legitimate.
Jeff MacPherson, director of monetization platforms and services at CBC, has a simple explanation for these low SIVT numbers: The CBC doesn’t purchase traffic merely to meet sales goals. “There’s no good way to buy traffic,” he explains. “You just have to live by the merit of your content. If that content works, you won’t need to buy traffic.”
MacPherson raises a significant concern about SIVT for purchased traffic. Traffic that is purchased or sourced through a third party has a significantly higher incidence of non-human traffic than “organic” traffic.
Rather than purchase traffic for promotional purposes, the CBC leverages their agency of record and agency trade desk for standard ad buys, strategic social media campaigns, and multi-platform messaging across their owned and operated sites.
To detect invalid traffic, the CBC employs a team internally that monitors system health, including suspicious traffic. When suspicious activity is identified, the user agent and IP are blocked. CBC also employs trusted technology solutions at key stages in the programmatic supply chain. Each tech partner measures and detects invalid traffic and entrusts Google Ad Manager to filter a significant portion out. Media owners will want to devote significant resources and forge similar relationships to take ad fraud seriously.
Case study: Publishers Clearing House (PCH)
Leading interactive media and digital entertainment company PCH markets merchandise and magazine subscriptions through sweepstakes and prized-based games on their sites. As such, it is a business imperative that the consumers who visit their site are human and engaged.
PCH has successfully reduced the prevalence of SIVT on its sites to more than 80% below the industry average for desktop display, according to Index Exchange data measured by HUMAN. To achieve this, PCH put measures in place to carefully vet the consumers who view their content.
“We restrict any advertiser-supported content to registered users only,” said Sal Tripi, AVP of digital operations and compliance at PCH. “Consumers must validate personal information such as email and postal address before being exposed to advertising.” Not only does this approach ensure that viewers of PCH’s content are legitimate and engaged, but it gives PCH first-party data they can use to target ads appropriately.
PCH also employs IP address blocklisting, keeping visitors outside their target geo from visiting their sites and blocking IP addresses they’ve identified as distributing invalid traffic. PCH has at least three industry-leading tools active on their sites to monitor, track, and act upon invalid traffic at any one time. Such commitment to limiting invalid traffic has earned PCH the Certified Against Fraud seal from the Trustworthy Accountability Group.
Marketers seek out media owners who eliminate invalid traffic
Though the ad industry has moved past the particular crisis created by Methbot, it’s a reminder that we must remain vigilant in recognizing and eliminating invalid traffic. Marketers will continue to take steps to combat bad practices and incentivize clean inventory by being selective with their ad spend, demanding ad fraud data, and partnering with media owners and exchanges actively engaged in fighting invalid traffic.
