Price scraping is the process of using bots for illegal competitive price monitoring and tracking other valuable information related to pricing intelligence from e-commerce and travel sites.
Competitors employ this strategy to copy dynamic pricing information (which is one of the most important strategies used by e-commerce portals to influence consumer-buying decisions and optimize revenue) in real-time, so that they can attract price-sensitive buyers by setting their prices lower than baseline prices in the marketplace. While pricing information is generally available to consumers, price scrapers cross an ethical line by trying to undercut competitors’ pricing and growth strategies. Price scraping also results in skewed analytics, cart abandonment, and website slowdowns.
Scraping Tools and Third-Party Services
Competitors can use readily available technologies and tools like cURL, HTTrack, Scrapy, Selenium, Wget, PhantomJS and similar tools. Alternatively, they can employ third-party scrapers to target one or more websites and pull pricing and catalog information. Competitors can then thoroughly analyze their opponents’ pricing strategy and undercut them.
This is a particularly malicious activity that is on a constant rise in online marketplaces. A new player in the market can quickly get up to speed using these scraping services. This is because when the bots intrude the website, they gain access to Stock Keeping Units (SKU), product listings, pricing over time and the complete product catalog that has been compiled over the years. This gives a huge competitive edge for the new rival businesses. The scraper or competitor can automate this activity to such an extent that the website automatically reflects the best price upon analyzing prices from other sites.
Combating Web Scraping In Online Businesses
The Impact of Price Scraping
Price scraping is increasing with the growing competition and growth in online marketplaces. A new player in the market can quickly get up to speed using price and content scraping services. This is because when scraper bots attack a website, they potentially gain access to information on Stock Keeping Units (SKU), product listings, historical pricing, and their entire product catalog. This breadth of information can give new rivals a huge competitive edge. Competitors can automate their scraping activity to such an extent that their website automatically reflects the best price upon analyzing prices from competing sites.
In most cases, scraping is done to undermine the target website’s value proposition and competitive edge. If the target website has terms and conditions that prohibit scraping, the scraper or the competitor sending the scraper bots can be sued. Legally speaking, however, it can be hard to classify this as a criminal activity.
In fact, many price comparison sites directly scrape prices from a range of e-commerce sites to provide consumers with purchasing information, but they are tolerated as they are likely or expected to send a good number of referral customers to the sites they scrape from. E-commerce sites should allow legitimate scraping partners only for this purpose, by whitelisting them and giving them a certain allowable bandwidth for scraping. Unauthorized or illegal price scrapers should be strictly prohibited.
How a European Electronics Retailer Ended the Bot Menace on its Portal
How to Prevent Price Scraping
Price scraping is a considerable threat for e-commerce websites, as their pricing strategies over time are exposed to competitors and upstarts looking to undercut their pricing and marketing strategy. Existing technologies such as WAFs are not fully equipped to detect the newest generations of sophisticated scraper bots. Moreover, in-house bot management solutions have several downsides and similarly cannot keep pace with advanced botnets and their attacks methods. Businesses under regular scraping attacks are increasingly choosing to deploy a dedicated bot management solution to block bots that carry out scraping and other more serious attacks.