The web with its HTTP protocol has become the prominent conduit for users accessing almost any service or application over the Internet, from casual browsing to online banking and critical enterprise applications accessed by employees. Consequently, web traffic is of strategic importance for every organization.
Inspecting and analyzing web traffic provides an excellent insight into customer behavior, preferences and experience bringing a major contribution to the Business Intelligence (BI) of an organization. Furthermore, having this insight in real timeenables immediate online actions to be taken based on it.
Value Proposition and Use Cases
The value in real time actionable traffic analysis is vast, although often overlooked. It spans multiple domains and departments in an organization. Here are just some typical examples:
- Marketing: increase conversion rate and boost repeat sales by offering customers the right promotion at the right time. The promotion or campaign is suited to individual users based on his or her observed behavior.
- Customer care: monitor customer experience and offer effective support for users that have difficulties completing business transactions.
- IT: monitor the system performance from the user perspective and identify problems ahead of time.
- Security: detect fraud patterns in user behavior and consequently block fraudulent users.
However, for all this to happen at low cost and in short time, some guidelines and best practices should be followed. Listed below are the four best practices that have proven to be the most important and the most effective.
Best Practice 1: Separate Web Traffic Analysis from Web Application
Web traffic analysis can be done on the web application path, either on the web server, on a reverse proxy or even on the client browser by enriching the web pages with special purpose scripts. The problem with these methods is that they interfere with the application, require long staged deployment cycles and often compromise the stability and performance of these critical applications. Often the team responsible for the analysis is not the same as the one responsible for the application thus creating unnecessary dependency and even conflicts.
Since traffic analysis is not part of the web application, the best way is to do it out-of-path, that is, by passively capturing the data on a network switch without interfering with or withholding the packets on their way to and from the web server. This way, one can eliminate all of the above problems and allow robust agile deployment of analysis solutions. Special attention should be given to the encrypted HTTPS traffic found in almost any site: the passive capturing needs to have the organization’s key pairs to be able to decrypt the traffic.
Best Practice 2: Use of a Flexible Scripting Environment
Web applications have no fixed “standard” for performing transaction. For example, the web page for placing an item in the shopping cart varies from one e-shop to another, and can change over time even in the same site. Analysis solutions must incorporate flexible, yet an easy to use scripting environment that can customize the extraction, the events and the transactions from raw HTTP and HTML traffic to the needs of the specific site.
Best Practice 3: Use of a Complex Event Processing (CEP) System
Extracting web events by themselves is not sufficient. The intelligence lies behind thepatterns of events. For example, a user that submits the same form 3 times in the 2 minutes has a problem and can be encouraged to call the helpdesk. CEP systems receive web events and correlate them into patterns in real time using programmable rules. They should be integral part of the web analysis solution.
Best Practice 4: Integration with Enterprise Systems
Web traffic analysis does not live on its own. Its data almost always needs to be fed to enterprise systems such as CRMs, event databases, data warehouses, management systems and more, depending on the purpose of the analysis. This way, the results of the analysis are integrated with other data sources, stored and most importantly - actions can be effectively based on it. For the integration to be successful, the analysis system must be equipped with industry standard protocols used to feed such systems, for example SQL and message queues.