Detecting attacks in encrypted traffic when no behavioral baseline exists
As web applications increasingly rely on HTTPS, security solutions face a growing challenge: how to identify malicious traffic when application-layer data is encrypted. While TLS encryption is essential for protecting user privacy and sensitive information, it also reduces visibility into the traffic flowing to protected applications.
TLS Fingerprinting (TLSFP) has emerged as an effective way to analyze encrypted traffic without requiring TLS decryption. By examining characteristics of the TLS handshake, security solutions can distinguish different client populations and identify behavioral patterns without inspecting encrypted application content.
This capability has become a fundamental component of modern Web DDoS protection for encrypted applications. However, some deployment scenarios require mitigation to begin immediately, leaving little or no opportunity to observe normal traffic patterns beforehand.
In these situations, the ability to identify malicious TLS fingerprint activity using only the traffic currently arriving at the mitigation platform becomes critical.
What Is a TLS Fingerprint?
Before an encrypted TLS session is established, a client sends a Client Hello message to the server. This message contains information about the client's TLS implementation, such as supported TLS versions, cipher suites, and extensions. These attributes are collected and combined into a unique identifier known as a TLS Fingerprint, or TLSFP.
Different clients produce different combinations of these attributes. A browser, mobile application, automation framework, bot, or custom TLS library may therefore generate a distinct TLS fingerprint.
TLS fingerprints do not expose encrypted application content. Instead, they provide metadata that can be used to group connections generated by similar client software.
This is valuable during a Web DDoS attack because attack tools often create large volumes of connections using a small number of common TLS implementations. Legitimate traffic, by contrast, is usually distributed across a broader mix of browsers, operating systems, and applications. The ability to group traffic by TLS fingerprint provides a powerful signal for identifying suspicious behavior, making TLSFP an effective foundation for encrypted Web DDoS detection.
The Patented Behavioral Foundation
A TLS fingerprint by itself is not inherently malicious. A popular fingerprint may belong to a widely used browser, while an uncommon fingerprint may belong to a legitimate business application.
For this reason, Radware's behavioral TLSFP protection learns activity per fingerprint and per protection policy. During normal operation, it builds a behavioral baseline that describes which fingerprints are typically present and how they normally behave.
During an attack, current activity is compared against this learned baseline. Fingerprints that generate significantly more traffic than expected, or otherwise deviate from their learned behavior, can be identified as anomalous and mitigated.
This approach is the basis of Radware's U.S. Patent No. 12,580,955 B2, "Method and System for Characterization and Mitigation of Encrypted Distributed Denial of Service (DDoS) Attacks". The patent names Guy Perez and Avi Chesla as inventors and Radware Ltd. as the assignee. It describes the use of TLS fingerprints and a native fingerprint baseline to characterize anomalous fingerprint groups and mitigate encrypted DDoS attacks without decrypting application traffic.
The behavioral approach is highly accurate and selective. However, it relies on having sufficient historical traffic to establish a reliable behavioral baseline. This dependency creates challenges in deployment scenarios where mitigation begins during an active attack and historical traffic information is unavailable.
The Learning Challenge
Behavioral protection depends on one critical requirement: it needs time to learn what normal traffic looks like.
In an always-on deployment, the mitigation platform has time to observe normal traffic before an attack. That history gives baseline-dependent detection something trustworthy to compare against. In an Out-of-Path deployment, that advantage disappears.
Traffic normally travels directly to the application and is diverted only when an attack is already underway. The mitigation platform may therefore encounter the application for the first time at the worst possible moment: underload, with the attack mixed into the very traffic it is expected to learn from.
This is where baseline-dependent detection breaks down. There is no clean learning period and no established picture of normal behavior. A detector that starts learning only after diversion is already late, and its first version of "normal" may already be contaminated by the attack.
The central question is therefore straightforward: how can a mitigation platform identify suspicious TLS fingerprint groups from the traffic arriving now, before it has learned what normal looks like?
A Fresh Approach: Learn From the Traffic That Exists Now
To address the limitations of behavioral learning, Radware introduced On-Demand TLSFP Web DDoS Protection, a dedicated detection and mitigation flow that provides immediate protection against Web DDoS attacks without relying on historical baselines or prior learning.
Unlike behavioral protection, which compares current activity against previously learned traffic patterns, On-Demand protection asks a different question:
"How does this fingerprint compare with the rest of the traffic right now?"
The system continuously examines TLS fingerprint activity within short traffic windows, evaluating the relative volume and distribution of TLSFP activity in real time. In normal traffic, activity is typically distributed across many fingerprints. During many Web DDoS attacks, however, a small number of TLS fingerprints generate a disproportionate amount of traffic, creating a visible separation between attack-related traffic and the broader client population.
The detection process begins as soon as diverted traffic reaches the mitigation platform. TLS fingerprints are extracted from Client Hello messages and grouped according to the protection policy. Fingerprint groups that stand out as anomalous compared to the surrounding traffic population are temporarily classified as attack-related and selectively mitigated.
As traffic conditions evolve, the system continuously reevaluates detected groups, allowing mitigation to adapt dynamically as attack patterns change. Fingerprint groups that no longer exhibit suspicious behavior are automatically removed from mitigation, allowing legitimate traffic to continue without unnecessary disruption.
By relying on live traffic analysis rather than historical learning, On-Demand protection provides effective mitigation from the moment traffic reaches the device, delivering immediate TLSFP-based protection even when mitigation begins during an active attack.
Detection flow
Traffic diversion → Client Hello collection → TLSFP distribution by policy → Suspicious-group detection → Selective mitigation → Continuous reevaluation
The result is immediate TLSFP-based protection without a historical baseline and without TLS decryption.
Illustrative Attack-Traffic Example
Consider a simplified traffic window, drawn from the validation data, containing 100 TLS fingerprints. Real application distributions vary, and the rates below are observations, not product thresholds.
Most fingerprints generate between 20 and 100 Client Hello messages per second. Four fingerprints, however, generate:
| Fingerprint group |
Client Hellos per second |
| Group A |
133,575 |
| Group B |
20,428 |
| Group C |
12,364 |
| Group D |
11,079 |
| Next-largest remaining group |
75 |
There is a clear gap between the fourth dominant fingerprint and the rest of the traffic: approximately 11,079 versus 75 Client Hellos per second.
On-Demand protection can identify the four dominant groups as suspicious and mitigate them while allowing the other 96 fingerprint groups to continue.
The important point is that the system did not require any prior knowledge of the application or a previously established behavioral baseline.
High volume alone is not enough to trigger mitigation. If several legitimate fingerprints are busy but remain part of natural traffic distribution, the system does not see the same sharp separation and permits them.
Validation Against Production Traffic
We evaluated the On-Demand detector using production TLS fingerprint statistics and compared its decisions with those of the established behavioral engine operating on the same traffic. The behavioral engine was used as a comparison reference; this evaluation measures agreement with its attack detections, not absolute ground-truth accuracy.
The dataset contained:
- 154,623 ten-second traffic windows
- Approximately 17.9 days of combined traffic
- Two devices and eight protection policies
- Two Web DDoS attack episodes
The On-Demand detector received no prebuilt behavioral baseline.
Results
| Metric |
Result |
| Behavioral-reference attack episodes also detected |
2 of 2 |
| Agreement on reference attack episodes |
100% in this dataset |
| Delay compared with behavioral detection |
0 seconds |
| Peacetime windows with a block |
2 of 154,545 |
| Peacetime block rate |
0.0013% |
| Highest mitigated rate in one window |
177,446 Client Hellos/s |
For both behavioral-reference attack episodes, On-Demand protection identified suspicious fingerprint groups in the same traffic window in which the trained behavioral engine declared the attack.
The validation also included 1,262 normal traffic windows from seven Frankfurt policies. The detector did not block any fingerprint in those windows, including windows in which a legitimate fingerprint reached 449 Client Hellos per second.
These results demonstrate the central value of the approach: immediate detection can be achieved without treating every busy fingerprint group as suspicious.
These results describe the evaluated dataset and should not be interpreted as a guarantee of identical performance in every environment. Traffic composition, policies, and attack characteristics vary.
Extending Behavioral TLSFP Protection to On-Demand Deployments
On-Demand protection is not intended to replace behavioral protection.
The patented behavioral foundation has the advantage of understanding the application's historical traffic behavior. It can understand the normal activity of individual fingerprints on each policy and identify changes relative to that established behavior.
On-Demand protection extends the same TLSFP-based protection strategy to a new operational setup: it can act when that history does not yet exist.
| Deployment situation |
Recommended approach |
| Always-on protection with established traffic history |
Behavioral TLSFP protection |
| Out-of-Path traffic diverted during an attack |
On-Demand TLSFP protection |
| Newly protected application without a baseline |
On-Demand protection |
| Emergency activation during an active incident |
On-Demand protection |
Together, the two modes address different stages of the protection lifecycle. On-Demand protection closes the immediate gap for diverted or newly onboarded traffic, while the patented behavioral approach provides deeper application-specific analysis when a learned baseline is available.
Considerations
TLSFP provides visibility into client populations, not individual users. Its effectiveness depends on meaningful differences in the traffic distribution:
- Legitimate and malicious connections may share the same TLS fingerprint.
- Separation may be less pronounced when attack traffic is spread across many fingerprints or closely resembles legitimate traffic volumes.
- Results vary with each application's clients, traffic profile, protection policy, and attack characteristics.
- TLSFP should be one layer of a broader Web DDoS protection strategy.
These considerations are also why On-Demand and learned behavioral protection are complementary.
Key Benefits
- Immediate protection: No peacetime learning period is required.
- Designed for Out-of-Path deployments: Detection begins when diverted traffic first reaches the mitigation platform.
- No TLS decryption: Decisions use TLS fingerprint statistics rather than encrypted application content.
- Selective mitigation: Suspicious fingerprint groups are isolated while other traffic continues.
- Automatic adaptation: Fingerprints are added to and removed from mitigation as traffic changes.
- Policy-aware operation: Detection and mitigation are applied within the relevant protection policy.
Closing the Protection Gap
Encryption should not force organizations to choose between user privacy and effective DDoS protection. TLS fingerprints provide useful visibility into client populations without exposing application content, but visibility alone is not enough. Protection must also work when diverted traffic arrives before a trustworthy behavioral baseline exists.
Radware's patented behavioral TLSFP approach addresses the steady-state case by learning how fingerprint groups normally behave. On-Demand TLSFP Web DDoS Protection addresses the first minutes after diversion: it analyzes the live relationship between fingerprint groups, isolates groups associated with concentrated attack traffic, and adapts as conditions change, all without a prior baseline or payload decryption.
Together, these modes provide a practical bridge between traffic diversion and fully learned behavioral protection: detection that starts when the traffic arrives, not after the learning period ends.
Assess Your First Minutes After Diversion
Before relying on an out-of-path mitigation architecture, security teams should verify three things:
- What happens during the first traffic window after diversion? Determine whether protection begins immediately or waits while the platform builds context.
- Does TLS-based detection require pre-attack history? Confirm whether the detector can make a selective decision when no clean baseline exists.
- Can suspicious client populations be isolated without decrypting traffic? Validate that mitigation can distinguish concentrated attack groups while allowing the broader client population to continue.
If any of these questions expose a delay or visibility gap, the architecture may be most vulnerable at the exact moment traffic is diverted for protection.
The validation figures in this article were produced by replaying On-Demand TLSFP detection against production traffic statistics and comparing its decisions with the behavioral engine operating on the same traffic. The behavioral engine served as the comparison reference rather than independently verified ground truth. Fingerprint identifiers were omitted, and example rates were rounded for readability.