Please refer to the screenshot below. My initial impression was that SSL and HTTPS traffic was the same but why is it reported as 2 different entries in this screen?
The customer has highlighted that there is a very large discrepancy between the hits on the Webserver logs and whatever is reported on DC RUM. Also, there is a large portion of the operations that is reported as failed. However they are all in the "all other operations" bucket.
Currently the software service is defined based on the "SSL Decrypted" decode. I was suspecting whether some of the traffic of the same IP and port is not recognized by the decode.
Do you noticed any SSL problems? Like missing keys or unknown cipher?
Sometimes we see that a HTTPS service only gets partialy decrypted because clients are using unsupported ciphers or any thing like this.
You can analyse this in the SSL Section of "Verify Quality of monitored traffic" or on the AMD itself.
No SSL problems. Using rcon, ssldecr status, 1 key found, 0 keys missing.
A large portion of the analysis of the ssldecr status for the server shows "session terminated due to alert". Any ideas on how we can overcome this or advise the customer to do?
I've never seen something like this. But I googled alerts in ssl sessions and found this describtion:
The Alert Protocol
The Alert Protocol is used by
parties to convey session messages associated with data exchange and
functioning of the protocol. Each message in the alert protocol consists
of two bytes. The first byte always takes a value, “warning” (1) or
“fatal” (2) , that determines the severity of the message sent. Sending a
message having a „fatal” status by either party will result in an
immediate termination of the SSL session. The next byte of the message
contains one of the defined error codes, which may occur during an SSL
Jonathan, can you post up the results of (from rcon)
show ssldecr status
show ssldecr ciphers
SSL alerts typically occur because of an application or security issue. Most commonly (that I see) an alert occurs when a SSL session is temrinated because the client and server could not agree on a cipher, or a bad certificate (expire, untrusted, etc.).
You can use a packet capure of the traffic to identify the issue, Wireshark is particularly good at identifying the alert values.
Hi @Chris Vidler,
I have the SSL Decrypt status output for the server.
show ssldecr status 22.214.171.124
SSL Decryption statistics for server: 126.96.36.199:443
Total number of sessions=8251535 (inProgress=532 Finished=8251003)
SSL protocol version breakdown per number of sessions:
supported versions: ssl3.0=0 tls1.0=1644475 tls1.1=56904 tls1.2=1397902
unsupported versions: ssl2.0=0 other versions=0 no version info=5151442
Long handshakes=1688029 Short handshakes=1411254 Compressed sessions=0 SessionTkt reused=0 SessionId reused=2749761
Finished sessions decrypted with no errors=2597884 (31% of all finished sessions)
Sessions in progress decrypting with no errors=490 (92% of all sessions in progress)
Finished sessions decrypted partially=159243 (1% of all finished sessions)
with a packet lost during payload data exchange=158934
with a corrupted payload data packet=3
with decryption failed during payload data exchange=0
terminated by alert during payload data exchange=306
Finished sessions not decrypted=5255777 (63% of all finished sessions)
with no private key found=0 (new sessions=0 reused sessions=0)
with a packet lost during handshake=144152 (new sessions=127700 reused sessions=16452)
with a corrupted handshake packet or incorrect handshake sequence=29547 (new sessions=29526 reused sessions=21)
with decryption broken during handshake=0 (new sessions=0 reused sessions=0)
with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
with unsupported SSL feature=0 (unsupported cipher=0 server key exchange=0)
with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
with RSA decryption failed=71, RSA invocations blocked=0 (new sessions=0 reused sessions=0)
reused sessions with no matching master session seen before=41821
with incomplete SSL handshake=13507 (new sessions=13248 reused sessions=259)
closed without data=52332
with invalid 'Hello' packet client=0, server=0
terminated by alert during handshake=4789084
reuse errors when PMS identified with session id=44328, with session ticket=0
session not seen from the beginning=185257
with other errors=6
Supplemental Data detected, server=0 client=0
+ SSL3-RSA-RSA-RC4-128-SHA ref=3099283
total server-certificate pairs=1
parsed properly=1 (matched=1 matching failed=0 not used=0)
parsing errors=0 (decode=0 extract=0 RSAerror=0)
I am not onsite but I remember running the show ssldecr ciphers and there wasn't anything weird about it. I'll try to get the output when I get onsite.
Shouldn't need the ciphers output, that there shows you've got a lot of packet loss affecting the SSL traffic, this will be the prime reason you're seeing less traffic than you expect. You need to reoslve the packet loss issue first (is it a SPAN or tap issue?).
@Chris Vidler Thanks for your input. They are using a mixture of taps & spans to monitor their DC traffic. We have feed-backed to the network team about the lost packet situation but the network people has changed many times and hence they are not certain about network infrastructure.
I will continue to try to get them to fix the issue but one thing stands out to me is "terminated by alert during handshake=4789084". Do you have any recommendations on how we can overcome this?
They could be a side effect of lost packets during the handshake causing it to fail. They could also be something else, the AMD doesn't provide any more info to help diagnose that.
As I mentioned earlier if you can capture the SSL handshake, Wireshark can extract the alert code from any failures and that'll give you more info to diagnose further with.
The alert codes are defined in the RFC5246 here: