cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Traffic not getting decrypted 100% on AMD

kapil_nemade
Inactive

Hi Guys - Certificates are matched successfully however application traffic is not getting 100% decrypted. I have attached snap shot of "show ssldecr status" for reference. I have already loaded the same certificate three times but still results are same. ssldecr-status.jpg

If anybody has idea on this please let me know.

31 REPLIES 31

raffaele_talari
Inactive

Hi @Kapil Nemade,

could you please provide us the output of the following rcon command:

show ssldecr ciphers

Thanks, Raff

Hi Raffaele - I have posted the snap shot of ssldecr ciphers

Thanks, Kapil.

It seems you're not using any unknown/unsupported ciphers. That's good.

By giving a second look to the decryption status it seems that something's wrong with the cache.

I believe we might ask for senior colleagues help on this. @David Alonso

Ciao, Raff

kapil_nemade
Inactive

ulf_thornander3
Inactive

As a general note on SSL decryption - you need to catch the first handshake.

If you turn on decryption in the middle of an ongoing conversation, you will not see anything.

Can you brief on how to catch the first handshake.

Thanks

You just have to wait until the ongoing session terminates and starts again. The handshake is what happens at the session setup.

The other way (of course) is to reboot the server in question.

wojciech_kurek
Inactive

The output on the screenshots is from Vantage 11.1 or older. The issue may be simply caused by the fact, that the AMD is so outdated that it fails to understand modern SSL traffic, even if the cipher itself is supported. In addition, the "packet errors" counter indicates there's a problem with traffic quality (packets lost on the traffic feed before the AMD).

You are correct that it is very older version of Vantage however we were able to see decrypted traffic until last month with same certificates only the monitoring url version is recently changed. is that causing issue?

Kapil Nemade

Try the command

SHOW SSLDECR STATUS ip_addr
So you can see if you have unsupported cipher in your traffic and you'll have statistics for the IP address

Hi Didier - There is no unsupported cipher in our traffic. kindly find snap shot attached.

kapil_nemade
Inactive

Hi All - I have loaded certificates multiple times but still traffic is not getting decrypted 100%. Any more ideas or suggestion.

ulf_thornander3
Inactive

Hi Kapil

Every time you load the certificates you interrupt the ongoing decode. The likelyhood that you will get to 100% decreases every time you interrupt the decode process. If you leave the AMD running, the chances are that you will get closer to 100%.

You need to have the AMD up running with the certificates when the first packets in the conversation pass by. This first 3 packets is called the "hand-shake", just after that, the key is exchanged. If the probe doesn't see that key, it has no chance to decode the packet stream following.

Hi Ulf,

i left AMD running for last two days without any interruption however no more positive results. All traffic is flowing and can be seen on show ssldecr status command output but i am curious which initial packets or how much time we should wait for handshake to happened.

didier-externe_
Inactive

Hi Kapil,


I do not see the snapshot you're talking about :


The snapshot ssldecr-cipher.jpg is the result of the "SHOW SSLDECR CIPHERS" command which show you the supported or not supported cipher by the AMD. But it does not show you if there is or not unsuported ciphers in your real traffic.

The snapshot ssldecr-status.jpg is the result of the "SHOW SSLDECR STATUS" command which is the aggregated informations.


Let's have a look at the help command : SHOW SSLDECR HELP or SHOW SSLDECR STATUS HELP

SHOW SSLDECR CIPHERS - information on the supported and unsupported cipher suites
SHOW SSLDECR STATUS - show aggregated information about SSL decryption status
SHOW SSLDECR STATUS * - show general information about SSL decryption status for all servers
SHOW SSLDECR STATUS ip_addr - show general information about SSL decryption status filtered by IP address


The 2 first commands are not very usefull.
If i where you, I would start with the 2 last commands to check my traffic.

Hi Didier,

i have attached snap shot of the two commands mentioned by you - SHOW SSLDECR STATUS HELP & SHOW SSLDECR STATUS ip_addr. Both end up with result wrong token.

kapil_nemade
Inactive

kapil_nemade
Inactive

Also i have observed below statement in diagnostic console. What does that mean exactly? Certificate invalid?

High number of SSL decode failures (100.0%) - check if all decryption keys are valid and properly assigned

Erik_Soderquist
Dynatrace Pro
Dynatrace Pro

To me it looks like poor traffic quality (lots of errors due to missing packets) and too old of an AMD to correctly recognize modern TLS traffic.

I recommend upgrading the environment to a current version.

-- Erik

Hi Erik - Traffic hasn't changed. We were getting decrypted data until 1st week of this month. Any idea about below error getting on AMD.

High number of SSL decode failures (100.0%) - check if all decryption keys are valid and properly assigned

didier-externe_
Inactive

Hi Kapil,

I think the same way as Erik, it's time to upgrade... What is your AMD version exactly ? i think you can get the version when you start the "rcon" command.

You're AMD is too old and does not have all the commands i gave you...

I'll try to help you, but i do not have an old AMD...

Have you tried other commands ? the help command should give you some ways : "HELP" or "NameOfACommand HELP".

Try (I don't know if those commands exist in your old version) :

SHOW SSLDECR HELP

SHOW SSLDECR STATUS *

SHOW SSLDECR STATUS ip_addr (but this time, replace "ip_addr" with one of your software IP address)

SHOW SSLDECR KEYS

Does the application changed something ? a IP addresse ? a port ? a certificate ?

Have you check the certificate with your application responsable ? may be he has changed it on his application and you need to install the new one on your AMD ?

Hi Didier - AMD version is - ver. ndw.10.3.747. I know it is too old that it is not accepting these commands (SHOW SSLDECR STATUS * / SHOW SSLDECR STATUS ip_addr). Also we are in process of lifting that to new version of DCRUM which will take time to implement that in prod however by that time we can not continue with outage to existing application hosted on old version. I also checked with application guys and they confirmed that no certificates are changed at their end.

Hi Kapil

Would you ask Microsoft why IE6 doesn't work with HTML5?

Same thing here.

As most of us has tried to explain, the updates to your environment is nothing you can control and hence, you need to keep DCRUM updated so that it understands the traffic. What you are seeing is simply DCRUM not having the means to decode the traffic because the traffic is being generated by newer binaries in the browsers.

ulf_thornander3
Inactive

Hi Kapil

Perhaps you have not changed anything but probably the web server is being patched (normally they do) and that can also cause issues as the binaries are being updated to cover for all the zero day exploits and other nasties that run rampant.

That can cause an older AMD (and SSL decode) to not work as intended.

Even if the server was not patched, client browsers frequently receive automatic updates. Such client browser updates may also change which SSL features are used and in consequence break decryption on an ancient AMD.

I checked that possibility as well with application team and got a confirmation that servers are not patched from last year's time.

It's still possible that client browsers have been updated.

Are you seriously saying that these servers have not received any OS or application patches in over a year?!? If they are public internet facing, I would be absolutely amazed if they were not being controlled my remote botnets that had broken in...

That said, to even begin a diagnosis beyond what has already been listed here (poor traffic quality and very likely unsupported ciphers), the AMD would need to be upgraded just for better diagnostic tools. In the current AMD builds, the AMD diagnostic tools can provide a breakdown per monitored server of the exact reasons it can't decrypt the traffic, even including a per server breakdown of ciphers that had not been created yet when the AMD was built by detailing what unknown ciphers were seen for each monitored server.

Short of that, we'd need a long capture that demonstrates the failures and the SSL private key to match the traffic. Since the AMD version is too old to be supported, this analysis of the capture would have to be done by a volunteer on personal time.

-- Erik

waikeat_chan
Mentor

Hi Kapil, just curious about it because seems like I'm running into the same issue too: this incident happened all the time (never get 100% decrypted) or occasionally (sometimes 100% decrypted and sometimes partially decrypted)?

Hi Wai

So a bare minimum for anyone to give you any tip on what is going on at your server, is to tell us what version you are on and a few screendumps - the same as Kapil has shared.

ed_staszko
Inactive

We came across this same symptom and it came down to the Cipher Suite being negotiated between the client and server. If it is a Diffie-Hellman based cipher suite then a "listener" type of interpretation will not be successful, even if you have the correct certificate. We re-ordered the cipher suites on some of our internal servers that required us to get the correct interpretation and brought the Non-Diffie-Hellman ones to the top. This helped in some instances, but some modern clients will still choose only those types and will not be interpreted.