- last edited on by
MaciejNeumann
Over the course of the year we have run into issues impacting our core application. These issues did have different triggering factors but the common denominator between them was the presence of undesired content in the request header.
Issue: Duplicate and/or additional malformed data in the request header. This caused users to be unable to access our business critical application.
Solution Ideas:
The overall goal is to be able to detect and alert on the presence of a construct such as a key:value pair in the request header.
Idea 1 (Resulted in failure) - Formulate a Request Attribute (RA) that counts the cookie request header and pull the count (either unique in value or total sum) of the cookie values.
Idea 1 Implemented:
We formulated a RA that looks at the HTTP Request header on the server side which focuses on the cookie parameter, and set the RA to be an Integer data type where we could then set it as the number of values or the number of distinct values.
The Key Value par we were interested in was the rxVisitor=<VALUE> segment of the cookie since a duplication/ malformed duplication causes problems. And with this count, it should count the number of these Key:Value pairs to be 1 in a normal state and 2 in a bad state.
This proved to be problematic as there were cases where the value was higher than 1 or even 2... Upon a deeper dive we concluded that the RA as an integer was not able to count the parsing in the cookie, rather counting the presence of the cookie overall.
Soo... back to the drawing board.
Idea 2 (Success!) - It's now the holiday season so I decided to get up early and get my Wife her favorite Starbucks holiday coffee. While sitting in an endlessly long line, the idea of a method came to me. 'If the RA cant count targeted values in a construct like cookie, but it can extract values.... Why not build two RAs, one looking at and pulling the first rxVisitor Value Pair (Desired state), and then a second RA looking at the presence of a second rxVisitor value (Undesired state.
Hurtles to overcome - Formulating these RAs is easy, but what is tricky was the regex. So we built two RAs 'First RxVisitor Value | Second RxVisitor Value' and leveraged the text type and selected to capture all the values. now our biggest hurtle, formulating a regex (that Dynatrace will allow) to grab the RxVisitor values. That was formulated but now I had to figure out how to skip the first value and look for the second value, which could be anywhere in the header. More importantly I cant do start of the string and end of string as both could hit the single value and give a false positive of a second value. After goolging and reading up I found the solution ... the look ahead, which ultimately didn't work out (could be my lack of regex skills). so we looked at it again and found a clever way to use the Dynatrace Parsing rules to formulate the data we wanted to see.
Setting the parsing rules - As mentioned we needed to target the first value for one RA and then the Second value for the Second RA, if and when that value is present. So we attempted to leverage a look ahead method, this regex script grabs the first qualifier of the rxVisitor: rxVisitor=(?=1)[^\-;]*+, but we found that the (?=<Value>) only starts the capture if and when the string starts with your value as defined. So we came up with a clever solution.
First Cookie Present - For the first cookie RA we did in fact use Regex since it will scour the entire cookie header and provide us the value of the entity anywhere in the string, and once its found, it stops and wont return a second. The string we used was: rxVisitor=[^\-;]*+
Second Cookie Present - This is where we get clever with the parsing of the data. In step 1 we define out that we want to extract the substring 'After', rxVisitor. That means the first value becomes 123456 rather than the original value of rxValue=123456. Step two we split the data via the ';' since all the key value pairs in the header are separated by that character. Finally we use step 4 where we then further define it via only using the value if it 'begins with' rxVisitor. Which we know the second value will, since we cut off the key value pair of the initial - first seen - rxVisitor Value pair.
Now that we have the RA's built, you will need to allow them to start to qualify since they are forward facing. While that populated we then formulated a dashboard that is looking at the number of services with a single vs duplicate cookie value we just defined. Taking it a step further, we added the RAs as a session string value and are now able to leverage USLQ and provide user names, actions and associated applications that has two RA's we built present. Keep in mind we are not in a healthy state so the duplication ones are empty but will come alive once detected.
Expanding to User Sessions - This step is quite simple since the RA is built, just go to the desired app and edit the settings > Capturing > Session and Action Properties:
Now user sessions are linked!
Final Step: Alerts - We wanted to be alerted, so since the RA is linked to the services, we formulated a Metric event and targeted a silly metric for this case - request count. I say its silly because it is for this solution, the number or requests has no bearing on my alert, but I need a metric that be use din parallel for the RA, so my logic is - if there are requests, there is a possibility for the Cookie RA to be present, single, or both single and double.
So we used the request count for services, but added a filter to only include requests that have the RA of Second rxVisitor value. So logically, even though there are requests for services as a whole, none of them have the second RA value thus resulting in no qualifications. Now for the alert criteria this is where it gets really silly, I selected count and said alert me when the count is above 0... yes 0, because the metric selector I defined is looking at service count but is filtered in the metric selector to look at my second RA, which is only present when he issue occurs. So now once a request comes in and that second RA is present, I'm immediately over my count of 0 thus triggering the alert.. I then took it a step further and included exact verbiage of what the issue is in the details/body of the metric event, that way there is no confusion as to what the issue is, cookie or the number of requests.
Then you just need to link this event to your desired alert profile/notification integration.
You can use this method for anything inside the request header where the presence or lack of causes an impact on your core applications.
Unique problems require unique solutions. 🙂
Here is the single pane of glass dashboard we developed:
Featured Posts
