Christmas hands-on: Who stole Santa’s secret file?

[Tiit_Hallas]

With Christmas on the way, Dynatrace is happy to bring to you: the Christmas Scavenger Hunt! 

This blog post will guide you through an incident response hands-on exercise. Each assignment introduces a goal to reach and steps to follow to reach this goal. You can either solve it by yourself, follow the steps to solve it or just look at the solved investigation available in Security Investigator.

Twas the night before Christmas, when all through the house, 
Not a creature was stirring, not even a mouse.
The servers were humming, the data secure,
But little did Santa know, he'd face a big blur.

His sensitive files, all locked up tight,
Were stolen by hackers, who worked through the night.
With keystrokes so nimble, and firewalls breached,
They found all his secrets, the ones he'd kept leached.

St Claus was not worried, he knew what to do.
He’s calmness returned in a minute or two.
He remembered his secret—his tech-saving grace.
AppSec at North Pole is done by Dynatrace.

About this hands-on exercise

This hands-on exercise, with steps to embrace,
Guides you through forensics, at a steady pace.
With logs as your clues, and analytics in play,
It walks you through what security pros say.

With Security Investigator, you’ll dive deep and clear,
Into logs and their secrets, no matter how near.
You’ll follow the steps that the experts all take,
To uncover the truth, and the patterns they make.

You’ll dig through the data, with focus and care,
Reconstructing the events, piece by piece, laid bare.
What really happened? The truth you will find,
As you sift through the logs, with a keen, sharp mind.

So dive into this journey, with knowledge in hand,
As a Security Engineer, you’ll take your stand.
And piece by piece, the evidence will show,
The sequence of events, and how they did flow.

 

Scenario

You get a worrying call from Santa that a secret document called Emuneration.xlsx (id: 31337) containing all the elf bonuses and extra presents has leaked from North Pole Document Management application. Since all the engineers in Santa’s support crew have their hands full with upgrading the firmware on Rudolf’s blinking nose, he asks you to help him with analyzing the application logs. You are given access to the log files at Dynatrace Playground to figure out what really happened and who stole the file.

For conducting the log analysis, you can use the Dynatrace Security Investigator. You can find North Pole Document Management application logs from Grail with the log.source value my_app.log.

 

Let’s start the investigation

With no time to waste, let us start with the investigation to get the culprit and get to the bottom of this!

1. Find logs from Grail

To start analyzing your application logs, see if you can find any logs that match the given log source my_app.log. To start a new case and fetch these logs, follow these steps:

1. Go to the Dynatrace Playground environment and start the Security Investigator
2. Choose the “+ Case” to start a new Investigation.
3. Fetch logs from the last 24h with the log.source my_app.log

To do that, copy the following DQL query to the Query section in Security Investigator, and select Run to display the results:

fetch logs, from: -24h
| filter log.source == "my_app.log"

 

2. Preparing the data

Nice! You got the logs! To analyze the log records faster and with increased precision, you must extract the data from the log records intoto separate fields. You can do this using Dynatrace Pattern Language (DPL). Extract the following elements from the log records:

• clients IP address;
• session ID;
• username;
• request;
• request payload;
• response;
• user-agent.

PS! Use the correct field types when extracting the fields to simplify further analysis. This will help you in log analysis later.

1. Right-click on a record in the results table and choose Extract Fields. To add more than one record to the DPL Architect, hold the Shift key while clicking on the records in the results table before choosing Extract fields.
   
   
2. Construct a new DPL pattern based on the log format. When developing the pattern, you can see the matching portion of the logs highlighted in the match preview editor section. To see which values are extracted as new fields, check the results tab.

3. To extract the previously listed elements from the records, use the following pattern:

   IPADDR:src_ip '\t'
   LD:session_id '\t'
   LD:user_id '\t'
   DQS:request '\t'
   JSON:payload '\t'
   INT:response '\t'
   JSON:result '\t'
   DQS:user_agent

    

4. Click Insert Pattern to add the pattern to your DQL query and close the DPL Architect.

5. Click Run to execute the query and display the extracted data as new columns. Double-click any of the records in the results table to open record details and verify if your fields were extracted as expected.
   
   PS! You can navigate among records in your results using the arrow keys on your keyboard or using the navigation arrows at the top of the Record details modal.
    

6. To keep working with just the fields you extracted, use the following fields command and re-execute the query by choosing Run:

   | fields src_ip, session_id, user_id, request, payload, response, user_agent

   PS! Notice how each query execution has created a new node to the query tree. The query tree enables you to navigate back to previous queries and see the previous results as you got them at the time of the execution. 

7. To keep the current node for future as a reference, give it a name: right-click the node, choose Rename and name it app requests. We will need it later.

 

3. Find out who has opened the file

From the logs you can see requests that are related to documents. They all have the following format: document.php?action=open&doc_id=<doc_id>. Continue the investigation by looking into who opened the leaked document.

To create a filter to fetch only the events where the action parameter is open and the doc_id is the ID of the leaked document, follow these steps:

1. Extract the request parameters into a separate field called params for easier filtering using the KVP matcher. Add the following parse command to the query and click Run:

   | parse request, "LD '?' KVP { LD:key '=' LD:value ('&'|' ') }:params"

   When viewing record details, the new params field is visible and usable along with the other fields.

   

2. Add the following filter command to see only the events where the document 31337 was opened and execute the query by choosing Run:

   | filter params[action] == "open" and params[doc_id] == "31337"

3. From the results it shows that thomas and eric have tried to open the document.
   

 

4. What did Thomas and Eric do?

Let’s dive in deeper and see what else Thomas and Eric have done in Santa’s environment. Follow these steps to set a color for the current node and look into both of their sessions to understand if there’s something suspicious happening in either of them.

1. Open the node menu, choose Color and select a color from the available options
   

2. Add a label for the color: open the legend menu and choose Edit labels. Label the selected color as opened a document
   

3. Make the color labels visible under the query tree by adding a check mark to the “Color labels” option in the legend menu.

4. Remove the last filtering line (|filter params[action] == "open" and params[doc_id] == "31337") from the query

5. Right-click on Thomas’ session_id value, choose “Filter for” to add it as a filter statement and Run the query.
6. It seems that there’s nothing too interesting in Thomas’ activities. Navigate back to the previous node in the query tree and repeat steps 4 and 5 with Eric’s session_id value.

PS! Did you notice that by changing the query in the tree it created a new query branch to the query tree? You can navigate among your previous queries as you to follow different investigation paths so you can keep track of what you investigated and where.

 It seems that neither Thomas nor Eric are relevant to this investigation: they both got HTTP 401 response for the document and from the rest of the logs it seems, that they probably just tried to open the document by accident (missed the right document ID by one number).

5. All actions done with the document

Let’s go back to the previous (the first colored) node to see if we can gather any new knowledge when looking at all the actions done to the document.

1. Change the filter statement for the query to see all the actions done with the doc_id 31337. Use the following filter command and by choose Run: 

   | filter params[doc_id] == “31337”

2. Choose a color for the newly created node and set the label for the color as “other actions with the document”. The legend should look similar to this:

   

   In the results table you see six sessions (including Thomas and Eric) that have performed actions with the leaked document. Let’s look into each of the new sessions in more detail. To do that, follow these steps: 

3. Remove the last filter command from the DQL query
4. Right-click the first value in the session_id column and add it as a filter using Filter for and Run the query to create a new node with the results about the session.
5. Navigate back to the previous node and repeat step 3 with next values in the session_id column. You can skip thomas and eric, since we already went through their sessions.

6. As a result, you should have a query tree that looks like this:
   

6. Gathering evidence from a suspicious session

Going through the session branches, one session stands out with really suspicious requests. In Bobs' branch with the session_id value c9dde8d659d, you can see lots of suspicious activities, including uploading a file with the name shell.php. Let’s Mark the branch with a distinct color and gather and store relevant evidence needed for further investigation to find out who leaked the file. To do that, follow these steps:

1. Change the node color to orange and add the label a suspicious session to the color.
2. Right-click on any of the session_id values in the results table, choose Add to evidence list, choose + Add new evidence list and name it Bob’s session ID for future reference.
3. Right-click any of the IP addresses in the src_ip column and add Bob’s IP address as a new evidence called Bob’s IP address.

4. Select the /upload/shell.php portion from the content column, right-click it and save it as a new evidence called suspicious script.
   

5. The evidence in your case should look now something like on the following image:
    

 

7. Decoding the commands

When watching the log records in Bob’s session, you discovered that the payload posted to shell.php script is base64 decoded.

src_ip	session_id	user_id	request	payload
84.50.138.122	c9dde8d659d	bob	POST /upload/shell.php HTTP/1.1	{"p":"eyJhY3Rpb24iOiJzZXR1cCJ9"}
84.50.138.122	c9dde8d659d	bob	POST /upload/shell.php HTTP/1.1	{"p":"eyJleGVjIjoid2hvYW1pIn0="}

To understand, what is posted to the script, follow these steps:

1. Add a filter to see only the POST requests to the shell.php script by right-clicking any of the POST requests in the content column and choosing Filter for option from the menu:
   
   

2. Add a new field called decoded to the results using the fieldsAdd command. The value of the field should be the base64-decoded value of the parameter p from the payload value.

   | fieldsAdd decoded=decodeBase64ToString(payload[p])

   After running the query, a new field is present in the results table that contains the decoded command:
   

8. Find shell users

By now, you’ve discovered how the web shell was set up and the first executions of the commands using the web shell. Let’s now continue and investigate who else has used the shell. To find out and to gather evidence about what has been done with the suspicious script and by which sessions, follow these steps:

1. Navigate back to the query node we named app requests in the 2. Section at step 6.

2. Open the evidence suspicious script menu at the right side panel, choose Filter for and select the column request to apply the evidence as a filtering statement to the DQL query.

   

3. Since you already viewed Bob’s session, you can leave his session out. You can do this by opening the evidence Bob’s session id menu, selecting Filter out. Select the column session_id and choose Run.
4. Choose a purple color for the created node and label the color shell access to make the branch easier to find.

5. Seems that we found another session that used the web-shell. To understand what was done with the script, let’s decode the payload as we did with Bob’s session by adding the following fieldsAdd command: 

   | fieldsAdd decoded=decodeBase64ToString(payload[p])

   From the results it seems this session is the culprit behind stealing the file (and as we see, not only this file!).

   

9. Simple statistical analysis

To move further with analyzing this session, let’s do some statistical analysis for the session and see if something stands out. Let’s start by looking at all the distinct user-agents and IP addresses from this session.

1. Add a summarize command that will aggregate the results based on session_id, src_ip and user_agent fields:

   | summarize count(), by: { src_ip, session_id, user_agent }

2. Execute the DQL query and analyze the results
   The results reveal two records:
   

   We have found two IP addresses within the same session with the same user-agent. This might mean that if the attacker was using an anonymizer to hide their tracks, the service might have failed for some requests and might reveal the attacker’s real IP address. This requires some further investigation. 

    Let’s save this evidence and continue the investigation by following these steps:

3. Right-click a value in the session_id column and save it to a new evidence list called attacker’s session ID.
4. Right-click the user-agent value in the user_agent column and save it to a new evidence list called attacker’s userAgent
5. Select both the IP addresses by holding down the Shift key, right-click them, and add them to the Suspicious evidence list

10. Unveiling the attacker

We have found the attacker’s IP address, session ID and the session ID from where the data was stolen. Let’s see if we can also find the same indicators from another session involving some authenticated users. Follow these steps:
 

1. Navigate back to the node “app requests“
2. Build the DQL query:
   • From the Suspicious evidence list menu, select Filter for and choose the src_ip as the field. Both IP addresses from the evidence lists are added to the filter.
   • From the user_agent column header, select Filter for and choose attacker’s userAgent evidence list
   • From the session_id column header, select Filter out and choose attacker’s session ID

3. The query filters created look like this:

   | filter ipIn(src_ip, {"84.50.141.4", "99.47.29.66"})
   | filter contains(user_agent, "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0")
   | filterOut contains(session_id, "ec3d68248c9")

4. Add a summarize command to aggregate the results based on the source_ip, session_id and username:

   | summarize count = count(), by: {src_ip, session_id, user_id}

5. Run the query, set the node color to yellow and label the color unveiling the attacker! Only one other user has used the same IP address that was accidentally used for a brief moment when stealing the documents. It is now time to call Santa back and let him know who should be added to the Naughty list!

Conclusion

It seems at last we've found the truth,
The culprit hiding in plain view.
An IP trace, a fleeting spark,
Reveals the shadow in the dark.

Though the documents did slip away,
We’ve caught the hand that led astray.
The evidence is clear and bright,
The guilty party now in sight.

So with our task now done and right,
Where Dynatrace yet shed some light.
We bid farewell to this long fight,
Merry Christmas to all, and to all, good night!

Congratulations! You have finished the Hands-on Exercise! The query tree that I constructed during this investigation looks like this:

For reference, there is a read-only investigation of this walkthrough available at https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/share/fe877938-7e71-47b2...

[AntonioSousa]

Very funny!

You are right in that the attacker might be playing special tricks. The IPs are very credible 😁, but the browser version reveals some things. Although a Firefox user, I was not the one 🤣

[PacoPorro]

Nice one!!!!!

[marina_pollehn]

Love it! We need more scavenger hunts like this one <3.

[DanielS]

Brilliant!!!!

[Mizső]

Excellent!!!

[Peter_Youssef]

Thanks @Tiit_Hallas for insightful and organized contents.

[zietho]

I hope your trees don't look like the one from the investigation, or otherwise, they will likely fall when your cat visits them. 

@Tiit_Hallas hilarious! kudos!