# Christmas hands-on: Who stole Santa’s secret file? 📁

**URL:** <https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481>\
**Category:** Dynatrace Tips\
**Tags:** dql, application-security, security-investigato, dynatrace-is-fun, dpl\
**Created:** [December 23, 2024, 9:32pm UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481 "2024-12-23T21:32:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tiit\_Hallas](https://avatars.discourse-cdn.com/v4/letter/t/cdc98d/32.png) [@Tiit\_Hallas](https://community.dynatrace.com/u/Tiit_Hallas)\
**Post date:** [December 23, 2024, 9:32pm UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/1 "2024-12-23T21:32:19Z")

</div>

With Christmas on the way, Dynatrace is happy to bring to you: the **Christmas Scavenger Hunt**!

 ![File_hunt_2.jpg](https://global.discourse-cdn.com/dynatrace/original/3X/c/a/ca655641156456d6a9bcf0429573e0b206a7d986.jpeg)

This blog post will guide you through an incident response hands-on exercise. Each assignment introduces a goal to reach and steps to follow to reach this goal. You can either solve it by yourself, follow the steps to solve it or just look at the [solved investigation](https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/share/fe877938-7e71-47b2-8f03-dae29593daec) available in Security Investigator.

_Twas the night before Christmas, when all through the house,_&nbsp;  
_Not a creature was stirring, not even a mouse._  
_The servers were humming, the data secure,_  
_But little did Santa know, he’d face a big blur._

_His sensitive files, all locked up tight,_  
_Were stolen by hackers, who worked through the night._  
_With keystrokes so nimble, and firewalls breached,_  
_They found all his secrets, the ones he’d kept leached._

_St Claus was not worried, he knew what to do._  
_He’s calmness returned in a minute or two._  
_He remembered his secret—his tech-saving grace._  
_AppSec at North Pole is done by Dynatrace._

### About this hands-on exercise

_This hands-on exercise, with steps to embrace,_  
_Guides you through forensics, at a steady pace._  
_With logs as your clues, and analytics in play,_  
_It walks you through what security pros say._

_With **Security Investigator** , you’ll dive deep and clear,_  
_Into logs and their secrets, no matter how near._  
_You’ll follow the steps that the experts all take,_  
_To uncover the truth, and the patterns they make._

_You’ll dig through the data, with focus and care,_  
_Reconstructing the events, piece by piece, laid bare._  
_What really happened? The truth you will find,_  
_As you sift through the logs, with a keen, sharp mind._

_So dive into this journey, with knowledge in hand,_  
_As a Security Engineer, you’ll take your stand._  
_And piece by piece, the evidence will show,_  
_The sequence of events, and how they did flow._

# Scenario

You get a worrying call from Santa that a secret document called **Emuneration.xlsx** (id: **31337** ) containing all the elf bonuses and extra presents has leaked from North Pole Document Management application. Since all the engineers in Santa’s support crew have their hands full with upgrading the firmware on Rudolf’s blinking nose, he asks you to help him with analyzing the application logs. You are given access to the log files at Dynatrace Playground to figure out what really happened and who stole the file.

For conducting the log analysis, you can use the [Dynatrace Security Investigator](http://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/). You can find North Pole Document Management application logs from Grail with the log.source value my\_app.log.

# Let’s start the investigation

With no time to waste, let us start with the investigation to get the culprit and get to the bottom of this!

## 1. Find logs from Grail

To start analyzing your application logs, see if you can find any logs that match the given log source my\_app.log. To start a new case and fetch these logs, follow these steps:

1. Go to the [Dynatrace Playground](https://www.dynatrace.com/signup/playground/) environment and start the [Security Investigator](https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/)
2. Choose the “ **+ Case** ” to start a new **Investigation**.
3. Fetch logs from the last 24h with the log.source my\_app.log

To do that, copy the following DQL query to the **Query** section in Security Investigator, and select **Run** to display the results:

```auto
<span class="fontSizeMedium"><span>fetch logs, from: -24h</span></span><br><span class="fontSizeMedium"><span>| filter log.source == "my_app.log"</span></span>

```

## 2. Preparing the data

**Nice!** You got the logs! To analyze the log records faster and with increased precision, you must extract the data from the log records intoto separate fields. You can do this using [Dynatrace Pattern Language](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-pattern-language) (DPL). Extract the following elements from the log records:

- clients IP address;
- session ID;
- username;
- request;
- request payload;
- response;
- user-agent.

**PS!** Use the correct field types when extracting the fields to simplify further analysis. This will help you in log analysis later.

1. Right-click on a record in the results table and choose **Extract Fields**. To add more than one record to the DPL Architect, hold the Shift key while clicking on the records in the results table before choosing **Extract fields**.

2. Construct a **new DPL pattern** based on the log format. When developing the pattern, you can see the matching **portion of the logs highlighted** in the match preview editor section. To see which values are extracted as new fields, check the **results** tab.

3. To extract the previously listed elements from the records, use the following pattern:

4. Click **Insert Pattern** to add the pattern to your DQL query and close the DPL Architect.

5. Click **Run** to execute the query and display the extracted data as new columns. Double-click any of the records in the results table to open **record details** and verify if your fields were extracted as expected.

6. To keep working with just the fields you extracted, use the following fields command and re-execute the query by choosing **Run** :

7. To keep the current node for future as a reference, give it a name: right-click the node, choose **Rename** and name it _ **app requests** _. We will need it later.

## 3. Find out who has opened the file

From the logs you can see requests that are related to documents. They all have the following format: document.php?action=open&doc\_id=\<doc\_id\>. Continue the investigation by looking into **who opened the leaked document**.

To create a filter to fetch only the events where the action parameter is open and the doc\_id is the ID of the leaked document, follow these steps:

1. Extract the request parameters into a separate field called params for easier filtering using the KVP matcher. Add the following parse&nbsp;command to the query and click **Run** :

2. Add the following filter command to see only the events where the document 31337 was opened and execute the query by choosing **Run** :

3. From the results it shows that **thomas** and **eric** have tried to open the document.

## 4. What did Thomas and Eric do?

Let’s dive in deeper and see what else Thomas and Eric have done in Santa’s environment. Follow these steps to set a color for the current node and **look into both of their sessions** to understand if there’s something suspicious happening in either of them.

1. Open the node menu, choose **Color** and select a color from the available options

2. Add a label for the color: open the legend menu and choose **Edit labels**. Label the selected color as _ **opened a document** _

3. Make the color labels visible under the query tree by adding a check mark to the “Color labels” option in the legend menu.

4. Remove the last filtering line (|filter params[action] == “open” and params[doc\_id] == “31337”) from the query

5. Right-click on Thomas’ session\_id value, choose “ **Filter for** ” to add it as a filter statement and **Run** the query.

6. It seems that there’s nothing too interesting in Thomas’ activities. Navigate back to the previous node in the query tree and repeat steps 4 and 5 with Eric’s session\_id value.

_ **PS!** Did you notice that by changing the query in the tree it **created a new query branch** to the query tree? You can navigate among your previous queries as you to follow different investigation paths so you can keep track of what you investigated and where._

 ![content-8.png](https://global.discourse-cdn.com/dynatrace/original/3X/6/0/60ee9c289f2fd6e2819e5fe96ec6aceafcef5fa0.png)

It seems that neither Thomas nor Eric are relevant to this investigation: they both got **HTTP 401 response** for the document and from the rest of the logs it seems, that they probably just tried to open the document by accident (missed the right document ID by one number).

## 5. All actions done with the document

Let’s go back to the previous (the first colored) node to see if we can gather any new knowledge when looking at **all the actions** done to the document.

1. Change the filter statement for the query to see all the actions done with the doc\_id 31337. Use the following filter command and by choose **Run** :

2. Choose a color for the newly created node and set the label for the color as “_ **other actions with the document** _”. The legend should look similar to this:

3. Remove the last filter command from the DQL query

4. Right-click the first value in the session\_id column and add it as a filter using **Filter for** and **Run** the query to create a new node with the results about the session.

5. Navigate back to the previous node and repeat step 3 with next values in the session\_id column. You can skip **thomas** and **eric** , since we already went through their sessions.

6. As a result, you should have a query tree that looks like this:

## 6. Gathering evidence from a suspicious session

Going through the session branches, one session stands out with really suspicious requests. In Bobs’ branch with the session\_id value c9dde8d659d, you can see lots of suspicious activities, including uploading a file with the name shell.php. Let’s **Mark the branch** with a distinct color and **gather and store relevant evidence** needed for further investigation to find out **who leaked the file**. To do that, follow these steps:

1. Change the node color to orange and add the label _ **a suspicious session** _ to the color.

2. Right-click on any of the session\_id values in the results table, choose _ **Add to evidence list** _, choose _ **+ Add new evidence list** _ and name it _ **Bob’s session ID** _ for future reference.

3. Right-click any of the IP addresses in the src\_ip column and add Bob’s IP address as a new evidence called _ **Bob’s IP address.** _

4. Select the /upload/shell.php portion from the **content** column, right-click it and save it as a new evidence called _ **suspicious script** _.

5. The evidence in your case should look now something like on the following image:  
&nbsp;  

## 7. Decoding the commands

When watching the log records in Bob’s session, you discovered that the payload posted to shell.php script is base64 decoded.

| **src\_ip** | **session\_id** | **user\_id** | **request** | **payload** |
| --- | --- | --- | --- | --- |
| 84.50.138.122 | c9dde8d659d | bob | POST /upload/shell.php HTTP/1.1 | {“p”:“eyJhY3Rpb24iOiJzZXR1cCJ9”} |
| 84.50.138.122 | c9dde8d659d | bob | POST /upload/shell.php HTTP/1.1 | {“p”:“eyJleGVjIjoid2hvYW1pIn0=”} |

To understand, what is posted to the script, follow these steps:

1. Add a filter to see only the POST requests to the shell.php script by right-clicking any of the POST requests in the content column and choosing **Filter for** option from the menu:

2. Add a new field called decoded to the results using the fieldsAdd command. The value of the field should be the base64-decoded value of the parameter p from the payload value.

## 8. Find shell users

By now, you’ve discovered how the web shell was set up and the first executions of the commands using the web shell. Let’s now continue and investigate **who else has used the shell**. To find out and to gather evidence about **what has been done** with the suspicious script and by **which sessions** , follow these steps:

1. Navigate back to the query node we named _ **app requests** _ in the 2. Section at step 6.

2. Open the evidence **suspicious script** menu at the right side panel, choose **Filter for** and select the column **request** to apply the evidence as a filtering statement to the DQL query.

3. Since you already viewed Bob’s session, you can leave his session out. You can do this by opening the evidence **Bob’s session id** menu, selecting **Filter out.** Select the column session\_id and choose **Run**.

4. Choose a purple color for the created node and label the color _ **shell access** _ to make the branch easier to find.

5. Seems that we found another session that used the web-shell. To understand what was done with the script, let’s decode the payload as we did with Bob’s session by adding the following fieldsAdd command:

## 9. Simple statistical analysis

To move further with analyzing this session, let’s do some statistical analysis for the session and see if something stands out. Let’s start by looking at all the distinct **user-agents** and **IP addresses** from this session.

1. Add a summarize command that will aggregate the results based on session\_id, src\_ip and user\_agent fields:

2. Execute the DQL query and analyze the results  
The results reveal two records:

3. Right-click a value in the session\_id column and save it to a new evidence list called _ **attacker’s session ID**._

4. Right-click the user-agent value in the user\_agent column and save it to a new evidence list called _ **attacker’s userAgent** _

5. Select both the IP addresses by holding down the Shift key, right-click them, and add them to the **Suspicious** evidence list

## 10. Unveiling the attacker

We have found the attacker’s IP address, session ID and the session ID from where the data was stolen. Let’s see if we can also find the same indicators from another session involving some authenticated users. Follow these steps:

1. Navigate back to the node “_app requests_“

2. Build the DQL query:

3. The query filters created look like this:

4. Add a summarize command to aggregate the results based on the source\_ip, session\_id and username:

5. Run the query, set the node color to **yellow** and label the color _ **unveiling the attacker!** _ Only **one other user** has used the same IP address that was accidentally used for a brief moment when stealing the documents. It is now time to call Santa back and let him know who should be added to the Naughty list!

# Conclusion

_It seems at last we’ve found the truth,_  
_The culprit hiding in plain view._  
_An IP trace, a fleeting spark,_  
_Reveals the shadow in the dark._

_Though the documents did slip away,_  
_We’ve caught the hand that led astray._  
_The evidence is clear and bright,_  
_The guilty party now in sight._

_So with our task now done and right,_  
_Where Dynatrace yet shed some light._  
_We bid farewell to this long fight,_  
_Merry Christmas to all, and to all, good night!_

**Congratulations!** You have finished the Hands-on Exercise! The query tree that I constructed during this investigation looks like this:

 ![content-3.png](https://global.discourse-cdn.com/dynatrace/original/2X/9/91004c1b9d5cf6ecd18abdea58735562bd989553.png)

For reference, there is a read-only investigation of this walkthrough available at [Dynatrace](https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/share/fe877938-7e71-47b2-8f03-dae29593daec)

---

<div class="post-metadata">

**Author:** ![AntonioSousa](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/antoniosousa/32/50_2.png) [@AntonioSousa](https://community.dynatrace.com/u/AntonioSousa)\
**Post date:** [December 23, 2024, 11:15pm UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/2 "2024-12-23T23:15:12Z")

</div>

Very funny!

You are right in that the attacker might be playing special tricks. The IPs are very credible 😁, but the browser version reveals some things. Although a Firefox user, I was not the one 🤣

---

<div class="post-metadata">

**Author:** ![PacoPorro](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/pacoporro/32/36087_2.png) [@PacoPorro](https://community.dynatrace.com/u/PacoPorro)\
**Post date:** [December 24, 2024, 8:10am UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/3 "2024-12-24T08:10:36Z")

</div>

Nice one!!!

---

<div class="post-metadata">

**Author:** ![marina\_pollehn](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/marina_pollehn/32/39120_2.png) [@marina\_pollehn](https://community.dynatrace.com/u/marina_pollehn)\
**Post date:** [December 24, 2024, 9:46am UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/4 "2024-12-24T09:46:06Z")

</div>

Love it! We need more scavenger hunts like this one \<3.

---

<div class="post-metadata">

**Author:** ![DanielS](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@DanielS](https://community.dynatrace.com/u/DanielS)\
**Post date:** [December 24, 2024, 3:29pm UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/5 "2024-12-24T15:29:54Z")

</div>

Brilliant!!!

---

<div class="post-metadata">

**Author:** ![mizso74](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/mizso74/32/41729_2.png) [@mizso74](https://community.dynatrace.com/u/mizso74)\
**Post date:** [December 24, 2024, 4:21pm UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/6 "2024-12-24T16:21:56Z")

</div>

Excellent!!!

---

<div class="post-metadata">

**Author:** ![Peter\_Youssef](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/peter_youssef/32/14668_2.png) [@Peter\_Youssef](https://community.dynatrace.com/u/Peter_Youssef)\
**Post date:** [December 25, 2024, 6:57am UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/7 "2024-12-25T06:57:54Z")

</div>

Thanks&nbsp;[@Tiit\_Hallas](https://community.dynatrace.com/user/56833)&nbsp;for insightful and organized contents.

---

<div class="post-metadata">

**Author:** ![zietho](https://avatars.discourse-cdn.com/v4/letter/z/94ad74/32.png) [@zietho](https://community.dynatrace.com/u/zietho)\
**Post date:** [December 25, 2024, 8:02am UTC](https://community.dynatrace.com/t/christmas-hands-on-who-stole-santa-s-secret-file/49481/8 "2024-12-25T08:02:02Z")

</div>

I hope your trees don’t look like the one from the investigation, or otherwise, they will likely fall when your cat visits them.

[@Tiit\_Hallas](https://community.dynatrace.com/user/56833)&nbsp;hilarious! kudos!