cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

System Profile Design - Guidelines and Recommendations

kia_forooghi
Dynatrace Helper
Dynatrace Helper

System Profile 101

 

The “System Profile” in Dynatrace contains the following:

 

1)      Configuration of all the agents of the system under instrumentation

2)      Configuration of the sensors and measures necessary to capture data from those agents

3)      Configuration of the alerts, and monitoring of key business transactions of the system under monitoring

4)      Container of the data collected from all of the connected agents of this profile, for the system under monitoring

 

The System Profile then, is both the container of all of the config and all of the data collected for a system under monitoring/instrumentation.

 

What’s a System?

 

A system is collection of related processes that communicate with each other on an ongoing basis. Each related process in turn, processes some part of a transaction in that system.

 

Is a System the same as an Application?

 

No, not always.

 

There is one key factor to keep in mind when designing a System Profile: when we place a Dynatrace agent on a process (JVM, w3wp, Apache, etc.), that agent is bound to the System Profile in two ways:

1)      The configuration of the agent is in the profile, and can only be in one profile; one agent instrumenting one process cannot be configured differently from different profiles

2)      The data the agent collects is processed by the Dynatrace Server, and stored on the Dynatrace Server, under only the System Profile which configured the agent in the first place; i.e., the System Profile is not only the container of agent config, but agent data

 

As such, while it is tempting to create one System Profile per application, if the application in question is merely part of a larger system of applications, and if these applications communicate with each other on the backend, or share components, then it is best to create one (larger) System Profile that contains all the agents of these applications, so that the transactions can be captured in an end-to-end fashion.

 

What happens if we decide to have one System Profile per Application?

 

This decision has some pros and cons:

 

Pros:

1)      Easier to manage access to the data contained in that profile, from that application

2)      Easier to find ‘your’ data for a specific application, as you won’t have to look for it within the bigger set of data from the other applications

 

Cons:

1)      The transaction may not be captured in an end-to-end fashion because if the transaction transitions from one agent to another, and the other agent is in a different System Profile, then in essence, Dynatrace will treat that as a call to an ‘external’ system, and won’t necessarily follow it across the boundary you’ve defined (i.e., the other system); as such, the PurePath would be short and potentially incomplete, as it would reflect the call to the other system profile agents as an ‘external’ call to an ‘external’ system

2)      One System Profile means various team members from different applications would all need the same level of access to all the data in that profile, regardless of whether they care to see data from other apps that they are not directly involved with/part of

 

On that last point, Dynatrace does try to help: It can group transactions within one System Profile by some combination of Host/WebapplicationID/URL/URI. As such, when a user of App A logs into Dynatrace, they can easily find and only focus on transactions that initiate in their application, without having to dig through everyone else’s transactions first.

 

Summary

 

Given the Cons of the ‘one app per profile’ approach, we recommend less profiles, but bigger profiles that contain all (or as much of) the agents that communicate with each other, and share components with each other, as part of a larger system. In essence, you have a “system”, not an “app”, let’s instrument it as such.

 

 

_____________________________________

 

 

 

20 REPLIES 20

jeff_rowell
Helper

While deploying monitoring for a large company, in which a very large number of applications communicate with one another, I have struggled with this topic. I.e., where to draw the line in terms of grouping applications within profiles.  I have a few questions that relate to your post:

1. You state that when a transaction starts in one profile (lets call it Profile A) and transitions into an application monitored by Profile B, that dynaTrace "won’t necessarily follow it across the boundary you’ve defined". This statement implies that the transaction will, in some cases, be traceable accross the profile boundary. Can you clarify in which cases a transaction will/will not be traceable between profiles? I had been under the impression that it would generally (if not always) be possible to follow a PurePath that starts in one monitoring profile into another... i.e., that you would be able to trace the call stack into the external profile.

2. If a transaction starts in one monitoring profile (Profile A) and transitions to another profile (Profile B), will a PurePath for the transaction be found within Profile B? I.e., will Profile B detect the transaction and start a PurePath despite the fact that a PurePath for the transaction was started in Profile A? I had once been told that no PurePath would be found in Profile B, since the PurePath started in Profile A and would only be found there.

3. If a transaction starts in one monitoring profile (Profile A) and transitions to another profile (Profile B), will we "see" this transaction at all in Profile B? I.e., will we see a Web Request for the transaction, and will the Transaction Based Measures count/measure the transaction?

kia_forooghi
Dynatrace Helper
Dynatrace Helper

Hi Jeff,

With regards to (1) as of DT 6.1, the product does not officially support full transaction tracing across System Profiles. While this may happen, it's not an officially supported feature at this time. As such, we don't have specifics on the criteria under which it does or does not happen. Hence, to ensure proper end-to-end tracing, it is recommended that shared applications be instrumented with agents that reside in the same profile.

Similarly for (2) and (3), again given that it's not a supported feature of the product to have PurePaths jump across Profiles and continue tracing, it's not definitive as to how it behaves. From experience I can tell you that in most cases, the PurePath resides in Profile A under the scenario you describe, but there is no guarantee that it will a the moment. Assuming the PurePath remains in A, so will its Transaction Based Measures data.

Kia

 

rob_vollum
Dynatrace Pro
Dynatrace Pro

Hi Jeff,

I'll say it a little bit stronger than "not officially supported". The only supported configuration when agents being monitored on a single dynaTrace server communicate with each other is that is to include those agents into a single System Profile. Doing any other way results in behavior that is undefined. Just because you see something today does not mean that you will see it tomorrow. So to your point #1, maybe you're always seeing it happen but you can't count on it.

To your points 2 and 3: If/when the PurePath transitions from Profile A to Profile B, the current exhibited behavior (but you can't count on it!) is that Profile A would "own" the PurePath and there would be no evidence of it at all in Profile B. And of course you may easily have the absence of certain data from Profile B in Profile A. For example, how would profile A be expected to know anything (for filtering, etc) about agents or measures defined in Profile B?

Bottom line: for agents monitored by the same dT server, the only defined and supported behavior is the single Profile approach.

I can offer one suggestion though. I'm not sure if this is appropriate for you but I've used this approach in the past, in a specific situation.

Suppose you have two distinct "flavors" of agents in your app. Suppose one set of (just an example) 10 agents define what you consider to be the end-user application (such as an online banking app, or an airline reservation system, and another set of (again, just an example) 100 agents define a set of "utility" app such as web services, that are used by the end-user app. In that case you may want to have an app team deal with the app, and a "common services" team deal with the service calls. In fact, the "app" folks may not care at all to go "end-to-end" all the way into the web service code - they don't own it and there's nothing they can do with it anyhow. For them, it's fine to see the call to the web service and they're fine just knowing the response time of that "black box". In this case, you can set up 2 dynaTrace servers. On dT server 1 you would monitor the "app agents". On dT server 2 you would monitor the "services agents". Now the PurePath would start on dT server 1 in Profile "App". When a call was made to one of the web services, a new PurePath would start on dT server 2 in Profile "Services". Each team (App and Services) would manage their own code. Again, there are plusses and minuses to this approach, but the good news is that if this works for you, it is both defined and supported.

Rob

 

jeff_rowell
Helper


Thank you very much, Kia Rob, for your clarifications. I found the suggestion posted by Rob above, for shared service monitoring, to be particularly interesting. Are there any plans to address these limitations in a future release? I.e., it would be of great value to be able to have a profile for a shared service application capture PurePaths and realted measures in cases where the profile is on the same DT server as the applications that consume the service.

c_schwarzbauer
Dynatrace Champion
Dynatrace Champion

hi guys,

just to add some more information about Cross-System-Profile-PurePaths (XSPPPs) here:

  • though not officially supported the behavior actually is defined: a PurePath starting in Profile A and transitioning into another Profile B will always be completely contained in Profile A. this means that Profile B will not know about it and the whole PurePath will be correlated & analyzed in Profile A.
  • this behavior might be changed in the future, but knowing that some customers use this as a "feature" we will definitely not change this behavior without giving it lots of thoughts upfront
  • there's a debug option on the Collector since 6.1 (com.dynatrace.diagnostics.enableCrossSystemProfileSplit) which allows to split those XSPPPs at System Profile boundaries (same behavior as for Cross-Server-PurePaths, AKA XSPPs (smile)). it's no officially supported feature yet, but I'm pretty sure it will become in the future.

hope this helps,
Christian 

rob_vollum
Dynatrace Pro
Dynatrace Pro

That's fantastic news that you've shared, Christian. And I didn't know we had an official acronym for our "feature": XSPPP. Nice. (smile)

I was using the "undefined" in the external-facing way - meaning, if you use it, you are at your own risk. It's my experience too that it always works this way. You just can't count on it (yet, apparently).

I love the new 6.1 debug flag. Very nice!

Rob

sreerag_mooleka
Inactive

I think for XSPPPs there is limitation on dashlets like the web requests, web services. It shows only the data from the originating profile. We had to combine system profiles to get all details.

-Sreerag

c_schwarzbauer
Dynatrace Champion
Dynatrace Champion

hi Sreerag,

actually I could not think of a reason why that would be the case. PurePaths are always analyzed as a whole.

I also just double-checked that the whole XSPPP is analyzed in the originating system profile and its dashlets.

best,
Christian 

aftab_alam
Organizer

Hi Christian,

Having all agents groups in single profile is not always possible as various applications/agent grps might be owned by different department in same org.

e.g. we have close 600+ agents in a multichannel retail environment. it consists of many applications - eCOM website, POS store system, various SOA service. All these services owned by different department.  Each department wants to monitor SLA, transactions and report on their performance.

Let me give an example:

Profile A( POS system), Profile B - eCOM, Profile C - Order Management and profile X - a SOA service that A, B and C are using.

I have requirement to report on system X performance. I have created measures/BT in profile X but it will not have any data once purepath start from proflie A, B or C in production.

Right now, we are able to see complete purepath, web requests, web methods , BT and Measures in original profile only. Details of profile X methods/call stack will appear as grey.

splitting XSPPP at system profile boundary level will solve monitoring and system health report by each department but we will loose dynatrace main attraction i.e. distributed system purepath in single view.

 

 We will also loose 1:1 visibility in inter tier time, errors etc. Although having same DT tag will solve error correlation for each purepath. 

 I think since we have all data in single dynatrace server, we should add features to maintain "distributed system purepath in single view" .

in my above example:

if profile A, B or C user start analysis,  he should see all details, BT, Measures related to profile A, B, or C. details of prifile X cab grey out[ this is as-is](tick)

if Profile X user logins, he should see complete purepath details[ grey out details that are outside of profile X), measures etc.

 

Thanks

Aftab

Enrico_F
Pro

I'm intrigued by the unofficial debug flag "com.dynatrace.diagnostics.enableCrossSystemProfileSplit" as I've been struggling with system profile boundaries for a long time (as many others have I think).

However, I'm not sure I properly understand the consequences of enabling it. Does that mean that I will lose all capability to exactly correlate requests spanning system profiles?

I mean, I could probably live with an option that would at least let me (easily) find (Sub-)Purepaths originally belonging to the same distributed transaction but which were created (and stored) separately as a result of the forced splitting...

Ideally, I think it should be possible to configure "custom split points" as part of the Dynatrace server configuration but without discarding request correlation information (HTTP tagging etc.). IMO this would allow for example to still automatically correlate Purepaths which have crossed system profiles but which eventually return to the original system profile that contained the entry point. This is commonly the case in SOA environments where all service calls are mediated by some middleware component. In that case it would probably be okay if the middleware component would show up as an "External Purepath" but with a returning path. And if need be one could still manually request to "jump" to that External Purepath while in the tree view (as an example).

I hope I make sense 🙂

Enrico

PS: The reasons why having one big system profile is not feasible for us are:


  • Access permissions and incidents need to be defined and managed on a per-application basis (it's not enough to be able to filter results by application in dashboards, there needs to be some separation according to application context in the management of the system profile, too)
  • Managing a big number of agent groups becomes difficult very quickly (i.e. the UI does not support rearranging or creating "super groups" for applications)
  • As inter-connectivity and dependencies between components keep increasing the tendency with the recommended approach is to have a single huge system profile which IMO becomes unmanagable very quickly.
  • Hot Sensor Placement can only be forced for the whole system profile (AFAIK)
  • Incidents: Downtimes need to be defined per-application (currently this is per system profile)

dennis_mispelba
Inactive

Hi Enrico,

  • Hot Sensor placement also works on a "per agent" level from the "Agent overview" dashlet and right click on the agent for which you want to place sensors using "Hot sensor placement".
  • Incident downtimes can be either set on a system profile level or for each contained incident rule itself. I agree though having a lot of incidents in one system profile would be maintenance overhead

I think in your case - and for many other user cases - one approach dynaTrace could take, is to think about further application related configurations, like we have it for "User Experience". There we can set everything on a "per application" base. If we would have that for example also for user rights, not only to limit on system profile, but also on application level, I guess this would help in the most cases.

Best regards,
Dennis

PS: Technically how it could work is e.g. in a first step to preselect the filter of a dashboard. Let's assume User A only has rights to see data of Application "easyTravel". Then every dashlet or dashboard he opens could have a preselected filter of "easyTravel" which is fixed and cannot be adjusted by the user. This way the whole technical background does not need to change, but something which is already there could be used. In this case it would just be necessary to think about someone who exports a dashboard as XML, changing the filter in the XML, and reimporting the dashboard. So the user rights should be always checked on charting the measures on the dashboard ;-). Just an idea ..

Link to the application level user rights RFE: https://community.dynatrace.com/community/display/DTFORUM/RFE+-+User+Access+at+Application+level

Arnaud_GERMAIN
Organizer

Hi,

could you have an official status of this com.dynatrace.diagnostics.enableCrossSystemProfileSplit parameter because a lot of people is very insteresting?
Is it Ok to use it with 6.1 version for production environment?
Thanks,

Arnaud

mohaideen_shand
Participant

Hi,

We have multiple application in one big System Profile. We would like to split the system profile at the same time we want to get E2E transaction flow. If we enable

com.dynatrace.diagnostics.enableCrossSystemProfileSplit parameter, it helps? What is the support from DynaTrace?

Note : We are in 6.2

Arnaud_GERMAIN
Organizer

Hi,

We added the com.dynatrace.diagnostics.enableCrossSystemProfileSplit environment property to our collector and it seems that purepaths that are crossing agent groups are also split (within the same System profile).

Example :
Agent group 1 contains an agent A1
Agent group 2 contains an agent A2

A1 is calling A2 with an HTTP connection.

Without enableCrossSystemProfileSplit , you get 1 purepath showing the whole transaction (with nodes from A1 and A2)

With enableCrossSystemProfileSplit set to true, you get 2 purepaths one on A1 the other on A2.

That would be the expected behaviour between 2 System profile, but not 2 agent groups inside a same system profile.

Is this normal ?

We created a SUPDT-17275 for this.

Hello Arnaud, did you get any answer/solution for that?

alfred_raouf3
Inactive

Hello Kia, Rob and Christian,

It is 2 years later and I am wondering if anything has changed. I have a system to deploy with 750 agents, to have them all in the same system profile is going to make it very unmanageable.

The original post was written in cons and pros style, reality is, I have requirements not "favorables". Even if we can manage 750 agents in one System profile, I do not know if there is a capability to define users, roles and groups based on Applications not System profiles. If I cannot do so, I will not be able to fulfill the requirements of security on teams' access to data. So is there a way?

Also, is the debug flag officially supported yet? Is there any documentation about it as of what and what is possible?

Thanks in advance,
Alfred

kia_forooghi
Dynatrace Helper
Dynatrace Helper

Hi Alfred,

Officially nothing has changed but I do have clients who have split up Profiles in 6.5+ and seen success with PurePath Stitching across profiles (the complete PurePath would reside in the originating Profile). No config changes were necessary as far as I am aware.

I recommend trying this out on the latest release to see if it meets your needs.

Kia

richard_kryspin
Newcomer

Is there a known large scale DT implementation with Single Profile?

jeff_rowell
Helper

In the large deployment that I work with one of the main reasons that we had to utilize a single profile is that there is allot of inter-connectedness in the applications being monitored... and if we split across multiple profiles we would completely loose any visibility to transactions within the "downstream" profiles... I.e., as Kia notes above, "the complete PurePath would reside in the originating Profile"... I take it that nothing has changed in the architecture with 6.x that would allow us to "see" the transactions in the downstream profile(s), or has it? Is there any plan to change the design/behavior to allow this visibility to occur?.