The “System Profile” in Dynatrace contains the following:
1) Configuration of all the agents of the system under instrumentation
2) Configuration of the sensors and measures necessary to capture data from those agents
3) Configuration of the alerts, and monitoring of key business transactions of the system under monitoring
4) Container of the data collected from all of the connected agents of this profile, for the system under monitoring
The System Profile then, is both the container of all of the config and all of the data collected for a system under monitoring/instrumentation.
A system is collection of related processes that communicate with each other on an ongoing basis. Each related process in turn, processes some part of a transaction in that system.
No, not always.
There is one key factor to keep in mind when designing a System Profile: when we place a Dynatrace agent on a process (JVM, w3wp, Apache, etc.), that agent is bound to the System Profile in two ways:
1) The configuration of the agent is in the profile, and can only be in one profile; one agent instrumenting one process cannot be configured differently from different profiles
2) The data the agent collects is processed by the Dynatrace Server, and stored on the Dynatrace Server, under only the System Profile which configured the agent in the first place; i.e., the System Profile is not only the container of agent config, but agent data
As such, while it is tempting to create one System Profile per application, if the application in question is merely part of a larger system of applications, and if these applications communicate with each other on the backend, or share components, then it is best to create one (larger) System Profile that contains all the agents of these applications, so that the transactions can be captured in an end-to-end fashion.
This decision has some pros and cons:
1) Easier to manage access to the data contained in that profile, from that application
2) Easier to find ‘your’ data for a specific application, as you won’t have to look for it within the bigger set of data from the other applications
1) The transaction may not be captured in an end-to-end fashion because if the transaction transitions from one agent to another, and the other agent is in a different System Profile, then in essence, Dynatrace will treat that as a call to an ‘external’ system, and won’t necessarily follow it across the boundary you’ve defined (i.e., the other system); as such, the PurePath would be short and potentially incomplete, as it would reflect the call to the other system profile agents as an ‘external’ call to an ‘external’ system
2) One System Profile means various team members from different applications would all need the same level of access to all the data in that profile, regardless of whether they care to see data from other apps that they are not directly involved with/part of
On that last point, Dynatrace does try to help: It can group transactions within one System Profile by some combination of Host/WebapplicationID/URL/URI. As such, when a user of App A logs into Dynatrace, they can easily find and only focus on transactions that initiate in their application, without having to dig through everyone else’s transactions first.
Given the Cons of the ‘one app per profile’ approach, we recommend less profiles, but bigger profiles that contain all (or as much of) the agents that communicate with each other, and share components with each other, as part of a larger system. In essence, you have a “system”, not an “app”, let’s instrument it as such.
While deploying monitoring for a large company, in which a very large number of applications communicate with one another, I have struggled with this topic. I.e., where to draw the line in terms of grouping applications within profiles. I have a few questions that relate to your post:
1. You state that when a transaction starts in one profile (lets call it Profile A) and transitions into an application monitored by Profile B, that dynaTrace "won’t necessarily follow it across the boundary you’ve defined". This statement implies that the transaction will, in some cases, be traceable accross the profile boundary. Can you clarify in which cases a transaction will/will not be traceable between profiles? I had been under the impression that it would generally (if not always) be possible to follow a PurePath that starts in one monitoring profile into another... i.e., that you would be able to trace the call stack into the external profile.
2. If a transaction starts in one monitoring profile (Profile A) and transitions to another profile (Profile B), will a PurePath for the transaction be found within Profile B? I.e., will Profile B detect the transaction and start a PurePath despite the fact that a PurePath for the transaction was started in Profile A? I had once been told that no PurePath would be found in Profile B, since the PurePath started in Profile A and would only be found there.
3. If a transaction starts in one monitoring profile (Profile A) and transitions to another profile (Profile B), will we "see" this transaction at all in Profile B? I.e., will we see a Web Request for the transaction, and will the Transaction Based Measures count/measure the transaction?
With regards to (1) as of DT 6.1, the product does not officially support full transaction tracing across System Profiles. While this may happen, it's not an officially supported feature at this time. As such, we don't have specifics on the criteria under which it does or does not happen. Hence, to ensure proper end-to-end tracing, it is recommended that shared applications be instrumented with agents that reside in the same profile.
Similarly for (2) and (3), again given that it's not a supported feature of the product to have PurePaths jump across Profiles and continue tracing, it's not definitive as to how it behaves. From experience I can tell you that in most cases, the PurePath resides in Profile A under the scenario you describe, but there is no guarantee that it will a the moment. Assuming the PurePath remains in A, so will its Transaction Based Measures data.
I'll say it a little bit stronger than "not officially supported". The only supported configuration when agents being monitored on a single dynaTrace server communicate with each other is that is to include those agents into a single System Profile. Doing any other way results in behavior that is undefined. Just because you see something today does not mean that you will see it tomorrow. So to your point #1, maybe you're always seeing it happen but you can't count on it.
To your points 2 and 3: If/when the PurePath transitions from Profile A to Profile B, the current exhibited behavior (but you can't count on it!) is that Profile A would "own" the PurePath and there would be no evidence of it at all in Profile B. And of course you may easily have the absence of certain data from Profile B in Profile A. For example, how would profile A be expected to know anything (for filtering, etc) about agents or measures defined in Profile B?
Bottom line: for agents monitored by the same dT server, the only defined and supported behavior is the single Profile approach.
I can offer one suggestion though. I'm not sure if this is appropriate for you but I've used this approach in the past, in a specific situation.
Suppose you have two distinct "flavors" of agents in your app. Suppose one set of (just an example) 10 agents define what you consider to be the end-user application (such as an online banking app, or an airline reservation system, and another set of (again, just an example) 100 agents define a set of "utility" app such as web services, that are used by the end-user app. In that case you may want to have an app team deal with the app, and a "common services" team deal with the service calls. In fact, the "app" folks may not care at all to go "end-to-end" all the way into the web service code - they don't own it and there's nothing they can do with it anyhow. For them, it's fine to see the call to the web service and they're fine just knowing the response time of that "black box". In this case, you can set up 2 dynaTrace servers. On dT server 1 you would monitor the "app agents". On dT server 2 you would monitor the "services agents". Now the PurePath would start on dT server 1 in Profile "App". When a call was made to one of the web services, a new PurePath would start on dT server 2 in Profile "Services". Each team (App and Services) would manage their own code. Again, there are plusses and minuses to this approach, but the good news is that if this works for you, it is both defined and supported.
Thank you very much, Kia Rob, for your clarifications. I found the suggestion posted by Rob above, for shared service monitoring, to be particularly interesting. Are there any plans to address these limitations in a future release? I.e., it would be of great value to be able to have a profile for a shared service application capture PurePaths and realted measures in cases where the profile is on the same DT server as the applications that consume the service.
just to add some more information about Cross-System-Profile-PurePaths (XSPPPs) here:
hope this helps,
That's fantastic news that you've shared, Christian. And I didn't know we had an official acronym for our "feature": XSPPP. Nice.
I was using the "undefined" in the external-facing way - meaning, if you use it, you are at your own risk. It's my experience too that it always works this way. You just can't count on it (yet, apparently).
I love the new 6.1 debug flag. Very nice!
I think for XSPPPs there is limitation on dashlets like the web requests, web services. It shows only the data from the originating profile. We had to combine system profiles to get all details.
actually I could not think of a reason why that would be the case. PurePaths are always analyzed as a whole.
I also just double-checked that the whole XSPPP is analyzed in the originating system profile and its dashlets.
Having all agents groups in single profile is not always possible as various applications/agent grps might be owned by different department in same org.
e.g. we have close 600+ agents in a multichannel retail environment. it consists of many applications - eCOM website, POS store system, various SOA service. All these services owned by different department. Each department wants to monitor SLA, transactions and report on their performance.
Let me give an example:
Profile A( POS system), Profile B - eCOM, Profile C - Order Management and profile X - a SOA service that A, B and C are using.
I have requirement to report on system X performance. I have created measures/BT in profile X but it will not have any data once purepath start from proflie A, B or C in production.
Right now, we are able to see complete purepath, web requests, web methods , BT and Measures in original profile only. Details of profile X methods/call stack will appear as grey.
splitting XSPPP at system profile boundary level will solve monitoring and system health report by each department but we will loose dynatrace main attraction i.e. distributed system purepath in single view.
We will also loose 1:1 visibility in inter tier time, errors etc. Although having same DT tag will solve error correlation for each purepath.
I think since we have all data in single dynatrace server, we should add features to maintain "distributed system purepath in single view" .
in my above example:
if profile A, B or C user start analysis, he should see all details, BT, Measures related to profile A, B, or C. details of prifile X cab grey out[ this is as-is]
if Profile X user logins, he should see complete purepath details[ grey out details that are outside of profile X), measures etc.
I'm intrigued by the unofficial debug flag "com.dynatrace.diagnostics.enableCrossSystemProfileSplit" as I've been struggling with system profile boundaries for a long time (as many others have I think).
However, I'm not sure I properly understand the consequences of enabling it. Does that mean that I will lose all capability to exactly correlate requests spanning system profiles?
I mean, I could probably live with an option that would at least let me (easily) find (Sub-)Purepaths originally belonging to the same distributed transaction but which were created (and stored) separately as a result of the forced splitting...
Ideally, I think it should be possible to configure "custom split points" as part of the Dynatrace server configuration but without discarding request correlation information (HTTP tagging etc.). IMO this would allow for example to still automatically correlate Purepaths which have crossed system profiles but which eventually return to the original system profile that contained the entry point. This is commonly the case in SOA environments where all service calls are mediated by some middleware component. In that case it would probably be okay if the middleware component would show up as an "External Purepath" but with a returning path. And if need be one could still manually request to "jump" to that External Purepath while in the tree view (as an example).
I hope I make sense 🙂
PS: The reasons why having one big system profile is not feasible for us are:
I think in your case - and for many other user cases - one approach dynaTrace could take, is to think about further application related configurations, like we have it for "User Experience". There we can set everything on a "per application" base. If we would have that for example also for user rights, not only to limit on system profile, but also on application level, I guess this would help in the most cases.
PS: Technically how it could work is e.g. in a first step to preselect the filter of a dashboard. Let's assume User A only has rights to see data of Application "easyTravel". Then every dashlet or dashboard he opens could have a preselected filter of "easyTravel" which is fixed and cannot be adjusted by the user. This way the whole technical background does not need to change, but something which is already there could be used. In this case it would just be necessary to think about someone who exports a dashboard as XML, changing the filter in the XML, and reimporting the dashboard. So the user rights should be always checked on charting the measures on the dashboard ;-). Just an idea ..
could you have an official status of this com.dynatrace.diagnostics.enableCrossSystemProfileSplit parameter because a lot of people is very insteresting?
Is it Ok to use it with 6.1 version for production environment?
We have multiple application in one big System Profile. We would like to split the system profile at the same time we want to get E2E transaction flow. If we enable
com.dynatrace.diagnostics.enableCrossSystemProfileSplit parameter, it helps? What is the support from DynaTrace?
Note : We are in 6.2
We added the com.dynatrace.diagnostics.enableCrossSystemProfileSplit environment property to our collector and it seems that purepaths that are crossing agent groups are also split (within the same System profile).
Agent group 1 contains an agent A1
Agent group 2 contains an agent A2
A1 is calling A2 with an HTTP connection.
Without enableCrossSystemProfileSplit , you get 1 purepath showing the whole transaction (with nodes from A1 and A2)
With enableCrossSystemProfileSplit set to true, you get 2 purepaths one on A1 the other on A2.
That would be the expected behaviour between 2 System profile, but not 2 agent groups inside a same system profile.
Is this normal ?
We created a SUPDT-17275 for this.
Hello Kia, Rob and Christian,
It is 2 years later and I am wondering if anything has changed. I have a system to deploy with 750 agents, to have them all in the same system profile is going to make it very unmanageable.
The original post was written in cons and pros style, reality is, I have requirements not "favorables". Even if we can manage 750 agents in one System profile, I do not know if there is a capability to define users, roles and groups based on Applications not System profiles. If I cannot do so, I will not be able to fulfill the requirements of security on teams' access to data. So is there a way?
Also, is the debug flag officially supported yet? Is there any documentation about it as of what and what is possible?
Thanks in advance,
Officially nothing has changed but I do have clients who have split up Profiles in 6.5+ and seen success with PurePath Stitching across profiles (the complete PurePath would reside in the originating Profile). No config changes were necessary as far as I am aware.
I recommend trying this out on the latest release to see if it meets your needs.
In the large deployment that I work with one of the main reasons that we had to utilize a single profile is that there is allot of inter-connectedness in the applications being monitored... and if we split across multiple profiles we would completely loose any visibility to transactions within the "downstream" profiles... I.e., as Kia notes above, "the complete PurePath would reside in the originating Profile"... I take it that nothing has changed in the architecture with 6.x that would allow us to "see" the transactions in the downstream profile(s), or has it? Is there any plan to change the design/behavior to allow this visibility to occur?.