You use the total memory or the selective memory dashlet within the AppMon client to analyze heap dumps (located under Runtime Analysis -v7 or Analyze Runtime -v6). I would suggest using the Total Memory dashlet and creating a memory snapshot. You will have two options for creating dumps: Deep Memory Leak and Memory Consumption Trending. The trending can be used if you are not sure if there is a leak in memory and can help you to see how classes are growing over time. The deep analysis can be used if you want to know the full heap data for pinpointing specific issues within your classes.
Once the dump is created you can tell your server to analyze it and then you will be able to go through the tabs at the bottom to see information such as hotspots, duplicate strings and warnings.
For more information, you can see this link: https://www.dynatrace.com/support/doc/appmon/application-monitoring/appmon-client/dashlets/total-memory-dashlet/
Is there a way to set a condition for JVM or .Net process which can automatically trigger a memory dump? e.g. If the memory utilization of the process goes beyond 4gig, generate a memory dump or if there are more than 5 hung threads for JVM, then generate a memory dump
Yes, it is possible to automatically trigger a dump. You will have to configure a measure to track the metrics that you would like to use as a trigger for your dump, something like memory used or thread count should work (System Profile -> Measures -> Create Measure). You can then set an upper threshold on these measures and add them to an incident (System Profile -> Incidents). In the 'Actions' tab of the incident configuration, you can select 'Advanced Configuration', then 'add', this will allow you to select from an array of actions when the thresholds on the measure are crossed. Select thread dump or memory dump as you desire and you can also add an email alert so that you can take action as well.