Prerequisites

The Big Data Business Transaction Bridge has been developed and tested using Flume 1.3.1 and Flume 1.6.0. Flume can be used to send, route and store the data anywhere. Flume is both configurable and extendable but already contains a lot of out of the box channels (ways to send data) and sinks (places to store data).

This page will focus on two areas of storage. In the first section will deal with the flume setup and describe how to use it to store the Business Transaction data in CSV or JSON files. It will also explain how we can route different types of Business Transactions into different files/directories.

The second section will explain how to store the produced files in Hadoop, setup external Hive tables for access and query the data via hive. Basic installation of Hadoop, Hive and Flume beyond a simple local setup is not covered here. See
http://hadoop.apache.org, https://cwiki.apache.org/confluence/display/Hive/GettingStarted and http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html for more information. 

Flume File based Setup with Flume

The supported Flume versions are 1.3.1 and 1.6.0.

For Flume 1.3.1 the Big Data Business Transaction Bridge consists of three CSV and one JSON serializer to store the Business Transaction results.

Flume 1.3.1 already comes with protobuf so you do not need to add it. Flume has to be started with the classpath pointing to 'dtFlume.jar', a configuration file that contains the source, channels and sinks and the name of the agent to execute. The simplest way to do this is to create a shell script like this:

FLUME_HOME=./apache-flume-1.3.1-bin
$FLUME_HOME/bin/flume-ng agent -C dtFlume.jar -n agent1 -c $FLUME_HOME/conf \
-f flume-conf.properties -Dflume.root.logger=INFO,console

You can also just download the attached runFlume.sh which will also attempt to create the data directories found in the flume configuration. For Windows it will look a little different, you can use the attached runFlume.bat as a starting point.

This will start flume as an agent, add the dtFlume.jar to the classpath, execute the agent1 as defined in the given flume-conf.properties configuration file

An example config, which connects a Http Source three File Sinks writing to individual directories looks as follows:

# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel

# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler

# Describe sinks
agent1.sinks.PurePathSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
# once every 10 min
agent1.sinks.PurePathSink.sink.rollInterval = 600
# Force cutoff at 10 MB
agent1.sinks.PurePathSink.sink.rollSize = 10
agent1.sinks.PurePathSink.sink.directory = data/pp
agent1.sinks.PurePathSink.sink.batchSize = 1000
# Attention the key for the serialize on file_roll is a little different
agent1.sinks.PurePathSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder


# Describe sinks
agent1.sinks.UserActionSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
agent1.sinks.UserActionSink.sink.rollInterval = 600
agent1.sinks.UserActionSink.sink.rollSize = 10
agent1.sinks.UserActionSink.sink.directory = data/pa
agent1.sinks.UserActionSink.sink.batchSize = 1000
# Attention the key for the serialize on file_roll is a little different
agent1.sinks.UserActionSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder

# Describe sinks
agent1.sinks.VisitSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
agent1.sinks.VisitSink.sink.rollInterval = 600
agent1.sinks.VisitSink.sink.rollSize = 10
agent1.sinks.VisitSink.sink.directory = data/visit
agent1.sinks.VisitSink.sink.batchSize = 1000
# Attention the key for the serialize on file_roll is a little different
agent1.sinks.VisitSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder

# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapacity = 100

agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapacity = 100

agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapacity = 100


# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel

agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.USER_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = PurePathChannel

The Bridge also contains a special File Sink that extends the default file_roll sink with the ability to define a maximum file size and removes empty files.

Icon

The File Sink does not create the directories it writes to. If those directories do not exist the setup will fail and no data will be written!

The BtSerializers additionally allow to specify a character encoding different from the platform's standard. Any character encoding supported by java is possible:

agent1.sinks.VisitSink.serializer.charset = ISO-8859-2

The Serializers also have two parameters that can be used to configure alternative delimiters if the default values are not desired.

#CSV Field delimiter
agent1.sinks.VisitSink.sink.serializer.delimiter=|
#CSV Array field delimiter
agent1.sinks.VisitSink.sink.serializer.collection-delimiter=;

Alternative Setup - JSON instead of CSV

You can also export Business Transaction data to JSON instead of CSV

agent1.sinks.VisitSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtExportJSONSerializerBuilder

Sample of exported data (JSON)

{    apdex : "1.0",
    application : "easyTravel portal",
    converted : false,
    dimensions : {[-]
        Client Family of Visits : "Desktop Browser",
        Client IP Address of Visit : "170.225.221.40",
        Client Type of Visit : "HTC",
        Client of Visits : "Firefox 16.0",
        Connection Type of Visits : "Broadband (>1500 kb/s)",
        Country of Visits : "United States - Arizona",
        Operating System of Visits : "Windows 7",
        User Experience of Visits : "satisfied"
    },
    endTime : "2013-04-19 08:14:29.871+0000",
    measures : {[-]
        Action Count : "20.0",
        Bounce Rate : "0.0",
        Failed Actions Count : "0.0"
    },
    name : "Detailed Visit Data",
    startTime : "2013-04-19 08:14:25.150+0000",
    type : "VISIT",
    visitId : 102020
}

Hadoop Flume Setup

The Basic setup describes the steps necessary to prepare the Hadoop/Hive environment for the Big Data Business Transaction Bridge, but does not describe how to setup Hadoop/Hive itself. Please refer to http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html for a basic setup of Hadoop. 

The are three different types of Business Transactions, depending on what they are based on:

As these types contain different data, three directories have to be created in HDFS.

$HADOOP_HOME/bin/hadoop fs -mkdir /user/bts/pp
$HADOOP_HOME/bin/hadoop fs -mkdir /user/bts/pa
$HADOOP_HOME/bin/hadoop fs -mkdir /user/bts/visit

$HADOOP_HOME/bin/hadoop fs -chmod g+w /user/bts/pp
$HADOOP_HOME/bin/hadoop fs -chmod g+w /user/bts/pa
$HADOOP_HOME/bin/hadoop fs -chmod g+w /user/bts/visit

Adjust the directory names according to your needs.

In order to let flume write to Hadoop you will need to modify the flume configuration to use the HDFS sink instead of the file sink:

# Name the components on this agent
agent1.sources = HTTPSource
agent1.sinks = PurePathSink UserActionSink VisitSink NullSink
agent1.channels = PurePathChannel UserActionChannel VisitChannel NullChannel

# Describe/configure HTTPSource
agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
agent1.sources.HTTPSource.port = 4321
agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler

# Describe sinks
agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://localhost:9000/user/bts/pp
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072
agent1.sinks.PurePathSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder

agent1.sinks.UserActionSink.type = hdfs
agent1.sinks.UserActionSink.hdfs.path = hdfs://localhost:9000/user/bts/pa
agent1.sinks.UserActionSink.hdfs.fileType = DataStream
agent1.sinks.UserActionSink.hdfs.filePrefix = export
agent1.sinks.UserActionSink.hdfs.fileSuffix = .txt
agent1.sinks.UserActionSink.hdfs.rollInterval = 120
agent1.sinks.UserActionSink.hdfs.rollSize = 131072
agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPageActionSerializerBuilder

agent1.sinks.VisitSink.type = hdfs
agent1.sinks.VisitSink.hdfs.path = hdfs://localhost:9000/user/bts/visit
agent1.sinks.VisitSink.hdfs.fileType = DataStream
agent1.sinks.VisitSink.hdfs.filePrefix = export
agent1.sinks.VisitSink.hdfs.fileSuffix = .txt
agent1.sinks.VisitSink.hdfs.rollInterval = 120
agent1.sinks.VisitSink.hdfs.rollSize = 131072
agent1.sinks.VisitSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtVisitSerializerBuilder

agent1.sinks.NullSink.type = null

# Use a channel which buffers events in memory
agent1.channels.PurePathChannel.type = memory
agent1.channels.PurePathChannel.capacity = 1000
agent1.channels.PurePathChannel.transactionCapactiy = 100

agent1.channels.UserActionChannel.type = memory
agent1.channels.UserActionChannel.capacity = 1000
agent1.channels.UserActionChannel.transactionCapactiy = 100

agent1.channels.VisitChannel.type = memory
agent1.channels.VisitChannel.capacity = 1000
agent1.channels.VisitChannel.transactionCapactiy = 100

agent1.channels.NullChannel.type = memory
agent1.channels.NullChannel.capacity = 1000
agent1.channels.NullChannel.transactionCapacity = 100

# Bind the source and sink to the channel
agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel NullChannel
agent1.sinks.PurePathSink.channel = PurePathChannel
agent1.sinks.UserActionSink.channel = UserActionChannel
agent1.sinks.VisitSink.channel = VisitChannel
agent1.sinks.NullSink.channel = NullChannel

agent1.sources.HTTPSource.selector.type = multiplexing
agent1.sources.HTTPSource.selector.header = btType
agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.USER_ACTION = UserActionChannel
agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
agent1.sources.HTTPSource.selector.default = NullChannel
Icon

Where as the FileSink has additional sink level when defining the serializer the hdfs sink omits this level!

Adjust the directory names according to your HDFS setup. When you start flume with this configuration files the result data will be stored your hadoop cluster.

Hive Setup

Hive is query engine that can execute against file data stored locally or in your hadoop cluster. In order for Hive to access the stored data you will need to use the CSV Serializers and create external tables:

create external table BT_PP (
    Name STRING,
    Application STRING,
    PathId STRING,
    StartTime TIMESTAMP,
    Dimensions MAP<STRING, STRING>,
    Values MAP<STRING, DOUBLE>,
    Failed BOOLEAN,
    VisitId BIGINT,
    ResponseTime DOUBLE,
    Duration DOUBLE,
    CpuTime DOUBLE,
    ExecTime DOUBLE,
    SuspensionTime DOUBLE,
    SyncTime DOUBLE,
    WaitTime DOUBLE,
    SystemProfile STRING,
    ServerName STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' ESCAPED BY '\\' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY '='
LOCATION '/user/bts/pp';

create external table BT_PA (
    Name STRING,
    Application STRING,
    PathId STRING,
    StartTime TIMESTAMP,
    Dimensions MAP<STRING, STRING>,
    Values MAP<STRING, DOUBLE>,
    Failed BOOLEAN,
    ActionName STRING,
    Url STRING,
    VisitId BIGINT,
    ResponseTime DOUBLE,
    Duration DOUBLE,
    CpuTime DOUBLE,
    ExecTime DOUBLE,
    SuspensionTime DOUBLE,
    SyncTime DOUBLE,
    WaitTime DOUBLE,
    ClientErrors INT,
    ClientTime DOUBLE,
    NetworkTime DOUBLE,
    ServerTime DOUBLE,
    UrlRedirectionTime DOUBLE,
    DnsTime INT,
    ConnectTime INT, 
    SslTime INT,
    DocumentRequestTime INT,
    DocumentResponseTime INT,
    ProcessingTime INT,
    SystemProfile STRING,
    ServerName STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' ESCAPED BY '\\' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY '='
LOCATION '/user/bts/pa';

create external table BT_VISIT (
    Name STRING,
    Application STRING,
    VisitId BIGINT,
    StartTime TIMESTAMP,
    EndTime TIMESTAMP,
    Dimensions MAP<STRING, STRING>,
    Values MAP<STRING, DOUBLE>,
    User STRING,
    Converted BOOLEAN,
    Apdex DOUBLE,
    NrOfActions INT,
    ClientFamily STRING,
    ClientIP STRING,
    Continent STRING,
    Country STRING,
    City STRING,
    FailedActions INT,
    ClientErrors INT,
    ExitActionFailed BOOLEAN,
    Bounce BOOLEAN,
    OsFamily STRING,
    OsName STRING,
    ConnectionType STRING,
    ConvertedBy ARRAY<STRING>,
    SystemProfile STRING,
    ServerName STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' ESCAPED BY '\\' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY '='
LOCATION '/user/bts/visit';

You can also run the command below using this file : bigdatabtbridge_table_creation.sql

Query data via Hive

Hive is very sql-like. To query data in a fully setup hadoop/hive environment simply start the hive command line interface via the hive command. You will get a prompt and can execute commands:

hive> show tables;
OK
bt_pa
bt_pp
bt_visit
Time taken: 11.495 seconds
hive>

To select all visit data from today simply execute

select * from BT_VISIT where
datediff(from_unixtime(unix_timestamp()), startTime) < 2;

If you want to get all user actions for a specific user you can execute the following

select pa.* from BT_PA pa join BT_VISIT v on (pa.visitId = v.visitId) where v.user="myname"

Try Hive Locally

in order to try out Hive locally without a full Hadoop setup you should do the following:

Contribution

Feel free to contribute any changes on Github.

62 Comments

Hide/Show Comments
  1. Anonymous (login to see details)

    Has this been tried on Windows as Flume 1.x is only supported on unix like OS ?

    1. Anonymous (login to see details)

      I added a windows start script this should work now, please make sure that FLUME_HOME is correct or that you place the runFlume along side the apache flume directory

  2. Anonymous (login to see details)

    Hi Michael.

    I´m installing and testing the "Business Transaction Bridge" on Redhat and it is running very good.

    I´m running the sample corresponding to exporting to CSV Files (SampleFileSetup-conf.properties)

    My goal is to export different "Business Transactions" to different CSV Files (and in the future to a DB).

    Is there a complete Documentation about how to configure the channels, sinks, etc ?

    Thanks in advance. 

  3. Anonymous (login to see details)

    I'm setting up the business transaction feed to store data in ElasticSearch.  It's working fairly well, but I'm having an issue with getting the agent name into the export.  Even when adding the splitting of 'Agent' in a server-side bt, its not passing into the export.  Any ideas?  Anyone else have this issue?

  4. Anonymous (login to see details)

    Hi Scott.

     

    Are the "Measures" you are including in your "Business Transaction" split by Agent ?

     

    Check if on your "Measure" definition you activated this Option: "Details -> create a measure for each agent"

     

    Regards.

  5. Anonymous (login to see details)

    Hi Juan,

    I thought of this as well and checked those measures to ensure they all had the checkbox checked for agents.

    I've also created a test BT that doesn't have any special measures and have Application and Agent for splittings.  It can be charted in DT client by agent but the agent does not show up in the HTTP export.

  6. Anonymous (login to see details)

    Anyone finding the User Action export feed does not calculate the failure of the user action correctly?  I'm finding user actions failed in Dynatrace but the exported data shows failed of false.

  7. Anonymous (login to see details)

    I am running into data missing issue: both Visit and PP data are missing. Only PA data is there. Do anyone know why? Also, do you know if there are any sample Response message that we could test installation? 

     

    Btw, with latest hadoop and flume, we run into protobuf 2.5 issue. We fixed it by recompile the dtFlume.jar with protobuf 2.5.  

    The Error in the log is like below==============================

    15/05/08 21:29:23 ERROR hdfs.HDFSEventSink: process failed 

    java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RecoverLeaseRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; 
    at java.lang.ClassLoader.defineClass1(Native Method) 

    ...

    ======================================

    1. Anonymous (login to see details)

      Hi!

      Aren't Business Transactions for PP and Visit exported at all or is some data missing on exported PA Business Transactions?

      Best Regards,
      Peter 

      1. Anonymous (login to see details)

        As of now, the PP data still not comes back. Visit data did comes back. 

        So, I was told that I need to setup different BT to get different set of data. I didn't get detail about how so I just guess. I think I make it right with Visit but not PP.  Here is what I did: 

        1. Setup BT for visit: Make BT capture visit data like visit count and so on. In BT, Set "Calculate business Transaction Per" ->  "Visit"
        2. Setup BT for PA: Make BT capture URL response time and W3C navigation timing API. In BT, Set "Calculate business Transaction Per" ->  "User Action"
        3. Setup BT for PP: Make BT capture page object level data like third party content response time: In BT, Set "Calculate business Transaction Per" ->  "User Action"

        Please note, our installation is browser inject UEM only. The data I am expecting to get are visit, page action and page object level response time or that we get report of what kind of page object drive our page load time. (All this are for the client side page performance). 

        1. Anonymous (login to see details)

          Hmm.. That looks like the BT for PP also has been set up for Page Actions in which case PurePath data wouldn't be exported. To me it sounds like there aren't any server side PurePaths so no PurePath data can be exported.

          Best Regards,

          Peter

          1. Anonymous (login to see details)

            Thank you for pointing out that #3 is not for me. 

            Do  you know what i need to do get the page object level response time (- like third party images response time) return? 

            Do you know how I should config my BT so that I could capture the max data that available in the system? For example, i have 3 BTs export PA data. (I created 3 BTs just to test, not intended for getting different data set) I found that different BT return different data for "Values" field:

            1. {"Load Time":4725.0,"Page Actions - Target URI":1.0,"Document request":2321.0,"Document response":1651.0,"First Byte Time":2321.0,"Processing":704.0,"SSL":38.0,"URL redirection":2361.0}
            1. {"Request Start":38.0,"Document Interactive":4088.0,"Document Fetch Done":4010.0,"BW 3rdparty adobedtm Busy time":0.0,"BW 3rdparty atdmt.com busy time":56.0,"Third party content load time":56.0}
            1. {"Document request":2321.0,"Document response":1651.0,"First Byte Time":2321.0,"Processing":704.0,"SSL":38.0,"URL redirection":2361.0,"Apdex of User Actions":0.5}

            I think it would be better that I could create only one BT to capture all PA data so that I don't have redundant data for same user action. 

  8. Anonymous (login to see details)

    Can anyone confirm if all data in one package is from the same date? We are running into performance issue with query data in Hadoop. We are looking to optimize the dtFlume.jar so that Flume will write data into different folder based on message date -- like partition by date. In such, we could improvement both maintainability of data as well as performance of query. If we could confirm that each package data are for the same date, the code logic could be simply look at the first line of message and we don't need to consider breakdown package by message date. We are going to write our custom code to do this. 

    If you have alternative way, please let me know.

    Here is example of the structure we are thinking to create:

    dynatraceUEM/20150515/pp.2015051501.txt
    dynatraceUEM/20150515/pp.2015051502.txt
    dynatraceUEM/20150516/pp.2015051601.txt

    1. Anonymous (login to see details)

      Hi!

      The exported data isn't split by date. So it would be necessary to inspect each BtOccurrence.

      Best Regards,

      Peter

      1. Anonymous (login to see details)

        Thank you for confirm this.

  9. Anonymous (login to see details)

    Is there any data dictionary that I could refer to? 

     

  10. Anonymous (login to see details)

    Question: Is there a way to include the Visit ID in the CSV, file-sink setup?

    Also, because of our huge amount of transactions, we had to modify the TransactionCapacity setting, because Flume was dying. However, the setting wouldn't take. I figured out why: there is a typo in the flume-config.properties file, which is replicated on this page. All lines with the TransactionCapacity entry have it; Capacity is spelled Capactiy:

    agent1.channels.PurePathChannel.transactionCapactiy = 100

    1. Anonymous (login to see details)

      Thanks for telling us about the typo - will fix this on this page

      As for your other question: check out the doc on our data feed - there you find a visitID field that we export for User Action and Visit-based BTs: Real Time Business Transactions Feed

  11. Anonymous (login to see details)

    Hi.

    This plugin does not work for DynaTrace 6.1.

    I have made my own a sollution for apache-flume-1.6.0, hadoop-2.7.1 and the proper dtFlume.jar for Dynatrace 6.1

     

     

    1. Anonymous (login to see details)

      Hi Krzysztof,

      what problem did occur?

      Thanks and Best Regards,

      Peter

      1. Anonymous (login to see details)

        Hi Peter.

        It looks like the protobuf message has been changed between version 5.5 and 6.1.

        After setting up the whole solution, I have had in flume logs messages:

        flume : deserializer threw unexpected exception

        I have properly suspected that the the BtExport.java file is out of date with DT 6.1.

        So I have unpacked the dtFlume.jar, regenerated the BtExport.java from protobuf and then recompile it.

        Now all is working.

        Besides I have one suggestion. Google proptobuf is now avaliable in verstion 3, Yours BtExport.proto is in version 2 but generating the BtExport.java form protobuf 3 gives an unsuable java file, that can not be compiled. You HAVE TO USE older releases of protobuf version 2 (for me it was protoc 2.5.0) or Dynatrace should give the proper proto file that is usable with protobuf v3.

        Regards

        Krzysztof

         

        1. Anonymous (login to see details)

          Hi,

          The "old" dtflume.jar was built using protobuf 2.4.1 and is incompatible with newer versions of flume/hadoop. Yesterday we added a new version which is compatible with protobuf 2.5.0. The protobuf message itself is still compatible with older versions of dynatrace.

          Thanks for the hint regarding protobuf 3, we will take a look at it.

          Best Regards,

          Peter

           

  12. Anonymous (login to see details)

    Hi,

    The intro page says: 

    • Full Detail and Context for every single transaction

    However, in practice I don't think the exported JSON data includes times per agent/database?

    Would it be possible please to include fields corresponding to the granularity of the Agent Breakdown dashboard per transaction?

    I.e. for each transaction, total time spent in each agent (and databases) per metric (e.g. exec, db, CPU, Sync, Wait, Suspension).

    If this is too much data to export all the time, perhaps a filter could be put in the subscription mechanism to allow selected agents/metrics to be exported, or for limited time periods?

    Regards, Paul

     

  13. Anonymous (login to see details)

    Alternatively, has anyone got the splitting by agent to work (with agent name in the exported data?)

    Which field in the message format specification is likely to be the agent name?

    https://community.dynatrace.com/community/display/DOCDT62/Real+Time+Business+Transactions+Feed

    Thanks, Paul

  14. Anonymous (login to see details)

    Does anyone have configured this to work with flume 1.6.0 and shipping data to elastic 2.1.0? I already copy from elastic appropriate elasticsearch-2.1.0.jar and lucene-core-5.3.1.jar to flume but it still do not work. Any ideas?


    Im getting error like this

    2015-12-07 13:57:22,576 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.sink.elasticsearch.ElasticSearchSink.stop(ElasticSearchSink.java:376)] ElasticSearch sink {} stopping
    2015-12-07 13:57:22,576 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:150)] Component type: SINK, name: VisitSink stopped
    2015-12-07 13:57:22,576 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:156)] Shutdown Metric for type: SINK, name: VisitSink. sink.start.time == 1449493042540
    2015-12-07 13:57:22,576 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:162)] Shutdown Metric for type: SINK, name: VisitSink. sink.stop.time == 1449493042576
    2015-12-07 13:57:22,576 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.batch.complete == 0
    2015-12-07 13:57:22,577 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.batch.empty == 0
    2015-12-07 13:57:22,577 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.batch.underflow == 0
    2015-12-07 13:57:22,577 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.connection.closed.count == 1
    2015-12-07 13:57:22,578 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.connection.creation.count == 0
    2015-12-07 13:57:22,578 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.connection.failed.count == 0
    2015-12-07 13:57:22,578 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.event.drain.attempt == 0
    2015-12-07 13:57:22,578 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: VisitSink. sink.event.drain.sucess == 0
    2015-12-07 13:57:22,578 (lifecycleSupervisor-1-3) [WARN - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:260)] Component SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@27c059a9 counterGroup:{ name:null counters:{} } } stopped, since it could not be successfully started due to missing dependencies

    My config looks like this

    # Name the components on this agent
    agent1.sources = HTTPSource
    agent1.sinks = PurePathSink UserActionSink VisitSink
    agent1.channels = PurePathChannel UserActionChannel VisitChannel
     
    # Describe/configure HTTPSource
    agent1.sources.HTTPSource.type = org.apache.flume.source.http.HTTPSource
    agent1.sources.HTTPSource.port = 4321
    agent1.sources.HTTPSource.handler = com.dynatrace.diagnostics.btexport.flume.BtExportHandler
     
    # Describe sinks
    agent1.sinks.PurePathSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
    # once every 10 min
    agent1.sinks.PurePathSink.sink.rollInterval = 600
    # Force cutoff at 10 MB
    agent1.sinks.PurePathSink.sink.rollSize = 10
    agent1.sinks.PurePathSink.sink.directory = /data/pp
    agent1.sinks.PurePathSink.sink.batchSize = 1000
    # Attention the key for the serialize on file_roll is a little different
    agent1.sinks.PurePathSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtExportJSONSerializerBuilder
     
    # Describe sinks
    agent1.sinks.UserActionSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
    agent1.sinks.UserActionSink.sink.rollInterval = 600
    agent1.sinks.UserActionSink.sink.rollSize = 10
    agent1.sinks.UserActionSink.sink.directory = /data/pa
    agent1.sinks.UserActionSink.sink.batchSize = 1000
    # Attention the key for the serialize on file_roll is a little different
    agent1.sinks.UserActionSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtExportJSONSerializerBuilder
     
    # Describe sinks
    agent1.sinks.VisitSink.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink  
    agent1.sinks.VisitSink.hostNames = 192.168.1.50:9200
    agent1.sinks.VisitSink.indexName = flume
    agent1.sinks.VisitSink.indexType = visits
    agent1.sinks.VisitSink.sink.batchSize = 1000
    # Attention the key for the serialize on file_roll is a little different
    agent1.sinks.VisitSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtExportJSONSerializerBuilder
     
    # Use a channel which buffers events in memory
    agent1.channels.PurePathChannel.type = memory
    agent1.channels.PurePathChannel.capacity = 1000
    agent1.channels.PurePathChannel.transactionCapactiy = 100
     
    agent1.channels.UserActionChannel.type = memory
    agent1.channels.UserActionChannel.capacity = 1000
    agent1.channels.UserActionChannel.transactionCapactiy = 100
     
    agent1.channels.VisitChannel.type = memory
    agent1.channels.VisitChannel.capacity = 1000
    agent1.channels.VisitChannel.transactionCapactiy = 100
     
    # Bind the source and sink to the channel
    agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel 
    agent1.sinks.PurePathSink.channel = PurePathChannel
    agent1.sinks.UserActionSink.channel = UserActionChannel
    agent1.sinks.VisitSink.channel = VisitChannel
     
    agent1.sources.HTTPSource.selector.type = multiplexing
    agent1.sources.HTTPSource.selector.header = btType
    agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
    agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
    agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
    agent1.sources.HTTPSource.selector.default = PurePathChannel
     
    # Bind the source and sink to the channel
    agent1.sources.HTTPSource.channels = PurePathChannel UserActionChannel VisitChannel
    agent1.sinks.PurePathSink.channel = PurePathChannel
    agent1.sinks.UserActionSink.channel = UserActionChannel
    agent1.sinks.VisitSink.channel = VisitChannel
     
    agent1.sources.HTTPSource.selector.type = multiplexing
    agent1.sources.HTTPSource.selector.header = btType
    agent1.sources.HTTPSource.selector.mapping.PUREPATH = PurePathChannel
    agent1.sources.HTTPSource.selector.mapping.PAGE_ACTION = UserActionChannel
    agent1.sources.HTTPSource.selector.mapping.VISIT = VisitChannel
    agent1.sources.HTTPSource.selector.default = PurePathChannel

     

     

    1. Anonymous (login to see details)

      Hi!

      Looks like ElasticSearch >= 2.0 isn't supported by the ElasticSearchSink:
      https://github.com/elastic/elasticsearch/issues/14187

      Best Regards,

       Peter

  15. Anonymous (login to see details)

    Did you tested this with ElasticSearch? If so which versions are tested and worked as expected?

    1. Anonymous (login to see details)

      No, this hasn't been tested with ElasticSearch. 

      In the discussion I've linked in my last post, the last post links to a sink that should work with ElasticSearch >= 2.0, maybe that works for you.

      For further information on compatibility issues please refer to the documentations/communities of flume/ElasticSearch.

      Best Regards,

      Peter

      1. Anonymous (login to see details)

        I have ben able to run flume 1.6.0 and elastic 2.1 using this solutions although my output looks like below even when I used serializer  

        agent1.sinks.VisitSink.serializer.charset = ISO-8859-2

        "@fields": {
        "btType": "VISIT",
        "btName": "Conversion visits"
        }
        }
        },
        {
        "_index": "flume-2015-12-08",
        "_type": "visits",
        "_id": "AVGCNCYcg_wWElJBPpuX",
        "_score": 1,
        "_source": {
        "@message": "\n\u0011Conversion visits\u0010\u0002\u001a\u0011easyTravel mobile\"\u000bVisit Count2�\u0001\b��*\u0010١�*1\u0000\u0000\u0000\u0000\u0000\u0000�?Xܟ\u0007i\u0000\u0000\u0000\u0000\u0000\u0000�?p\u0000�\u0001\u0007�\u0001\u0010Samsung Galaxy S�\u0001\u000f113.129.241.106�\u0001\u0006Europe�\u0001\u0007Austria�\u0001\u0004Linz�\u0001\u0001�\u0001\u0001�\u0001\u0000�\u0002\u0000�\u0002\u0007Android�\u0002\rAndroid 2.3.3�\u0002\u0007Unknown2�\u0001\b��뒘*\u0010���*1\u0000\u0000\u0000\u0000\u0000\u0000�?Xԟ\u0007b\u0007onesimei\u0000\u0000\u0000\u0000\u0000\u0000�?p\u0001�\u0001\t�\u0001\u0012Samsung Galaxy Tab�\u0001\u000e200.64.163.119�\u0001\u0006Europe�\u0001\u0006France�\u0001\u0011Neuilly-sur-seine�\u0001\u0000�\u0001\u0000�\u0001\u0000�\u0002\u0000�\u0002\u0007Android�\u0002\u000bAndroid 2.2�\u0002\u0007Unknown�\u0002\u0012easyTravel Payment2�\u0001\b؋���*\u0010�����*1\u0000\u0000\u0000\u0000\u0000\u0000�?X�\u0007i\u0000\u0000\u0000\u0000\u0000\u0000�?p\u0000�\u0001\b�\u0001\u0011Samsung Galaxy S2�\u0001\u000e245.10.179.240�\u0001\u0004Asia�\u0001\u0005Japan�\u0001\u0005Tokio�\u0001\u0000�\u0001\u0000�\u0001\u0000�\u0002\u0000�\u0002\u0007Android�\u0002\rAndroid 2.3.3�\u0002\tBroadband",

        Do you have any idea why is that?

        1. Anonymous (login to see details)

          It looks like the default serializer of the ElasticSearchSink is used.

          The serializer needs to be specified as follows:
           

          agent1.sinks.UserActionSink.serializer = com.dynatrace.diagnostics.btexport.flume.BtExportJSONSerializerBuilder

          Unfortunately different sinks don't use the same key to specify the serializer. E.g. the file_roll sink expects "....sink.serializer" whereas the RollingFileSink provided in dtflume.jar can handle both styles.

           

          Best Regards,

          Peter

          1. Anonymous (login to see details)

            Unfortunately if I changed serializer it dosn't chane the output

            "hits": {

            "total": 3,
            "max_score": 1,
            "hits": [
            {
            "_index": "flume-2015-12-17",
            "_type": "visits",
            "_id": "AVGvN-e86kQVLEoHq2Ax",
            "_score": 1,
            "_source": {
            "@message": "\n\u0011Conversion visits\u0010\u0002\u001a\u0011easyTravel portal\"\u000bVisit Count2�\u0001\b�����*\u0010�ۃ��*1\u0000\u0000\u0000\u0000\u0000\u0000�?XѨ\fi\u0000\u0000\u0000\u0000\u0000\u0000�?p\u0000�\u0001\u0003�\u0001\u0007Firefox�\u0001\u000e84.147.181.217�\u0001\u0006Europe�\u0001\u0007Germany�\u0001\nDürrbrunn�\u0001\u0000�\u0001\u0000�\u0001\u0000�\u0002\u0000�\u0002\u0007Windows�\u0002\tWindows 7�\u0002\u0007DSL low",
            "@fields": {
            "btType": "VISIT",
            "btName": "Conversion visits"
            }
            }
            },

            1. Anonymous (login to see details)

              Ok, I just found out that the ElasticSearchSink cannot be used with any serializer. The serializer would have to implement either ElasticSearchIndexRequestBuilderFactory or ElasticSearchEventSerializer.

              The message you are seeing is the exported data in binary protobuf format.

              Therefore it would be required to write a custom serializer (or maybe adapt the BtExportHandler which is part of the bt export bridge).

               

              Best Regards,

              Peter

  16. Anonymous (login to see details)

    First I must tell you the export data for HuaWei team and they have signed a technical confidentiality agreement with dynatrace before.

    We have export dynatrace data by Big Data Business Transaction Bridge but we should to correlation the PA and PP, so we need ralated id for them, we can only export business transaction data to file from flume but Page Id and Action Id in page action could not to export, I can not to upload the screen shot to you, this is my ticket id : SUPDT-16096, we must take this id to correlation the PA and PP, please help us, thanks you very much!

     

    1. Anonymous (login to see details)

      Hello,

      Unfortunately that's not possible. The business transactions are evaluated during realtime analysis. Linking of server side and end user paths happens at a later time when all paths are available.

      Best Regards,

      Peter

      1. Anonymous (login to see details)

        Hi Peter Kaiser,

        Does this also include using the User Action Purepaths? From the documentation it looks like User Actions contain both a visit reference as well as a purepath reference or am I misreading the table here.

        Best,

        Kyle

        1. Anonymous (login to see details)

          Hi,

          no you are not misreading the table. But here the purePathId refers to the User Action PurePath itself, so it cannot be used to correlate User Action and Server PurePaths.

          Best Regards,

          Peter

          1. Anonymous (login to see details)

            So if I am understanding this correctly, I would be able to correlate 2 User Action purepaths but not a User action to a Server side?

            1. Anonymous (login to see details)

              No, the PurePath reference can be used to find a PurePath on the Dynatrace server. Correlation of PurePaths is not possible using this field.

              E.g. a report which contains data for the PurePath in question could be generated as follows:
              http://localhost:8020/rest/management/reports/create/easyTravel%20PurePaths%20Dashboard?filter=pp:PT=1176499;PA=-1160641869;PS=-1121751920

              PT is the tag id of the PurePath. In the PurePath dashlet it's possible to set the "Tag ID" column visible, search for the exported Tag ID and do further drilldowns from there. That way it's also possible to find linked server PurePaths for a User Action PurePath and vice-versa.

               

              1. Anonymous (login to see details)

                Thanks for the Clarification Peter.

  17. Anonymous (login to see details)

    Hi!

    Has anyone else experienced throughput issues with flume using this?  If I get a spike in traffic flume starts returning a 503 and never recovers until restart.  That was using the normal batch size of 100.  I've tried increasing it to 1000 for now and I'm monitoring resource usage closely (actually stuck a DT agent on it for the time being (smile)).  For the developers:  do you think conceptually it would be more efficient to set batching high or low (essentially streaming it)?

    Thanks,

    Rick B

    1. Anonymous (login to see details)

      Hi!

      Higher batch size would be more efficient as it reduces overhead when sending and processing the data. Each request contains Business Transaction configuration data like name and measure names once per Business Transaction that has data since the last export. In Flume one event will be created per Business Transaction and request. So the more data is exported per Business Transaction in a single request the less configuration data will be sent to and decoded by Flume.

      However, depending on the configuration of the Business Transactions, this could lead to quite large requests which could then cause other problems.

      Which flume version are you using? Do you use a flume configuration as provided on this page? Did you get any specific error messages in the flume logs?

      Best Regards,

      Peter

      1. Anonymous (login to see details)

        I'll try to be more diagnostic about it next time it happens, but it gave me a JSON Parse Exception at some point, then began writing the same PurePaths over and over, while giving the server a 503 on a continuous basis.  I'm using Flume 1.3.1 and am using essentially the configuration on this page

        1. Anonymous (login to see details)

          Had a similar issue and ended up with the below config (has been stable since)...we have 1 BT generating a json o/p (this BT produces a lot of data)...on flume 1.6.0

          agent1.sinks.PurePathSink.sink.batchSize = 100

          agent1.channels.PurePathChannel.capacity = 10000

          Increased heap to 1G

          1. Anonymous (login to see details)

            For clarification, why reduce the sink batch size?

  18. Anonymous (login to see details)

    Hi

    I have an issue with exporting "User actions":

    testing with DTAM 6.3, I could setup the whole chain with Flume 1.6, "dtFlume_1.1.1.jar" and I try to generate flat files according to "SampleFileSetup-conf.properties".

    My BT is exporting User Actions.

    No error or stacktrace in the log, but repeating Warning lines like the following :

     

    WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializer.write:?) - Skipping serialization of event of wrong type: 'USER_ACTION', btName: '_ET_ Test Export Protobuf'

    No data is generated.

    Any hint ?

    Thanks

    --

    Laurent

    1. Anonymous (login to see details)

      Hi!

      The btType "PAGE_ACTION" has been renamed to "USER_ACTION" to match the product.

      Unfortunatly the file based flume config hasn't been updated accordingly. I've done so now and by using the new config or adding the following line to your config everything should work as expected:

      agent1.sources.HTTPSource.selector.mapping.USER_ACTION = UserActionChannel

       

      Best Regards,

      Peter

       

      1. Anonymous (login to see details)

        I highly recommend additionally the use of this guy from the above config, which has kept me from needing to update that field (albeit combining the UA and PP sinks effectively):

        agent1.sources.HTTPSource.selector.default = PurePathChannel

  19. Anonymous (login to see details)

    Hi,

    has anyone used this to export to Elastic Search ?

    Could it be as simple as setting up an "ElasticSearch" sink in flume ?

    Laurent

    1. Anonymous (login to see details)

      I'm using Flume -> JSON filesystem -> Logstash -> Elasticsearch

      I tried the more direct routes but the elastic search sink seemed to have an issue with the BT Serialization if I recall correctly.  The other advantage of this is if I update Logstash to include a new filter or output or something I can drop the indices to reset and re-index the JSON data with the new configuration rather than waiting for new data to see the outcome.

      1. Anonymous (login to see details)

      2. Anonymous (login to see details)

        Hi Rick,

        The current elasticsearch sink for flume v1.6 is not working with the lastest elasticsearch v2.x, due to API incompatibility. I managed to grep some code from github and adjust to the new elasticsearch API and made it work. The data flow is dynaTrace > flume > elasticsearch > kibana.

        Thanks,

        Wei

         

        1. Anonymous (login to see details)

          Hi,

          could you share the details of what you did to make it work with elasticSearch 2 ?

          Thanks

          --

          Laurent

          1. Anonymous (login to see details)

            I rewrite the code of flume elasticsearch link to make it compatible to elasticsearch 2

            1. Anonymous (login to see details)

              Hi,

              would you consider sharing that code you rewrote ?

              Regards

              --

              Laurent

  20. Anonymous (login to see details)

    Hi All,

    When I try and run the batch file to start this on my windows server it doesn't seem to pull in the BtExportHandler class. I am using the basic configuration file as that is sufficient for what I am testing right now. Is there any changes that I need to make to ensure this is working correctly? My JRE is 1.8.0_77 so it should work with java 6

    11 Oct 2016 09:58:03,558 ERROR [conf-file-poller-0] (org.apache.flume.source.http.HTTPSource.configure:152) - Error while configuring HTTPSource. Exception follows.
    java.lang.ClassNotFoundException: com.dynatrace.diagnostics.btexport.flume.BtExportHandler
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:141)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

  21. Anonymous (login to see details)

    Hi,

    I had a similar problem.

    I believe the batch references a file called "dtflume.jar" that does not exist. Make sure you use "dtflume_1.1.1.jar"...

    Regards

    --

    Laurent

    1. Anonymous (login to see details)

      That was it, silly problem. Thanks for the quick help Laurent IZAC

  22. Anonymous (login to see details)

    Hi there,

    I'm having an issue where it's not finding my path correctly because it seems to be removing a / from the path. Any help in fixing this is appreciated 

    LOG FILE

    14 Oct 2016 09:29:41,283 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
    org.apache.flume.EventDeliveryException: Failed to open file D:bigDatapp\1476389031085-1 while delivering event

    CONFIG

    # Describe sinks
    agent1.sinks.PurePathSink.type = com.dynatrace.diagnostics.flume.RollingFileSink
    # once every 10 min
    agent1.sinks.PurePathSink.sink.rollInterval = 600
    # Force cutoff at 10 MB
    agent1.sinks.PurePathSink.sink.rollSize = 10
    agent1.sinks.PurePathSink.sink.directory = D:\bigData\pp
    agent1.sinks.PurePathSink.sink.batchSize = 1000
    # Attention the key for the serialize on file_roll is a little different
    agent1.sinks.PurePathSink.sink.serializer = com.dynatrace.diagnostics.btexport.flume.BtPurePathSerializerBuilder

  23. Anonymous (login to see details)

    Hi,

    it's Java: try with slashes rather than backslashes...(or double your backslashes)

  24. Anonymous (login to see details)

    Dear expert
    We have met a problem that when they try to export data through Big Data Business Transaction Bridge there is some visit ID was missing. These visit ID can be found in user action data but they cannot found in Visit data. 
    Is there a way to check why they are missing?

    1. Anonymous (login to see details)

      Hi. Sorry for the late reply. Do you still see this problem? Also - are you talking about the Dynatrace Visitor Tag that we keep internally - or - do you talk about the Visit Tag that you have configured as a Measure - so that dynatrace can tag your visitors based on something like a username?

      1. Anonymous (login to see details)

        Hi. This issue was solve. It turns out that the flume setting have some problem. It's not APPMON related.