cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Alerts not reaching Splunk with version 2.3.0 of the Splunk Plugin

romain_bigeard
Newcomer

Hey @Michael V. (I hope you are the right Michael 😉 ).

I spotted two issues with the latest version of the Splunk plugin (2.3.0).

1) There is still a reference to 1.3.1 in the path for log4j in runFlume.py at line 39, which breaks logging for flume itself:

It should be: log4j = os.path.join(appdir,"bin", "apache-flume-1.6.0-bin", "conf", "log4j.properties")

and not

log4j = os.path.join(appdir,"bin", "apache-flume-1.3.1-bin", "conf", "log4j.properties")

2) Secondly, every time an alert is being sent to Splunk using the latest version of the Dynatrace Splunk Alert Plugin, flume logs the following error and the alert isn't logged (and therefore not indexed):

12 May 2017 10:16:17,936 WARN [650949596@qtp-1653254447-0] (org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:242) - Received bad request from client.
org.apache.flume.source.http.HTTPBadRequestException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.dynatrace.diagnostics.btexport.flume.BtExportHandler.getEvents(Unknown Source)
at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:241)
at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488)
at com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:193)
at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.<init>(Unknown Source)
at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.<init>(Unknown Source)
at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions$1.parsePartialFrom(Unknown Source)
at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions$1.parsePartialFrom(Unknown Source)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.parseFrom(Unknown Source)

7 REPLIES 7

pkharbanda
Participant

Same issues Alerts are not coming to Splunk with 2.3.0 ...... @Shane K @Ari P.

So I have downloaded the dynatrace 2.3.0 app and configured the same way as the old version and it sends the pp data but not alerts data any solutions or who can help please suggest guys

francesco_gigli
Participant

Hello, have you solved this issue? We have the same situation. Updating the Splunk App to 2.4.0 doesn't solve the issue. No alerts are being reported to flume port and we have no logging both on AppMon and Splunk sides. We have no clues. Any help? Maybe from developer? Thank you

pkharbanda
Participant

any updates on this ?

pkharbanda
Participant

Can you please get any updates on the above error @Shane K.

I am not aware of any updates... @Michael V. ?

kenneth_lynch
Inactive

was anyone able to resolve this or work around the issue


kenneth_lynch
Inactive

Dynatrace, I understand that the plugin is not supported, however, there does not seem to be a supported method for sending alerts from APPMON to splunk. It is significant that the events be sent at the time the incident occurs. REST API or Dashboard retrieves historical data and causes multiple events per alert.
Can someone reach out to the developer of the plugin and see if he can provided assistance, or suggest a supported method.