cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Flume - Hadoop errror

pranilc
Newcomer

Hi,

I am trying to setup dynatrace big data plugin for hadoop using flume.I have configured flume and hadoop.

However,after running flume ,i am getting following error :

5 May 2015 15:03:49,963 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance of sink: VisitSink, type: hdfs
05 May 2015 15:03:49,967 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:207) - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:214)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more

 

Can Somebody help please?

4 REPLIES 4

Jeffrey_Fynboh
Dynatrace Organizer
Dynatrace Organizer

It seems like you are missing *hadoop-core.jar* in the class path of flume. (see here or here) Make sure your HADOOP_HOME variable points to where your HADOOP installation is.

Can you provide some more details on the environment on where you are running flume?

 

-Jeff

 

Hi Jeff,


The hadoop core jar has been placed in flume lib folder and the slf4j jars too :


01 Jun 2015 15:56:11,368 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:211)  - Unhandled error
java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;Ljava/lang/Throwable;)V
 at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:120)
 at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:220)
 at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:54)
 at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:106)
 at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:208)
 at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:529)
 at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
 at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
 at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
 at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
01 Jun 2015 15:56:42,692 INFO  [node-shutdownHook] (org.apache.flume.node.FlumeNode.stop:67)  - Flume node stopping - QKDSK305
01 Jun 2015 15:56:42,692 INFO  [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215)  - Node manager stopping
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.conf.file.AbstractFileConfigurationProvider.stop:91)  - Configuration provider stopping


 


Kindly help.

Jeffrey_Fynboh
Dynatrace Organizer
Dynatrace Organizer

What version of slf4j are you using?

It look like this method signature changed between SLF4J 1.5x and SLF4J 1.6.x.

Can you provide some more details on the environment on where you are running flume including the version of the different components? I.e. Flume, Hadoop, SLF4J, etc

-Jeff

Hi Jeff,

Below are the versions i am using :

Dynatrace - 6.1 running on Windows 7 64 bit

Apache flume - 1.3.1 running on Centos 6.6

Hadoop - 2.3.0 (build for 64 bit) running on Windows 7 64 bit

hadoop-common-2.3.0.jar

hadoop-core-1.0.4.jar

hadoop-hdfs-2.3.0.jar

hadoop-mapreduce-client-core-2.3.0.jar

commons-configuration-1.6.jar

slf4j-api-1.6.1.jar

slf4j-log4j12-1.6.1.jar

 

Thanks,

Pranil