hadoop - Haddop MRUnit MapDriver.addInput() giving NotSerializableException: java.nio.HeapByteBuffer error -

-

i using mrunit unit test our mapreducer below pom changes

<dependency>     <groupid>org.apache.mrunit</groupid>     <artifactid>mrunit</artifactid>     <version>1.1.0</version>     <classifier>hadoop1</classifier>     <scope>test</scope> </dependency>

i not using avro.

i have added below lines rid of serialization issue

configuration conf = driver.getconfiguration(); conf.set("io.serializations", "org.apache.hadoop.io.serializer.javaserialization,"             + "org.apache.hadoop.io.serializer.writableserialization");

now, using below api add inputs

mapdriver.addinput(map<string, bytebuffer> key, map<string, bytebuffer> val)

but when adding inputs above call, getting below error

java.lang.runtimeexception: java.io.notserializableexception: java.nio.heapbytebuffer @ org.apache.hadoop.mrunit.internal.io.serialization.copy(serialization.java:86) @ org.apache.hadoop.mrunit.internal.io.serialization.copy(serialization.java:97) @ org.apache.hadoop.mrunit.internal.io.serialization.copywithconf(serialization.java:110) @ org.apache.hadoop.mrunit.testdriver.copy(testdriver.java:675) @ org.apache.hadoop.mrunit.testdriver.copypair(testdriver.java:679) @ org.apache.hadoop.mrunit.mapdriverbase.addinput(mapdriverbase.java:120) @ com.pearson.psn.mapreduce.eventmappertest.testmapper(eventmappertest.java:102) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ org.junit.runners.model.frameworkmethod$1.runreflectivecall(frameworkmethod.java:44) @ org.junit.internal.runners.model.reflectivecallable.run(reflectivecallable.java:15) @ org.junit.runners.model.frameworkmethod.invokeexplosively(frameworkmethod.java:41) @ org.junit.internal.runners.statements.invokemethod.evaluate(invokemethod.java:20) @ org.junit.internal.runners.statements.runbefores.evaluate(runbefores.java:28) @ org.junit.internal.runners.statements.runafters.evaluate(runafters.java:31) @ org.junit.runners.blockjunit4classrunner.runchild(blockjunit4classrunner.java:73) @ org.junit.runners.blockjunit4classrunner.runchild(blockjunit4classrunner.java:46) @ org.junit.runners.parentrunner.runchildren(parentrunner.java:180) @ org.junit.runners.parentrunner.access$000(parentrunner.java:41) @ org.junit.runners.parentrunner$1.evaluate(parentrunner.java:173) @ org.junit.internal.runners.statements.runbefores.evaluate(runbefores.java:28) @ org.junit.internal.runners.statements.runafters.evaluate(runafters.java:31) @ org.junit.runners.parentrunner.run(parentrunner.java:220) @ org.eclipse.jdt.internal.junit4.runner.junit4testreference.run(junit4testreference.java:50) @ org.eclipse.jdt.internal.junit.runner.testexecution.run(testexecution.java:38) @ org.eclipse.jdt.internal.junit.runner.remotetestrunner.runtests(remotetestrunner.java:459) @ org.eclipse.jdt.internal.junit.runner.remotetestrunner.runtests(remotetestrunner.java:675) @ org.eclipse.jdt.internal.junit.runner.remotetestrunner.run(remotetestrunner.java:382) @ org.eclipse.jdt.internal.junit.runner.remotetestrunner.main(remotetestrunner.java:192) caused by: java.io.notserializableexception: java.nio.heapbytebuffer @ java.io.objectoutputstream.writeobject0(objectoutputstream.java:1164) @ java.io.objectoutputstream.writeobject(objectoutputstream.java:330) @ java.util.hashmap.writeobject(hashmap.java:1001) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:39) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25) @ java.lang.reflect.method.invoke(method.java:597) @ java.io.objectstreamclass.invokewriteobject(objectstreamclass.java:940) @ java.io.objectoutputstream.writeserialdata(objectoutputstream.java:1469) @ java.io.objectoutputstream.writeordinaryobject(objectoutputstream.java:1400) @ java.io.objectoutputstream.writeobject0(objectoutputstream.java:1158) @ java.io.objectoutputstream.writeobject(objectoutputstream.java:330) @ org.apache.hadoop.io.serializer.javaserialization$javaserializationserializer.serialize(javaserialization.java:80) @ org.apache.hadoop.io.serializer.javaserialization$javaserializationserializer.serialize(javaserialization.java:65) @ org.apache.hadoop.mrunit.internal.io.serialization.copy(serialization.java:80) ... 30 more

any idea missing here?

your problem caused trying serialise bytebuffers. not serialisable , need convert them arrays of bytes.


Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

gradle error "Cannot convert the provided notation to a File or URI" -

-
i have defined 2 system properties in gradle.properties: systemprop.builddir=build systemprop.cfgdir=build\\cfg and have following task defined in build.gradle: task clean(group:'clean',description:'clean project') << { ant.sequential { delete(dir: system.properties.builddir) mkdir(dir: system.properties.builddir) delete(dir: system.properties.cfgdir) mkdir(dir: system.properties.cfgdir) } } this generates following error: execution failed task ':clean'. > cannot convert provided notation file or uri: {dir=build}. following types/formats supported: - string or charsequence path, e.g 'src/main/java' or '/usr/include' - string or charsequence uri, e.g 'file:/usr/include' - file instance. - uri or url instance. but equivalent block of code not generate error , works expected: task clean(group:'clean',description:'clean project') << { ...
Read more

python - NameError: name 'subprocess' is not defined -

-
i'm trying make rng website: http://jamesdotcom.com/?p=417 . when run following code: captureimage = subprocess.popen( [ "fswebcam", "-r", "356x292", "-d", "/dev/video0", "static.jpg", "--skip", "10" ], stdout=devnull, stderr=devnull) captureimage.communicate() i following error message: nameerror: name 'subprocess' not defined what should do? thanks have imported first? import subprocess captureimage = subprocess.popen(["fswebcam", ...) subprocess included in python standard library since python 2.4. should not necessary install it.
Read more