scala - Cannot compile Spark Streaming example: getting updateStateByKey is not a member of org.apache.spark.streaming.dstream.DStream error -

-

i want understand spark streaming better copied statefulnetworkwordcount.scala example in own directory, , pasted as-is, created simple sbt config file, , tried compile doesnt seem work. why complaining api? idea? original example compile fine, want change it, without having compile whole spark directory set default.

thanks! matt

$ more build.sbt name := "simple project"  version := "1.0"  scalaversion := "2.10.4"  librarydependencies += "org.apache.spark" %% "spark-core" % "1.2.1" librarydependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1" urbanlegends-2:streamingtest mlieber$

the error:

urbanlegends-2:streamingtest mlieber$sbt package [info] set current project simple project (in build file:/users/mlieber/app/spark2/spark/examples/lieber/streamingtest/) [info] compiling 2 scala sources /users/mlieber/app/spark2/spark/examples/lieber/streamingtest/target/scala-2.10/classes... [error] /users/mlieber/app/spark2/spark/examples/lieber/streamingtest/src/main/scala/com/cloudera/streamingtest/statefulnetworkwordcount2.scala:58: value updatestatebykey not member of org.apache.spark.streaming.dstream.dstream[(string, int)] [error]     **val statedstream = worddstream.updatestatebykey[int](newupdatefunc, [error]**                                    ^ [error] 1 error found [error] (compile:compile) compilation failed [error] total time: 3 s, completed mar 16, 2015 4:23:15 pm urbanlegends-2:streamingtest mlieber$

looks problem in libraries version incapability. try replace "spark-core" % "1.2.1" , "spark-streaming_2.10" % "1.2.1" with

"spark-core" % "1.2.0"  "spark-streaming_2.10" % "1.2.0"

[update] or need add import org.apache.spark.streaming.streamingcontext._


Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

gradle error "Cannot convert the provided notation to a File or URI" -

-
i have defined 2 system properties in gradle.properties: systemprop.builddir=build systemprop.cfgdir=build\\cfg and have following task defined in build.gradle: task clean(group:'clean',description:'clean project') << { ant.sequential { delete(dir: system.properties.builddir) mkdir(dir: system.properties.builddir) delete(dir: system.properties.cfgdir) mkdir(dir: system.properties.cfgdir) } } this generates following error: execution failed task ':clean'. > cannot convert provided notation file or uri: {dir=build}. following types/formats supported: - string or charsequence path, e.g 'src/main/java' or '/usr/include' - string or charsequence uri, e.g 'file:/usr/include' - file instance. - uri or url instance. but equivalent block of code not generate error , works expected: task clean(group:'clean',description:'clean project') << { ...
Read more

ios - Possible to get UIButton sizeThatFits to work? -

-
i have been using method of resizing uibuttons depreciated, , not robust. , other reasons, want sizethatfits work uibuttons. i've read online, i'm not sure if should work (seems working some, not others, difference maybe between style, i'm using custom). here simple test code recreate issue (i put in viewdidload test, shouldn't matter, , real code part of large project): uibutton *btn = [uibutton buttonwithtype:uibuttontypecustom]; [btn settitle:@"this test long title need word wrap more single line when displayed on tiny ipod in portrait view." forstate:uicontrolstatenormal]; btn.titlelabel.linebreakmode = uilinebreakmodewordwrap; // depreciated - nothing replace it? cgrect r = btn.frame; r.origin.x = 0; r.origin.y = 0; r.size.width = 320; r.size.height = [btn.titlelabel.text sizewithfont:btn.titlelabel.font constrainedtosize:cgsizemake(r.size.width,100000) linebreakmode:btn.titlelabel.linebreakmode].height; // returns approx 86 , changes correctly ...
Read more