eclipse - Debugging Spark Applications -

-

i trying debug spark application on cluster using master , several worker nodes. have been successful @ setting master node , worker nodes using spark standalone cluster manager. downloaded spark folder binaries , use following commands setup worker , master nodes. these commands executed spark directory.

command launching master

./sbin/start-master.sh

command launching worker node 

./bin/spark-class org.apache.spark.deploy.worker.worker master-url

command submitting application 

./sbin/spark-submit --class application --master url ~/app.jar

now, understand flow of control through spark source code on worker nodes when submit application(i want use 1 of given examples use reduce()). assuming should setup spark on eclipse. eclipse setup link on apache spark website seems broken. appreciate guidance on setting spark , eclipse enable stepping through spark source code on worker nodes.

thanks!

it's important distinguish between debugging driver program , debugging 1 of executors. require different options passed spark-submit

for debugging driver can add following spark-submit command. set remote debugger connect node launched driver program on.

--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

in example port 5005 specified, may need customize if running on port.

connecting executor similar, add following options spark-submit command.

--num-executors 1 --executor-cores 1 --conf "spark.executor.extrajavaoptions=-agentlib:jdwp=transport=dt_socket,server=n,address=wm1b0-8ab.yourcomputer.org:5005,suspend=n"

replace address local computer's address. (it's idea test can access spark cluster).

in case, start debugger in listening mode, start spark program , wait executor attach debugger. it's important set number of executors 1 or multiple executors try connect debugger, causing problems.

these examples running sparkmaster set yarn-client although may work when running under mesos. if you're running using yarn-cluster mode may have set driver attach debugger rather attaching debugger driver, since won't know in advance node driver executing on.


Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

gradle error "Cannot convert the provided notation to a File or URI" -

-
i have defined 2 system properties in gradle.properties: systemprop.builddir=build systemprop.cfgdir=build\\cfg and have following task defined in build.gradle: task clean(group:'clean',description:'clean project') << { ant.sequential { delete(dir: system.properties.builddir) mkdir(dir: system.properties.builddir) delete(dir: system.properties.cfgdir) mkdir(dir: system.properties.cfgdir) } } this generates following error: execution failed task ':clean'. > cannot convert provided notation file or uri: {dir=build}. following types/formats supported: - string or charsequence path, e.g 'src/main/java' or '/usr/include' - string or charsequence uri, e.g 'file:/usr/include' - file instance. - uri or url instance. but equivalent block of code not generate error , works expected: task clean(group:'clean',description:'clean project') << { ...
Read more

[C++][SFML 2.2] Strange Performance Issues - Moving Mouse Lowers CPU Usage -

-
i've been working on making 2d map editor past 2 weeks or so, , i've run strange problem. trying optimize code, example, making functions should've been functions begin with, did made cpu usage go through roof. i've tried commenting out different sections of code thought culprits (scrolling, rendering, other calculations) no avail. think issue might trying call things within different functions, sake of making modular possible, i'd functionality. the other major change i've made since file i'm comparing i've ported variables external cpp file, since having variables global makes lot easier use within different functions. simple issue, can't life of me understand why happens. with original code, around 2-3% cpu usage. new code, around 20-35%, but, if move mouse around, it'll drop around 6-8%. tl;dr: after trying optimize code using functions, i've somehow killed performance, , found strange quirk moving mouse reduces cpu usage.. /...
Read more