hadoop - Flume folder routing based on HTTP header -

-

using curl , flume, post csv files on local machine/hdfs @ different locations based on values of http header. example, http header (network-element: ggsn) files stored on local machine in folder named ggsn.

i have following flume configuration

  • a http source
  • a memory channel
  • a hdfs sink routes events files different locations depending on http header

i post csv files using curl:

find /path/files -type f -exec curl -x post http://localhost:9043 -h "content-type: text/xml" -h "network-element: ggsn" --data-binary "@{}" -v \;

these logs generated:

* connect() localhost port 9043 (#0) *   trying ::1... connection refused *   trying 127.0.0.1... connected * connected localhost (127.0.0.1) port 9043 (#0) > post / http/1.1 > user-agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 nss/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > host: localhost:9043 > accept: */* > content-type: text/xml > network-element: ggsn > content-length: 972660 > expect: 100-continue > < http/1.1 100 continue < http/1.1 200 ok < transfer-encoding: chunked < server: jetty(6.1.26) < * connection #0 host localhost left intact * closing connection #0

flume logs show following:

2015-03-16 19:41:14,887 debug org.apache.flume.sink.solr.morphline.blobhandler: requestheaders: {expect=100-continue, host=localhost:9043, content-length=972660, network-element=ggsn, user-agent=curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 nss/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2, content-type=text/xml, accept=*/*} 2015-03-16 19:41:14,891 debug org.apache.flume.sink.solr.morphline.blobhandler: blobevent: [event headers = {content-type=text/xml}, body.length = 972660 ]

i use flume configuration:

sa.sources  = httpsource1 sa.channels = memorychannel1 sa.sinks    = localsink1  sa.sources.httpsource1.type     = http sa.sources.httpsource1.handler     = org.apache.flume.sink.solr.morphline.blobhandler sa.sources.httpsource1.port     = 9043 sa.sources.httpsource1.channels = memorychannel1  sa.channels.memorychannel1.type   = memory sa.channels.memorychannel1.capacity   = 10000 sa.channels.memorychannel1.transactioncapacity   = 1000  sa.sinks.localsink1.type         = file_roll sa.sinks.localsink1.channel      = memorychannel1 sa.sinks.localsink1.sink.directory   = /path/%{network-element} sa.sinks.localsink1.sink.rollinterval = 36000

for reason files cannot placed under path: /path/%{network-element} looks path not exist, if have manually created ggsn folder , set permissions it.


Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

gradle error "Cannot convert the provided notation to a File or URI" -

-
i have defined 2 system properties in gradle.properties: systemprop.builddir=build systemprop.cfgdir=build\\cfg and have following task defined in build.gradle: task clean(group:'clean',description:'clean project') << { ant.sequential { delete(dir: system.properties.builddir) mkdir(dir: system.properties.builddir) delete(dir: system.properties.cfgdir) mkdir(dir: system.properties.cfgdir) } } this generates following error: execution failed task ':clean'. > cannot convert provided notation file or uri: {dir=build}. following types/formats supported: - string or charsequence path, e.g 'src/main/java' or '/usr/include' - string or charsequence uri, e.g 'file:/usr/include' - file instance. - uri or url instance. but equivalent block of code not generate error , works expected: task clean(group:'clean',description:'clean project') << { ...
Read more

[C++][SFML 2.2] Strange Performance Issues - Moving Mouse Lowers CPU Usage -

-
i've been working on making 2d map editor past 2 weeks or so, , i've run strange problem. trying optimize code, example, making functions should've been functions begin with, did made cpu usage go through roof. i've tried commenting out different sections of code thought culprits (scrolling, rendering, other calculations) no avail. think issue might trying call things within different functions, sake of making modular possible, i'd functionality. the other major change i've made since file i'm comparing i've ported variables external cpp file, since having variables global makes lot easier use within different functions. simple issue, can't life of me understand why happens. with original code, around 2-3% cpu usage. new code, around 20-35%, but, if move mouse around, it'll drop around 6-8%. tl;dr: after trying optimize code using functions, i've somehow killed performance, , found strange quirk moving mouse reduces cpu usage.. /...
Read more