Allowed Memory in R and C Stack Usage -

-

i'm trying read in large file (~5gb) r , process data. can read in entire 5gb file, trouble comes when apply processing. don't have great grasp on memory basics in r, , i'm hoping of can me understand better.

here example of i'm running 

file = fread("file.txt") #file.txt 5gb of unprocessed data t.str <-strptime(file$time, "%m/%d/%y %h:%m:%s"")#convert column date class month = as.numeric(format(t.str, "%m"))#create vector file column high = ifelse(file$age>70,1,0) #create vector file column #there ten more lines operate on file.

fread fine job of reading in file. , first 3 or 4 operations run on 'file' data frame work. however, after number of them run, error says:

c stack usage 19923892 close limit

i'm pretty sure issue isn't command i'm running since worked on smaller data sets. i've read bit on stacks are, warning isn't totally making sense me. mean r using pointer run through these big vectors, , i've run out of pointer space (?). read similar issue here:

error: c stack usage close limit

one user suggested increasing stack size in shell. tried looking in further, i'm not sure how proceed. here suggested:

$ ulimit -s # print default 8192 $ r --slave -e 'cstack_info()["size"]' size  8388608

can me understand means, or explain bit stack usage in r? or know of better way process data doesn't exceed stack usage? i'm not sure how give guys reproducible data.

edit add in example of data:

personid     time              energy   age 1301839    07/24/2013 07:15:00  0.13    68 1301521    07/24/2013 07:30:00  0.19    68 1301890    07/24/2013 07:45:00  0.10    68 1301890    07/24/2013 08:00:00  0.06    68 1307112    07/24/2013 08:15:00  0.01    68

sorry, isn't answer, don't have enough points comment. try reading , processing data in chunks, or check out of large memory packages in cran task view high performance computing . can read memory usage here.


Comments

Post a Comment

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

-
this question has answer here: what's mongoose error cast objectid failed value xxx @ path “_id”? 7 answers even though has been asked many times because entity has no valid objectid field. i came across situation objectid set value can casted objectid still error. the schema: var serviceschema = parammongoose.schema({ servicename:{ type: parammongoose.schema.types.objectid , ref: 'servicenames',required:true } }); pseudo code: servicefromdb=new service({servicename:'some name'}); servicefromdb.servicename='000000000000000000000001'; servicefromdb.save(function(paramerror,paramdata){ if(paramerror){ console.log('but but...',servicefromdb,paramerror) } }); the output of code is: but but... { servicename: 000000000000000000000001, _id: 55079a90286f49280364f78b } { [casterror: cast objectid faile...
Read more

[C++][SFML 2.2] Strange Performance Issues - Moving Mouse Lowers CPU Usage -

-
i've been working on making 2d map editor past 2 weeks or so, , i've run strange problem. trying optimize code, example, making functions should've been functions begin with, did made cpu usage go through roof. i've tried commenting out different sections of code thought culprits (scrolling, rendering, other calculations) no avail. think issue might trying call things within different functions, sake of making modular possible, i'd functionality. the other major change i've made since file i'm comparing i've ported variables external cpp file, since having variables global makes lot easier use within different functions. simple issue, can't life of me understand why happens. with original code, around 2-3% cpu usage. new code, around 20-35%, but, if move mouse around, it'll drop around 6-8%. tl;dr: after trying optimize code using functions, i've somehow killed performance, , found strange quirk moving mouse reduces cpu usage.. /...
Read more

ios - Possible to get UIButton sizeThatFits to work? -

-
i have been using method of resizing uibuttons depreciated, , not robust. , other reasons, want sizethatfits work uibuttons. i've read online, i'm not sure if should work (seems working some, not others, difference maybe between style, i'm using custom). here simple test code recreate issue (i put in viewdidload test, shouldn't matter, , real code part of large project): uibutton *btn = [uibutton buttonwithtype:uibuttontypecustom]; [btn settitle:@"this test long title need word wrap more single line when displayed on tiny ipod in portrait view." forstate:uicontrolstatenormal]; btn.titlelabel.linebreakmode = uilinebreakmodewordwrap; // depreciated - nothing replace it? cgrect r = btn.frame; r.origin.x = 0; r.origin.y = 0; r.size.width = 320; r.size.height = [btn.titlelabel.text sizewithfont:btn.titlelabel.font constrainedtosize:cgsizemake(r.size.width,100000) linebreakmode:btn.titlelabel.linebreakmode].height; // returns approx 86 , changes correctly ...
Read more