javascript - Mongoose find one and push to array of documents -
i'm new mongodb , mongoose , i'm trying use save stock ticks daytrading analysis. imagined schema:
symbolschema = schema({ name:string, code:string }); quoteschema = schema({ date:{type:date, default: now}, open:number, high:number, low:number, close:number, volume:number }); intradayquotesschema = schema({ id_symbol:{type:schema.types.objectid, ref:"symbol"}, day:date, quotes:[quotesschema] }); from link receive information every minute:
date | symbol | open | high | low | close | volume
2015-03-09 13:23:00|aapl|127,14|127,17|127,12|127,15|19734
i have to:
- find objectid of symbol (aapl).
- discover if intradayquote document of symbol exists (symbol , date combination)
- discover if minute ohlcv data of symbol exists on quotes array (because repeated)
- update or create document , update or create quotes inside array
i'm able accomplish task without veryfing if quotes exists, method can creates repeated entries inside quotes array:
symbol.find({"code":mysymbol}, function(err, stock) { intradayquote.findoneandupdate({ { id_symbol:stock[0]._id, day: myday }, { $push: { quotes: myquotes } }, { upsert: true }, mycallback }); }); i tried:
- $addtoset instead of $push, unfortunatelly doesn't seems work array of documents
- { id_symbol:stock[0]._id, day: myday, 'quotes["date"]': mydate } on conditions of findoneandupdate; unfortunatelly if mongo doesn't find it, creates new document minute instead of appending quotes array.
is there way working without using 1 more query (i'm using 2)? should rethink schema facilitate job? appreciated. thanks!
basically put $addtoset operator cannot work because data not true "set" definition being collection of "completely distinct" objects.
the other piece of logical sense here working on data arrives, either sinlge object or feed. i'll presume feed of many items in form , can use sort of stream processor arrive @ structure per document received:
{ "date": new date("2015-03-09 13:23:00.000z"), "symbol": "aapl", "open": 127.14 "high": 127.17, "low": 127.12 "close": 127.15, "volume": 19734 } converting standard decimal format utc date since locale settings should domain of application once data retrieved datastore of course.
i @ least flatten out "intradayquoteschema" little removing reference other collection , putting data in there. still need lookup on insertion, overhead of additional populate on read seem more costly storage overhead:
intradayquotesschema = schema({ symbol:{ name: string, code: string }, day:date, quotes:[quotesschema] }); it depends on usage patterns, it's more effective way.
the rest comes down acceptable
stream.on(function(data) { var symbol = data.symbol, myday = new date( data.date.valueof() - ( data.date.valueof() % 1000 * 60 * 60 * 24 )); delete data.symbol; symbol.findone({ "code": symbol },function(err,stock) { intradayquote.findoneandupdate( { "symbol.code": symbol , "day": myday }, { "$setoninsert": { "symbol.name": stock.name "quotes": [data] }}, { "upsert": true } function(err,doc) { intradayquote.findoneandupdate( { "symbol.code": symbol, "day": myday, "quotes.date": data.date }, { "$set": { "quotes.$": data } }, function(err,doc) { intradayquote.findoneandupdate( { "symbol.code": symbol, "day": myday, "quotes.date": { "$ne": data.date } }, { "$push": { "quotes": data } }, function(err,doc) { } ); } ); } ); }); }); if don't need modified document in response benefit implementing bulk operations api here , sending updates in package within single database request:
stream.on("data",function(data) { var symbol = data.symbol, myday = new date( data.date.valueof() - ( data.date.valueof() % 1000 * 60 * 60 * 24 )); delete data.symbol; symbol.findone({ "code": symbol },function(err,stock) { var bulk = intradayquote.collection.initializeorderedbulkop(); bulk.find({ "symbol.code": symbol , "day": myday }) .upsert().updateone({ "$setoninsert": { "symbol.name": stock.name "quotes": [data] } }); bulk.find({ "symbol.code": symbol, "day": myday, "quotes.date": data.date }).updateone({ "$set": { "quotes.$": data } }); bulk.find({ "symbol.code": symbol, "day": myday, "quotes.date": { "$ne": data.date } }).updateone({ "$push": { "quotes": data } }); bulk.execute(function(err,result) { // maybe response }); }); }); the point 1 of statements there modify data, , since sent in same request there less , forth between application , server.
the alternate case might more simple in case have actual data referenced in collection. becomes simple matter of processing upserts:
intradayquotesschema = schema({ symbol:{ name: string, code: string }, day:date, quotes:[{ type: schema.types.objectid, ref: "quote" }] }); // , in steam processor stream.on("data",function(data) { var symbol = data.symbol, myday = new date( data.date.valueof() - ( data.date.valueof() % 1000 * 60 * 60 * 24 )); delete data.symbol; symbol.findone({ "code": symbol },function(err,stock) { quote.update( { "date": data.date }, { "$setoninsert": data }, { "upsert": true }, function(err,num,raw) { if ( !raw.updatedexisting ) { intradayquote.update( { "symbol.code": symbol , "day": myday }, { "$setoninsert": { "symbol.name": stock.name }, "$addtoset": { "quotes": data } }, { "upsert": true }, function(err,num,raw) { } ); } } ); }); }); it comes down how important have data quotes nested within "day" document. main distinction if want query documents based on data of "quote" fields or otherwise live overhead of using .populate() pull in "quotes" other collection.
of course if referenced , quote data important query filtering, can query collection _id values match , use $in query on "day" documents match days contain matched "quote" documents.
it's big decision matters path take based on how application uses data. should guide on general concepts behind doing want achieve.
p.s unless "sure" source data date rounded exact "minute" want employ same kind of date rounding math used discrete "day" well.
Comments
Post a Comment