python - Multiprocessing : More processes than cpu.count -


note: "forayed" land of multiprocessing 2 days ago. understanding basic.

i writing , application uploads amazon s3 buckets. in case file size larger(100mb), ive implemented parallel uploads using pool multiprocessing module. using machine core i7 , had cpu_count of 8. under impression if pool = pool(process = 6) use 6 cores , file begins upload in parts , uploads first 6 parts begins simultaneously. see happens when process greater cpu_count , entered 20 (implying want use 20 cores). surprise instead of getting block of errors program began upload 20 parts simultaneously (i used smaller chunk size make sure there plenty of parts). dont understand behavior. have 8 cores, how cant program accept input of 20? when process=6, use 6 threads?? can explanation of 20 being valid input there can 1000s of threads. can please explain me.

edit:

i 'borrowed' code here. have changed wherein ask user core usage choice instead of setting parallel_processes 4

the number of processes running concurrently on computer not limited number of cores. in fact have hundreds of programs running right on computer - each own process. make work os assigns 1 of 8 processors each process or thread temporarily - @ point may stopped , process take place. see what difference between concurrent programming , parallel programming? if want find out more.

edit: assigning more processes in uploading example may or may not make sense. reading disk , sending on network blocking operation in python. process waits chunk of data read or sent can halted process may start io. on other hand, many processes either file i/o or network i/o become bottleneck , program slow down because of additional overhead needed process switching.


Comments

Popular posts from this blog

node.js - Mongoose: Cast to ObjectId failed for value on newly created object after setting the value -

[C++][SFML 2.2] Strange Performance Issues - Moving Mouse Lowers CPU Usage -

ios - Possible to get UIButton sizeThatFits to work? -