Popen worth it?

[dcole.ath.cx]

I can't find a lot of info on the web about the pros and cons of using popen to help speed up a process. There are cases where somethings just have to be done one right after the other, but other things, such as web pages are often modular. Would web programs, such as CMS and Forums be faster if they used multiple popen functions to process different information at the same time? These programs often have very modular code and it seams like a page could be processed faster with the use of multiple processes. Or is there going to be very little improvement or slowing down by trying to have parallel processes in these kind of web programs? How much data crunching needs to be done to make it worth having more than one pipe?

[justsomeguy]

The reason there's not a lot of info is because it's usually specific to the situation if using multiple processes will help improve things. You usually don't see it very often in web applications, one example I saw recently was where I was installing Gallery 2 and there is an installation step where it activates a bunch of modules that you select. Instead of activating each module at a time it forks a bunch of processes to activate smaller groups of modules so that it can finish quicker.It's not real easy to determine if you're helping or hurting things by creating more processes. You need to analyze the server resources and figure out what is the reason for something taking a long time to finish. Specifically, you'll want to look at CPU utilization, memory access, and open file pointers (if you're opening files). If all of the CPU time is already being taken up, creating more processes isn't going to help, in fact it might hurt. If the CPU time is lower because the processes are waiting on memory or disk accesses, then creating more processes will probably help.The key term is "-bound", if a process is "CPU-bound" then the CPU is the bottleneck and more processes might cause things to slow down. If a process is memory-bound or disk-bound then typically offloading some of the CPU-intensive work to other processes while the other processes wait for the filesystem will help finish everything quicker. It also matters what work you choose to offload though. If you have a process that spends a lot of time waiting for files and you spawn 10 more processes that are also waiting for files, you're not going to speed things up much. But if you have one process waiting for a file to open and spawn other processes that are doing CPU-bound calculations or something, it will definately make the entire operation faster.The key is figuring out where your bottleneck is and then looking at what work you can run at the same time that isn't using the same bottleneck.