I have a situation where several servers are running FFMPEG, under a complex scripting system, where the developers refuse to use the "-threads" option to limit system resource hogging -- it literally brings the system down at times. My question is: is there another way from the system level I can limit how many cores can be bound to by these processes? Or another generic way to indicate that x number of cores are restricted to kernel use, etc. (if that's the proper tactic). There may be other ways I can limit overall resource usage through limits.conf, which I'm partially familiar with. There may be an even simpler, low-tech means of doing this, by inserting a shell script to catch the incoming command flags, append -threads X, then pass it to the real command. I think that would work? Thanks.
Instead of limiting the cores for the processes, I would use the nice command to run the ffmpeg processes with a lower priority. So they can take all cores if there are free ressources but get limited by the system automatically if processes with higher priority need the cpu capacity.
I would probably need to do that via shell script intercept, I would imagine? I wouldn't even know what a good "nice" value starting would be. For example, if we have 8 cores, I don't want FFMPEG binding to all of them - it brings the system down. Very frustrating.
It seems /etc/security/limits.conf is the place to do this, by setting a hard limit on priority. What I don't want is to prevent jobs from running, as that would be counter-productive. I just don't want them hogging all the cores, which means maybe they run a little slower? This isn't an area I've really ever had to deal with before, so learning a few things along the way.
You should check out nice, it is made for the purpose to ensure that applications run nicely side by side instead on sticking on cores as that makes not much sense for this task. Rename the ffmpeg command that is run by your customers and replace it with a shell script that calls the same command trough nice. For details on the niceness levels, see "man nice".
FFMPEG, by design I believe, will bind a thread to each core (it has to) -- so by inserting a priority adjustment, does that mean it can attach, but only get part of the core's available resources? Slower jobs, but not bringing down the system.
