How To Configure Background Process Killer?

Configuring a Background Process Killer in Linux

On Linux servers, it can be useful to configure a background process killer to free up RAM and CPU when certain thresholds are exceeded. This prevents any single process from overusing resources and degrading overall system performance. In this guide, we'll cover how to setup and configure a simple background process killer using common Linux tools.

What is a Background Process Killer?

A background process killer monitors system resource usage and can forcibly terminate processes when needed to free up RAM, CPU, or other constrained resources.

This allows you to set defined limits on what resources a given process can consume. If a process exceeds the limit, the process killer will automatically kill it so that resources can be reclaimed by the system.

Some scenarios where a background process killer is helpful:

• Kill buggy processes that are eating up all available RAM or CPU

• Free up resources on low-end machines by culling heavy processes

• Implement simple auto-scaling rules for processes running in the background

Overall, a tunable process killer gives you more control over how Linux server resources are allocated between processes.

Installing the Tools

The main tools we'll use are monit and htop:

• monit - a utility for managing and monitoring Unix systems

• htop - an interactive process viewer and monitor

To install on Debian/Ubuntu:

sudo apt install monit htop

Or on CentOS/RHEL:

sudo yum install monit htop

Htop allows us to view current resource usage by process. Monit can then kill processes based on configured rules.

Configuring monit

The monit daemon needs to be configured to actually kill processes when certain criteria are met.

Edit the /etc/monit/monitrc file to uncomment and add these lines:

set daemon 30 # check processes at 30 second intervals  

set logfile /var/log/monit.log

check system localhost

  if loadavg (1min) > 4 then alert

  if loadavg (5min) > 2 then alert

  if memory usage > 75% then alert

  if cpu usage (user) > 70% then alert

  if cpu usage (system) > 30% then alert

This instructs monit to check system resources every 30 seconds. If load average, memory, or CPU exceed these thresholds, it will trigger an alert.

Next we will configure monit to actually kill processes once an alert is triggered.

Add this block:

check process my_process with pidfile /var/run/my_process.pid

  start program "/etc/init.d/my_process start"

  stop program "/etc/init.d/my_process stop" 

  if cpu > 60% for 2 cycles then restart # kill process over 60% CPU for 2 cycles

  if total mem > 500 MB for 5 cycles then restart # kill over 500MB for 5 cycles

Now monit will kill my_process if it exceeds 60% CPU for 2 consecutive checks or exceeds 500MB RAM for 5 consecutive checks.

You can add additional blocks to target different processes by PID file.

Finally, tell monit to start on boot:

sudo systemctl enable monit

Now monit will run in the background, monitor resources, and kill processes that exceed limits.

Process Monitoring with htop

While monit runs in the background, use htop to interactively monitor current resource usage:

htop

This brings up an interactive process viewer where you can see live CPU, memory, and resource usage per process.

Sort by CPU or Memory to identify processes eating up the most resources. You can kill processes manually from within htop if needed as well.

Configuring Process Kill Rules

When configuring monit, consider setting kill rules based on:

• Type of process being monitored

• Available system RAM and cores

• Average vs peak usage patterns

• Minimum resources needed for process to function

You may need to experiment with different thresholds for each process. Make sure to test that killing the processes at the defined limits won't cause issues.

Additional features like only killing if usage exceeds limits for X consecutive checks help avoid killing due to brief spikes.

Conclusion

A background process killer like monit gives you granular control over Linux resource usage. By intelligently configuring thresholds and rules, you can reclaim RAM, CPU, and other constrained resources from runaway processes.

Pair monit with a tool like htop to actively monitor current usage. This allows you to fine tune the kill thresholds and identify the heaviest processes.

The end result is a Linux system that avoids sluggish performance and unresponsive apps caused by resource hogging. With a properly tuned monit configuration, you can maintain optimal healthy performance.