====== Ubuntu - Processes - List all non-kernel processes ======

Usually kernel processes are safe and clean.

For kernel processes, either PID (process id) is 2 or PPID (parent process id) is 2.

Here is how to get all non-kernel processes.

<code bash>
ps --ppid 2 -p 2 -p 1 \
   --deselect -o uid,pid,rss,%cpu,command
</code>

returns:

<code bash>
UID   PID   RSS %CPU COMMAND
   0   411  1848  0.0 /lib/systemd/systemd-
   0   572  2904  0.0 dhclient -1 -v -pf /r
 102   902  1244  0.0 dbus-daemon --system
   0   912  1948  0.0 /lib/systemd/systemd-
   0  5869   388  0.0 upstart-socket-bridge
 200  1953   904  0.0 /usr/sbin/apache2 -k
 200  3463  3700  0.0 /usr/sbin/apache2 -k
  ...  ...
  ...  ...
   0  5098  4224  0.0 sshd: ubuntu [priv]
   0  5139  1748  0.0 /usr/bin/python /usr/
 200  5140  3484  0.0 /usr/bin/python /usr/
 200  5176  1904  0.0 sshd: ubuntu@pts/3
 200  5177  3860  0.0 -bash
 200  5193  1200  0.0 tmux attach -t denny
   0  5297  4224  0.0 sshd: ubuntu [priv]
  ...  ...
  ...  ...
</code>

**NOTE**:

  * **rss** (resident set size): real RAM usage.
  * **-deselect**: rule out matched processes.

----

===== Rule out trusted procsses =====

We may have many processes running, which are expected and trusted. e.g apache2, tomcat7, mysqld, etc.

To avoid distraction, build a white list especial for your project.

----

===== Sort processes by memory and cpu =====

We’re more concerned about suspicious processes using noticeable resource.

<code bash>
# Sort by memory first, then cpu
ps --ppid 2 -p 2 -p 1 --deselect \
  -o uid,pid,rss,%cpu,command, \
  --sort -rss,-cpu
</code>

----

===== Automate Detection Process and Get Alerts =====

We hide all the complexities and white list configuration in a python script (detect_suspicious_process.py).

If you issue the python command, you may see output like "**Identified processes count: XXX.**"  Define a scheduled task to run periodical check and confirm the number.

If the number is not 0 or it changes, send alerts.

It might take a while to build a suitable white list.

Once it’s done, your servers are always more secured and managed!

<code bash>
wget -O /tmp/detect_suspicious_process.py \
https://raw.githubusercontent.com/\
DennyZhang/devops_public/tag_v2/python/\
detect_suspicious_process/\
detect_suspicious_process.py

# Detect suspicious process
python /tmp/detect_suspicious_process.py

# Detect by customized whitelist
python /tmp/detect_suspicious_process.py \
   --whitelist_file /tmp/whitelist.txt
</code>

----

===== detect_suspicious_process.py =====

<code python>
# -*- coding: utf-8 -*-
#!/usr/bin/python
##-------------------------------------------------------------------
## @copyright 2015 DennyZhang.com
## File : detect_suspicious_process.py
## Author : DennyZhang.com <denny@dennyzhang.com>
## Description : http://www.dennyzhang.com/suspicious_process/
##        python ./detect_suspicious_process.py
##        python ./detect_suspicious_process.py --whitelist_file /tmp/whitelist.txt
## --
## Created : <2016-01-15>
## Updated: Time-stamp: <2016-08-20 13:32:55>
##-------------------------------------------------------------------
import argparse
import subprocess
import os, sys

################################################################################
# TODO: move to common library
def string_in_regex_list(string, regex_list):
    import re
    for regex in regex_list:
        regex = regex.strip()
        if regex == "":
            continue
        if re.search(regex, string) is not None:
            # print "regex: %s, string: %s" % (regex, string)
            return True
    return False

################################################################################
DEFAULT_WHITE_LIST = '''
/sbin/getty -.*
dbus-daemon .*
 acpid -c /etc/acpi/events -s /var/run/acpid.socket$
 atd$
 cron$
 /lib/systemd/systemd-udevd --daemon$
 /lib/systemd/systemd-logind$
 dbus-daemon --system --fork$
 /usr/sbin/sshd -D$
 rsyslogd$
 /usr/sbin/mysqld$
 /usr/sbin/apache2 -k start$
'''

COMMAND_GET_NONKERNEL = '''
sudo ps --ppid 2 -p 2 -p 1 --deselect \
-o uid,pid,rss,%cpu,command \
--sort -rss,-cpu
'''

def get_nonkernel_process():
    process_list = subprocess.check_output(COMMAND_GET_NONKERNEL, shell=True)
    return process_list

def load_whitelist(fname):
    white_list = ""
    if fname is None:
        print "No white list file is given. Use default value."
        white_list = DEFAULT_WHITE_LIST
    else:
        print "load white list from %s" % (fname)
        with open(fname) as f:
            white_list = f.readlines()
    return white_list

def list_process(process_list, white_list):
    import re
    l = []
    for line in process_list.split("\n"):
        line = line.strip()
        if line == "":
            continue
        if not string_in_regex_list(line, white_list.split("\n")):
            l.append(line)
    return l

################################################################################
if __name__=='__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--whitelist_file', required=False,
                        help="config file for whitelist", type=str)
    args = parser.parse_args()
    white_list = load_whitelist(args.whitelist_file)
    nonkernel_process_list = get_nonkernel_process()
    process_list = list_process(nonkernel_process_list, white_list)

    # Remove header
    print "Identified processes count: %d." % (len(process_list) - 1)
    print "\n".join(process_list)
## File : detect_suspicious_process.py ends
</code>