Hands-on Guide to the Red Hat Exams: System Logging, Monitoring, and Automation
This chapter is from the book
This chapter is from the book
This chapter is from the book
- Working with Syslog—This section covers logging, both local and centralized.
- Monitoring System Performance—This section looks at how to monitor system performance.
- Automation with cron and at—This section covers system automation via scheduled and single-instance jobs.
The following RHCSA exam objectives are covered:
Identify CPU and memory-intensive processes, adjust process priority with renice, and kill processes
Locate and interpret system log files
Schedule tasks using cron
The following RHCE exam objectives are covered:
Produce and deliver reports on system utilization (processor, memory, disk, and network)
Use shell scripting to automate system maintenance tasks
Configure a system to log to a remote system
Configure a system to accept logging from a remote system
How do you know when something goes wrong on your system? You look at the logs, of course! In all seriousness, understanding system logging is important so that you can troubleshoot when something goes wrong. Logs are kept for almost everything that you can think of, including the kernel, boot process, and different services that are running. Aside from logs, there are also utilities you can use to monitor and check the status of your system. This chapter looks at how you can use these tools and logging to troubleshoot anything that is thrown your way. The chapter also touches on how to automate some of these tasks.
Working with Syslog
Whenever something goes wrong—and sometimes when things go right—on the system, a message is generated by the syslog service. These messages are used to keep track of how the system is performing, what actions are taking place, and if there is a problem with something. Red Hat comes with a syslog service built in and already set to start at boot. In RHEL6, the rsyslog package is installed by default and is the primary logging service used.
The rsyslog service has a main config file, /etc/rsyslog.conf, that controls where messages are sent when they are generated. Because logging is a critical piece of the operating system, it is installed by default as part of the core packages.
- Step 1. Just out of good habit, you should verify that the package is installed:
# rpm -qa | grep syslog rsyslog-4.6.2-2.el6.x86_64
Because you can see that it is installed, you should also check on two other issues. - Step 2. First, make sure that the service is set to start when the system boots:
# chkconfig rsyslog --list rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
- Step 3. Second, make sure that the service is currently running:
# service rsyslog status rsyslogd (pid 1279) is running...
For RHEL05:
- Step 1. Verify the installation of the package:
# rpm -qa | grep klog sysklogd-1.4.1-44.el5
- Step 2. Check that the service will start on boot:
# chkconfig syslog --list syslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Aside from the name difference, as we have already discussed, there are two daemons that run instead of one. They do, however, both run under a single service name. - Step 3. Verify that the service is running:
# service syslog status syslogd (pid 2214) is running... klogd (pid 2217) is running...
As you can see here, there are some slight differences between logging on RHEL5 and RHEL6 (such as the naming and single versus double daemons). The rsyslog service provides better logging features, so it is recommended to upgrade if possible.
All the messages sent to the rsyslog service are stored in the /var/log directory, and each message section has its own file or subdirectory. You can also define your own log files or directories. The biggest problem that arises from using any logging service is the files can become uncontrollable if left unchecked. Being able to log different alerts and to obtain information from logs, in my opinion, is an art all unto itself. There are so many ways to customize logs, and each network setup is different. You should know that there are nine different alerts that the syslog service uses. These alerts can be produced by different network devices and Red Hat systems.
The nine severity levels of syslog include the following:
- emerg
- alert
- crit
- err
- warning
- notice
- info
- debug
- none
The Config File
The /etc/rsyslog.conf file is broken down into the following parts:
|
Modules |
Indicate whether they can be loaded or unloaded (rsyslog is modular) |
|
Global directives |
Specify config options that apply to the daemon (all start with a $) |
|
Rules |
Indicate cooperation of a selector and an action |
|
Selector |
Is based on <facility>.<priority> |
|
Action |
Defines what to do with messages after they're sorted by the selector |
Let's look at the default config file to see some examples:
# cat /etc/rsyslog.conf #rsyslog v3 config file # if you experience problems, check # http://www.rsyslog.com/troubleshoot for assistance #### MODULES #### $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) #$ModLoad immark.so # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp.so #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp.so #$InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/spppl/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ###
You can see from the output where some of the files are stored. Also shown is a sample rule.
Log Rotation
If left alone, logs can grow to enormous size. Luckily for you, the logrotate command allows you to rotate logs before they become too big. By default, the logrotate command is called daily to cycle the log files into a new file. The parameters of the command are defined in its config file, /etc/logrotate.conf, and the /etc/logrotate.d directory. However, the command itself is called from /etc/cron.daily/logrotate, a cron job that is run daily (as already mentioned). You can always call the logrotate command yourself if you'd like to rotate your logs, but don't do this too frequently because you'll end up with tons of rotated logs and you'll be right back to square one with a mess of unorganized log files.
Syntax: logrotate [options] configfile
Options:
|
-d |
Debugs; doesn't do anything but test |
|
-f |
Forces file rotation |
|
-v |
Provides verbose output |
The following example rotates the current logs as defined in the config file:
# logrotate /etc/logrotate.conf
Centralized Logging
If you'd like to set up a centralized syslog server, you need to choose which server you'd like to store all your logs on. For this example, you can use the RHEL01 system as the centralized server and configure it to receive logs from the other systems on the network.
- Step 1. Look at a section of the /etc/rsyslog.conf file again:
#### MODULES #### $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) #$ModLoad immark.so # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp.so #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp.so #$InputTCPServerRun 514
In the modules section, you can see two areas that will provide the capability to receive logs from other systems. The standard is to use the UDP protocol on port 514. By uncommenting the section related to UDP reception, you essentially can provide a way for the syslog server to become the centralized server for the network. - Step 2. Uncomment the following two lines:
$ModLoad imudp.so $UDPServerRun 514
- Step 3. When you're finished, you need to save the file, exit, and restart the syslog service:
# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
As with any service that you'd like to make available to your network, you need to create a firewall allowing UDP 514 to be opened up. - Step 4. Use iptables to create the required firewall rule:
# iptables -I INPUT 5 -p udp -m udp --dport 514 -j ACCEPT - Step 5. Save the firewall rule you just created:
# service iptables save Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
- Step 6. Then restart the iptables service:
# service iptables restart iptables: Flushing firewall rules: [ OK ] iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Unloading modules: [ OK ] iptables: Applying firewall rules: [ OK ]
The firewall rules might not make sense at the moment because we haven't covered iptables or firewall rules yet. After you read Chapter 12, "System Security," return here to make sure you understand what is happening in the creation of firewall rules.
Now that the security requirements have been addressed and the centralized server is configured, you can shift your attention to the client systems. Let's look at RHEL02 for the client system. You can config RHEL02 to send some or all of its log files to the centralized syslog server (RHEL01).
You can use the following syntax when configuring log files to be sent to the centralized server:
<log file> @<hostname or IP of system (local or remote)>
You can edit the RHEL02 system to point all security logs to the central server:
# nano /etc/rsyslog.conf authpriv.* @172.168.1.1
Save the file and restart the syslog service on RHEL02:
# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
If you log out and log back in on RHEL02, it generates a security event. You can then jump over to the RHEL01 host and check the logs to see the event generated by the client system RHEL02.
Centralized Logging (The RHEL5 Way)
To set up centralized logging for RHEL5, do the following:
- Step 1. Edit the /etc/sysconfig/syslog file to include the -r option:
# nano /etc/sysconfig/syslog # Options to syslogd # -m 0 disables 'MARK' messages. # -r enables logging from remote machines # -x disables DNS lookups on messages received with -r # See syslogd(8) for more details SYSLOGD_OPTIONS="-m 0 -r"
- Step 2. Save the file, exit, and restart the syslog service:
# service syslog restart Shutting down kernel logger: [ OK ] Shutting down system logger: [ OK ] Starting system logger: [ OK ] Starting kernel logger: [ OK ]
With the syslog service now accepting remote logs, you need to create a firewall rule. This process is the same as on RHEL6. One last modification that needs to be made on RHEL5 is the adjustment of some SELinux protections. - Step 3. Query the required Boolean values:
# getsebool -a | grep logd klogd_disable_trans --> off syslogd_disable_trans --> off
- Step 4. Change the values to allow incoming logs:
# setsebool -P klogd_disable_trans=1 syslogd_disable_trans=1 - Step 5. Verify the preceding Booleans have been changed:
# getsebool -a | grep logd klogd_disable_trans --> on syslogd_disable_trans --> on
Although we haven't covered SELinux yet (it is covered in Chapter 11, "SELinux"), the Boolean values are required here if you are still using RHEL5. After you read through Chapter 11, you can refer back here to make sure you understand the changes being made.
User Login Events
Aside from the normal logs generated and used by the syslog service, there are two special commands that deal with system logins. These two commands have special logs that can be read only through the use of these commands:
|
lastlog |
Lists login records |
|
faillog |
Lists failed login attempts |
You can use these two commands for viewing login events. These two commands are useful for hunting down attacks where users are trying to force themselves into other users' accounts (called a brute-force attack).
Syntax: lastlog [options]
Options:
|
-b DAYS |
Displays results older than DAYS |
|
-u LOGIN |
Displays results for the user LOGIN |
Show the last time the user01 account logged in:
# lastlog -u user01 Username Port From Latest user01 tty Fri Sep 10 05:16:42 -0400 2010
You can also view more details using the faillog command.
Syntax: faillog [options]
Options:
|
-a |
Displays all events |
|
-l SEC |
Locks the account for SEC seconds after a failed login |
|
-u LOGIN |
Prints records for user LOGIN |
Show more information about the user01 account login events:
# faillog -u user01 Login Failures Maximum Latest On user01 0 0
The ability to work with syslog is important to being able to help troubleshoot any issue on Linux, not just Red Hat. When you're trying to find an issue or resolve one, the log files should always be your first stop.