Linux Log Analysis - Troubleshooting Your Linux OS Via System Log Files

Linux Log Analysis - Troubleshooting Your Linux OS Via System Log Files
Page content

Introduction

Linux logs everything starting from the system boot. You can see everything that happened on your system (if you are curious what happens in your system) and read the whole process line by line. You can use a text editor of your choice to read these logs - because they are text files - or you can go with the command line. Personally I wish to push users toward the use of the command line, but if you prefer the other way, you can open your text editor and go to File → Open to see the logs.

What Does the Logging? D(a)emons!

Yes, the daemons do the logging. A daemon (literally, “a little demon”) is a process that works in the background, not under the control of the user, responsible for performing many specific tasks, not only logging. Daemons have a “d” at the end of their names; for example sshd is actually a daemon that controls secure shell (ssh) connections. In our case, the two daemons that control the logging are syslogd and klogd.

Linux systems have a ‘logrotate’ command that also runs in the background and prevents the logs from “inflation,” that is, it renames the logs so that one log file does not become too big in size. To do that, it appends .1, .2, .3 to the end of the files so that you can see both “what is happening” and “what has happened”. The larger the number, the older the log is. (See the screenshot on the left, which is my computer’s /var/log directory. Note the .1.gz, .2.gz files.)

How Many Logs are There?

Many. There are really many logs present in your system. The logs are kept for boot, kernel, http, mail, news, security, currently logged users to count a few. We will go through each one to see what it does and how does it help us.

Which Events are Logged?

It depends on the configuration. The logging is configured in /etc/syslog.conf file. In this file, the events that are required to be logged are in the first field and the log files are in the second field. The first field consists of two different words that are separated with a full stop. The first word is “what application” is to be logged and the second word is the level of severity.

In logging, there are six severity levels: debugging, information, notice, warning, error, critical, alert/emergency. The bold parts show how they are written in the syslog.conf. To log everything, a * is placed. The * can also be used for the applications. For example,

*.crit /var/log/critical

Logs every critical happening in the /var/log/critical file. The same way,

mail.* /var/log/maillog

Writes every severity about mail to /var/log/maillog file.

Which Tools do I Need?

[inlineImage|right|DE9434CE940FE4D9CE6663FC75707D09FD295168|cat /var/log/messages | grep ports | more|]If you will work with a graphical text editor, you can skip this section.

To work with the log files in the command line, “cat”, “tail”, “head”, “more” and “grep” will be your best friends. “cat” displays a file on the screen (cat is the short for concatenate). “grep” grabs the specified regular expression from a file/folder (in fact ‘grep’ is an abbreviation of global regular expression print.) “tail” displays the last 10 lines of a given file and “head” displays the first 10 lines. “more” displays the text at one page at a time. You can hit Ctrl-c or q to quit from more.

We can use all these commands in combination, such as cat /var/log/messages will display all the /var/log/messages file. The lines will flow so fast from your terminal that you will not be able to see anything, therefore you can use it with a pipe ( | ) and “more” to display only one page: cat /var/log/messages | more. On the other hand, if you are looking for a specific word in a file you can use the “grep” command, such as cat /var/log/mail | grep root to see the mail logs that have the word “root” in them. If you are not comfortable to read from the terminal screen, you can output the contents to a text file, for example cat /var/log/messages > /home/your_user_name/Desktop/messages will create a text file on your user’s desktop with all the output that would normally be displayed on the screen. Furthermore issuing cat /var/log/messages | grep root > /home/your_user_name/Desktop/root_messages will find the word “root” in /var/log/messages and write the output in the root_messages file on your user’s desktop. The screenshot displays the output of cat /var/log/messages | grep ports | more command.

/var/log/dmesg

/var/log/dmesg is the file where the kernel messages are logged during boot time. You can also access this file by issuing dmesg command from the command line. These messages can be read by the users but /var/log/dmesg is the only log file that can be read by the users. All other log files can only be read by the root user.

If you have a problem -say with your sound card- in your computer and you want to see if everything went well during start up, you can check this by dmesg | grep sound or cat /var/log/messages | grep sound commands.

/var/log/boot.log

Everything that you see on the screen during the boot time are logged to /var/log/boot.log. If you have a graphical user boot screen, which is typical with the latest distributions, I suggest you to check the /var/log/boot.log file to see how your computer boots.

/var/log/messages

This is in fact a “general purpose” log file where various system applications and daemons record messages. This is probably the first file that I check to see if anything is going wrong if there is some other stuff that I want to check.

/var/log/messages contains important information. Let’s see from the output of cat /var/log/messages | grep port command from my computer:

Mar 22 11:18:03 paladin ntpd[4795]: kernel time discipline status change 41

Mar 22 11:18:09 paladin portmap[4991]: connect from 228.18.12.212 to callit(mountd): request from unauthorized host

Mar 22 11:18:57 paladin ntpd[4795]: kernel time discipline status change 1

See the bold line? Some nasty boy from 228.18.12.212 is trying to connect to my computer. As you see, one of the uses of this file can be to analyze your /var/log/messages to see if there is someone interested in your computer.

/var/log/secure

This file logs all the local and remote logins and sessions to your computer. You can use it to see if there are any successful attempts to log on to your computer by analyzing this file. Logically, the only logins that you should see have to be the ones that you or the users you know. If there is something else, begin to sound the bells.

A serious attacker -not the script kiddies that download some programs from the Internet and think themselves as wannabe hackers- will alter or destroy the log files. It will also be wise to analyze the log in more detail to see if there are any questionable entries. For example, if you have worked with your computer on Sunday and there is no clue about it in /var/log/secure file, you should have a question mark.

Note: Ubuntu users will not find the /var/log/secure file. This is because the system logs the entries to /var/log/syslog.

/var/log/lastlog

This file contains login times for the users. You can not use cat /var/log/lastlog command to view the logfile, because it is binary mode. Simply issue lasltog from the command line ar root to see the log contents. It will be a good start for your own computer’s security audit with /var/log/messages and /var/log/secure files.

/var/log/cron

Cron is the process in UNIX-like systems which is responsible for executing tasks at a specified time and/or with specified intervals. /var/log/cron keeps the status messages from cron.

/var/log/maillog

I believe this log file is self explanatory. I will not go into the details of the file and ask you to issue cat /var/log/messages | more to see the contents of the file. If your computer is taken over by somebody and is being used to send spam messages, you are most likely to find traces here.

Conclusion

We, Linux users, should all know what is going on with the log files and how to check for some basic information. Other than maintenance and troubleshooting, as you see, it also helps us a lot for security audits. It depends on which information you want to extract from the log files. If you want to have information about your hardware, run the grep command on the log with the name of the hardware as we said above: cat /var/log/messages | grep sound. Or go through the files once and see what information you can extract from them. Do not fear from trying.

This post is part of the series: Linux Tips We Should All Know

In this series we target all the Linux users by touching the points that make our Linux experience enjoyable in the long-term. We look at the best practices for using applications, tips for analyzing log files, distribution troubleshooting and many others.

  1. Linux Tips for Everyone: Using Applications in Linux
  2. Linux Tips for Everyone: Distro Troubleshooting
  3. Linux Tips for Everyone: Log File Analysis