My-Tiny.Net :: Networking with Virtual Machines



MultiTail


MultiTail is an ideal tool for seeing the effects of tests. MultiTail lets you view one or multiple files in separate windows on your console, or merge files with a different color for each so you can easily understand which file the lines are from. Output of programs can be viewed, programs can be automatically restarted, restart interval can be set, the difference since the previous run can be displayed instead of everything, a window can be cleared before a new iteration, can scroll back in a window, line wrapping can be switched off to allow scrolling left and right with the cursor keys, and a search function with regular expressions is included. [wow - this guy thinks of everything]

To see a list of keys to use while the program runs, press [F1] or at any time: it gives you context related information. Just quickly, if you look at multiple files with MultiTail press b select the window, use the up/down arrow keys to scroll in the window, gg to move to the top or G to move to the end, Y to deactivate and activate linewrap, and q to exit the program.

Here are some basic examples: (you should adjust the filenames ...)

View two logfiles two windows:

multitail /var/log/messages /var/log/auth.log

View two log files in columns

multitail -s 2 /var/log/messages /var/log/auth.log

Merge two log-files with a different color for each so you can easily understand which logfile the lines are from:

multitail -ci green /var/log/yum.log -ci yellow -I /var/log/mysqld.log

Run a command and view a log file: the -l option allows the command to execute in a window.
Careful: use lowercase L not one! Also do not forget to use quotes " if the external command needs parameter! (like "ping 127.0.0.1")

multitail /var/log/iptables.log -l "ping www.tinynet.edu"

Merge the output of a command in color with a log file:
the -L option is like -l but merges the output with the previous window

multitail /var/log/iptables.log -ci green -L "ping www.tinynet.edu"

Run a command at a specified interval: -r restarts, -R shows only what has changed since the previous run.
Use -rc or -Rc to clear the window before each iteration.

multitail -R 2 -l netstat -t

You can set up windows, colors and filters from within multitail, and use w to generate a shell script to start it with the current settings. Check F1 for the list of keys.

Here are two more useful multitail command switches from the man page:
--no-repeat
"Last message repeated x times" will be printed instead of the same line over and over again.

--label text
Put text in front of each line. Useful when merging multiple files and/or commands. 
Introduction to MultiTail (by the developer) 
    http://www.tldp.org/LDP/LG/issue95/vanheusden.html
Quickstart, with a summary of the man page at the end 
    https://arkit.co.in/multitail-command-to-view-multiple-logs-in-single-terminal/

If you keep getting bothered with a message about /var/spool/mail, 
at a command prompt enter (only need to do this once) 
      echo "check_mail:0" >>/etc/multitail.conf


Penetration Testing (PenTest)

Multitail is an ideal tool for watching tests of firewall rules. FIN scan, for example:

1. Add the IPtables rule with a LOG target

iptables -A INPUT -p tcp --tcp-flags FIN,ACK FIN -j LOG --log-level error --log-prefix "iptables FIN Scan: "

2. Use Multitail with the command in one window and the logfile in another (remember: use lowercase L not one). Multitail closes the command window when the command finishes, so it is better to send a few packets and sleep for a minute after.

multitail /var/log/messages -l "hping -c 3 -p 80 -F 192.168.56.101; sleep 60"
or, let multitail do the repetitions with -r 1 (minus r one) to repeat the command every second until you quit with q:

multitail /var/log/messages -r 1 -l "hping -c 1 -p 80 -F 192.168.56.101; sleep 60"
Merging the command and the log messages using different colors can also be effective:

multitail /var/log/messages -ci green -L "hping -c 3 -p 80 -F 192.168.56.101; sleep 60"
That may look like a lot of typing, but you can use the up arrow key to repeat the last command, and just edit it for a new variation.

You can also use w from within multitail to save the current settings to a shell script that can be used to start it later.



Monitor a logfile on another system

For this we need to setup a way to connect to the remote host and execute the tail command. One thing we have to love about Linux/Unix though is that there are any number ways to get things done. In this case we can use:
  1. ssh multiplexing and a manual login
  2. ssh with key based authentication
  3. xinetd listener
  4. netcat listener (hack it!)
SSH multiplexing and key based authentication are described in the Smoother SSH page under SSL and SSH on the menu.
SSH multiplexing is quite easy, and the two methods can be combined. A command like this on the Gateway (with lowercase L not one)

multitail -l "ssh root@mail.tinynet.edu tail -f /var/log/maillog" /var/log/maillog

requires either ssh multiplexing AND one manual login =or= key based authentication on the MailServer to keep SSH from asking for a password.

We should consider SSH the "most proper" method because information going across the network is encrypted - the other two methods would need a stunnel client and server setup for each connection, which would take essentially the same amount of effort. On the other hand, if we are only monitoring logfiles we don't really need to encrypt the data in transit.

Both the xinetd and netcat method need a script that the listener will execute when we connect to their port, and both use netcat to connect. The difference between these two methods is how the listener is configured.
  • The xinetd method is better in the sense that it can handle multiple simultaneous connections, and allows control over who can connect and when they can connect.

  • The netcat method is more educational because it takes advantage of some interesting operating system features, but it is a bit less reliable than xinetd.

Step-by-Step

[ 1 ]  First, we need to set up the script that the listener will execute when we connect to the port. We can use a similar script for Both the xinetd and netcat method. The script here is the same as the one in the xinetd and Arbitrary Services section of the xinetd and Telnet page under Utilities on the menu.

To create an empty script, use the command: echo '#!/bin/bash' >/usr/local/sbin/showlog
(use single quote) and then open the file in your editor and type in 

    read FILE theRest
    /usr/bin/tail -f /var/log/$FILE
Save it and set the executable bits on the file. It needs to work before the listener is started, so test it with any file that exists in /var/log: for example, echo "stunnel.log" |/usr/local/sbin/showlog

[ 2 ]  Next, we need to set up the listener.

For the xinetd persistent listener, see the xinetd and Arbitrary Services section of the xinetd page under Utilities on the menu for all the details on how to set up the service configuration.

For netcat you need to review the Standard I/O :: Redirection and Utilities :: Netcat pages on the menu. Then create another script, for example /usr/local/sbin/loglisner and put this in the body: (be careful with the spaces in the if) 
    # check for the named pipe
      if [ ! -p /tmp/piper ]; then 
        rm /tmp/piper
      fi
      mkfifo /tmp/piper

    # Start the listener for viewing logfiles on port 23432 
    #     use bsdnc because -lk makes it easy to reconnect 
    #     send it to the background with & so we can use the terminal 
      cat /tmp/piper |bsdnc -lk 23432 |/usr/local/sbin/showlog 1>/tmp/piper &
Save it and set the executable bits on the file. Run the script and check that the service is ready with netstat -tulp

If there is an error in the script, stop it with Ctrl c and then kill the process listening on port 23432 using htop or, from the command line, using the process ID number shown in netstat -tulp and kill -9 number When everything checks out, edit /etc/rc.d/rc.local to call your script every time the system boots up.

[ 3 ]  Once the listener is ready, you can use netcat (under Utilities on the menu) to connect to the service.
To connect from the Gateway to the service on the Mailserver, for example

multitail -l "echo maillog |nc -w 300 mail.tinynet.edu 23432" /var/log/maillog

The -w switch for netcat closes the connection after a given number of seconds, so you probably want to adjust that.
When using the netcat method, be sure to only quit multitail with  q  (not Ctrl c). 

File test operators in bash 
    https://www.tldp.org/LDP/abs/html/fto.html



Logfile Analysis with Regular Expressions (regex)

The real power of multitail comes with regular expressions. Regular expressions allow us to select the output we want to see and ignore the rest, which is super-useful when we work with logfiles.

The downside is that regular expressions can be very cryptic when they use multiple wildcard symbols to select sets of words, characters, or numbers, and (as if that isn't enough) we need to be aware that there are a number of regular expression dialects developers can choose to use.

Regular expressions are so useful for logfile analysis that it is worth a few basic examples. The standard unix tool for applying regular expressions is grep, which is capable of using three dialects of Regular Expressions: Basic (BRE) which is the default, Extended (ERE) with -E, and Perl-Compatible (PCRE) with -P

The developers of multitail decided to use the "Perl Compatible Regular Expressions" (PCRE) libraries, which have also been adopted by the developers of Python and JavaScript. The examples here will show the "native" grep syntax and the PCRE syntax to illustrate the differences.

Let's start by using grep to isolate the DHCP activity for today in the dnsmasq.log file on the gateway. Then we can look at how to do this in multitail, adding in the logfile from a remote client. There are more ideas in the Centralized syslog Server section of the Orientation :: Logfiles page on the menu.

Selecting the DHCP activity for today with grep

Scrolling through /var/log/dnsmasq.log we find that since every incoming message is passed to all of the interfaces, and dnsmasq is enforcing separate subnets for each interface, there are a lot of dnsmasq-dhcp messages that say "wrong network", "no address available", and "range" (at startup). So, we want to exclude these when we select lines that start with today's three-letter Month and day and have "-dhcp" in the line.

We don't want to change our script every day, so before we run the command we save the output of the date command to a variable, like this:
      bd="$(date +%b' '%_d)"
using percent underscore d because by default, for the first 9 days of the month the date command returns Feb 02 (for example), while syslog puts in an extra space rather than the leading zero: Feb  2

In the examples here the commands here also are being run from /var/log so we don't have to type the path to the file.

grep searches stdin and writes all of the lines with the search string to stdout. We have to use  -e  to protect a pattern that begins with  -  because grep is aggressive about its command line arguments (it does not respect the quotes alone).

Using grep we cannot select and exclude lines at the same time, so we need to pipe several commands together to filter the output of each one, like this: 

series of exclude filters:
    bd="$(date +%b' '%_d)"; cat dnsmasq.log |grep "^$bd" |grep -e '-dhcp' |grep -v 'no addr' |grep -v 'wrong' |grep -v 'range'

or, using immediate input redirection instead of cat and a pipe
    bd="$(date +%b' '%_d)"; grep "^$bd" <dnsmasq.log |grep -e '-dhcp' |grep -v 'no addr' |grep -v 'wrong' |grep -v 'range'
That's very inefficient: let's reduce the first part where we select all of the lines that start with today's date and the series of  -v  exclude filters to regular expressions: 

    bd="$(date +%b' '%_d)"; grep "^$bd.*-dhcp" < dnsmasq.log |grep -v "no addr\|wrong\|range"

Using multitail we can select and exclude lines at the same time, as long as we follow the rules
(the documentation is not clear on this!):
  • You can have multiple  -ev  (exclude) but only one  -e  (select)
    and
  • All  -ev  must come before  -e 

The multitail equivalents of the grep examples are: 
series of exclude filters: (only one -e is allowed, so that MUST be a regex)
    bd="$(date +%b' '%_d)"; multitail -ev "no addr" -ev "wrong" -ev "range" -e ^$bd".*-dhcp" dnsmasq.log next file or command

all regex:
    bd="$(date +%b' '%_d)"; multitail -ev ".*no addr+|.*wrong+|.*range" -e "^$bd+.*-dhcp" dnsmasq.log next file or command
Under the microscope 
Set the shell variable     bd="$(date +%b' '%_d)";
                grep    "^$bd.*-dhcp"	 "no addr\|wrong\|range"
                mtail   "^$bd+.*-dhcp"	 ".*no addr+|.*wrong+|.*range"

All Together

Once we have a listener set up on the mailserver, the whole multitail command looks like this when we use  \  to break a single command into multiple lines for easier typing, and put in an extra select filter to match the interface we want: 

    bd="$(date +%b' '%_d)"; multitail -s 2 \
    -ev ".*no addr+|.*wrong+|.*range" \
    -e "^$bd+.*-dhcp+.*eth3" \
    /var/log/dnsmasq.log \
    -l "echo dhcpcd-log |nc -w 300 mail.tinynet.edu 23435"
Remember: No Spaces After the  \ 

xinetd can "get tired" of launching the shell script after a few rounds of fast experimentation, so if you use an xinetd listener and the multitail command suddenly stops working, restart xinetd on the mailserver.

Final Notes

There are several tools around that use regular expressions to highlight strings in logfiles (rather than whole lines like multitail does), but this is a bit tricky in bash, so we'll leave it as a topic for further exploration. Both multitail and grep have a lot of other regex command line options, but the final word here is that even simple regular expressions can make logfile analysis much more efficient.

Lots of people ask "which regex dialect is better?". Two things to know: grep -P  means "this regex is PCRE dialect", and the PCRE dialect is widely adopted because of its extra capabilities. Check the links for details, but PCRE makes some things easy:
  • Select the line when it contains both foo and baz at the same time:

         .*(foo)+.*(baz)+|.*(baz)+.*(foo)+.* 

    The .* lets it match however many characters it needs to before it gets to "word1", then it looks for "word2" following any number of characters.

  • Select the complete line when it contains foo and bar and baz in any order:

         (?=.*foo)(?=.*bar)(?=.*baz) 

    Again, the .* in the lookahead (?= ) lets it match however many characters it needs to before it gets to "word1". Here, though, the match position is reset before the second lookahead seeks out "word2". Reset again, and the final part matches "word3"; since it's the last word to check for, it isn't necessary that it be in a lookahead, but it doesn't hurt. 



Want to impress at a job interview? Be a master of Regular Expressions!
    step-by-step interactive tutorial
       https://regexone.com/ 
    tutorial with solutions 
       https://ryanstutorials.net/regular-expressions-tutorial/ 
    useful example patterns
       http://www.cbs.dtu.dk/courses/27610/regular-expressions-cheat-sheet-v2.pdf
    interactive regex tester
        https://regex101.com/
    try it with this regex - RFC 2616 section 5 explains when HTTP is used
        GET (http(s?):\/)?\/.+\?(.+=.+)+( HTTP\/)
    and this test string
        GET /index.html?x=1&y=2 HTTP/1.1
  An interactive bash script for defining regular expressions in several dialects:
    txt2regex https://aurelio.net/projects/txt2regex/
    man page: https://man.cx/txt2regex(1)