Setup Central Logging Server on HA mode with Rsyslog in Linux

Rsyslog is a powerful, secure and high-performance log processing system which accepts data from different types of source (systems/applications) and outputs it into multiple formats

Centralized Logging System

It has evolved from a regular syslog daemon to a fully-featured, enterprise level logging system. It is designed in a client/server model, therefore it can be configured as a client and/or as a central logging server for other servers, network devices, and remote applications.
By Default system accept the logs only generated from local host. In this example we will configure a log server and will accept logs from client side.

For the purpose of Example, we will use the following hosts:

• Active Server: 10.194.168.128 ; Hostname: syslog-server-1
• Passive Server: 10.194.168.129 ; Hostname: syslog-server-2
• Client1 : 10.194.168.163 ; Hostname: syslog-server-3
• Client2 : 10.194.168.164 ; Hostname: syslog-server-4

STEP 1: INSTALL AND CONFIGURE SYSLOG SERVER (UNDER 10.194.168.128 MACHINE):

Run the following command on syslog server
$ sudo yum update && yum install rsyslog # For CentOS 7
$ sudo apt update && apt install rsyslog # For Ubuntu 16.04, 18.04

Once rsyslog installed, you need to start the service for now, enable it to auto-start at boot and check it’s status with the systemctl command.

$ sudo systemctl start rsyslog
$ sudo systemctl enable rsyslog
$ sudo systemctl status rsyslog

Create a new directory with the following command with named syslog where all client log will reside

$ sudo mkdir -p /var/syslog

The main rsyslog configuration file is located at /etc/rsyslog.conf, which loads modules, defines the global directives, contains rules for processing log messages and it also includes all config files in /etc/rsyslog.d/ for various applications/services. So Open rsyslog Config File

$ sudo vim /etc/rsyslog.conf

And uncomment the following lines

For Centos:

$ModLoad imtcp
$InputTCPServerRun 514

For Ubuntu:

module(load="imtcp")
input(type="imtcp" port="514")

And Add the Following Lines in rsyslog config file

$template RemoteLogs,"/var/syslog/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

Looking at the above ruleset, the first rule is “$template RemoteLogs,”/var/log/%HOSTNAME%/%PROGRAMNAME%.log””.

The directive $template tells rsyslog daemon to gather and write all of the received remote messages to distinct logs under /var/log, based on the hostname (client machine name) and remote client facility (program/application) that generated the messages as defined by the settings present in the template RemoteLogs.

The second line “*.* ?RemoteLogs” means record messages from all facilities at all severity levels using the RemoteLogs template configuration.

The final line (Optional )“& ~” instructs rsyslog to stop processing the messages once it is written to a file. If you don’t include “& ~”, messages will instead be be written to the local files.

That’s it with configuring the rsyslog server. Save and close the configuration file. To apply the recent changes, restart rsyslog daemon with the following command.

$ sudo systemctl restart rsyslog

Now verify the rsyslog network sockets. Use the ss command (or netstat with the same flags) command and pipe the output to grep to filter out rsyslogd connections.

$ sudo ss -tulnp | grep "rsyslog"

Same Configuration will do for Passive Syslog Server.

Step 2: Configure Rsyslog Client to Send Logs to Rsyslog Server

If it’s not installed, install it and start the service as shown earlier on.

$ sudo yum update && yum install rsyslog 	#CentOS 7
$ sudo apt update && apt install rsyslog	#Ubuntu 16.04, 18.04
$ sudo systemctl start rsyslog
$ sudo systemctl enable rsyslog
$ sudo systemctl status rsyslog

Once the rsyslog service is up and running, open the main configuration file where you will perform changes to the default configuration.

$ sudo vim /etc/rsyslog.conf

To force the rsyslog daemon to act as a log client and forward all locally generated log messages to the remote rsyslog server, add this forwarding rule, at the end of the file as shown in the following screenshot.

##Enable sending of logs over TCP add the following line:
*.* @@10.194.168.128:514
#Failover rsyslog server details (HA Mode)
$ActionExecOnlyWhenPreviousIsSuspended on
& @@10.194.168.129:514
$ActionExecOnlyWhenPreviousIsSuspended off
#########################################
##Set disk queue when rsyslog server will be down:
$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
# END ANSIBLE MANAGED BLOCK

The above rule will send messages from all facilities and at all severity levels. To send messages from a specific facility for example auth, use the following rule.

auth. *  @@10.194.168.128:514

Save the changes and close the configuration file. To apply the above settings, restart the rsyslog daemon.

$ sudo systemctl restart rsyslog

Step 3: Monitor Remote Logging on the Rsyslog Server

Run a ls command in syslog server to long listing of the parent logs directory and check if there is a directory called ip-10.194.168.163 (or whatever your client machine’s hostname is).

$ ls -l /var/syslog/

Output of ls -l /var/syslog/

Enjoy.