Howto setup Icinga2 and Icingaweb on CentOS

On your newly installed CentOS server:

# this is my network setup for my own usage, won't fit yours :)

cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE="Ethernet"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
NAME="eth0"
UUID="2ef9cace-1428-4dbf-aac7-7993463c359a"
DEVICE="eth0"
ONBOOT="yes"
IPADDR=192.168.183.235
NETMASK=255.255.254.0
NETWORKING=yes
HOSTNAME=s1
GATEWAY=192.168.183.254
NM_CONTROLLED=no 

yum -y install deltarpm
yum -y install wget net-tools bind-utils gcc mc

setenforce 0 # :( 
mcedit /etc/selinux/config
>> change enabled to SELINUX=disabled or SELINUX=permissive
yum -y update && yum -y upgrade
yum install -y epel-release
rpm --import http://packages.icinga.org/icinga.key
wget http://packages.icinga.org/epel/ICINGA-release.repo -O /etc/yum.repos.d/ICINGA-release.repo
yum makecache
yum install -y nagios-plugins-all icinga2 icinga2-ido-mysql icinga-idoutils-libdbi-mysql
yum install -y httpd php-cli php-pear php-xmlrpc php-xsl php-pdo php-soap php-gd php-ldap
mcedit /etc/php.ini
>> set date.timezone = Europe/YOURZONE
systemctl enable httpd && systemctl start httpd
yum install -y mariadb-server
systemctl start mariadb
systemctl enable mariadb
netstat -nlp | grep 3306 #(check if it runs)
mysql -u root
> use mysql;
> update user set password=PASSWORD("root_password") where User='root';
> flush privileges;
> exit
systemctl restart mariadb
mysql -u root -p
>CREATE DATABASE icinga2;
>GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga2.* TO 'icinga2'@'localhost' IDENTIFIED BY 'icinga2_password';
>flush privileges;
>exit
mysql -u root -p icinga2 < /usr/share/icinga2-ido-mysql/schema/mysql.sql
mcedit /etc/icinga2/features-available/ido-mysql.conf
>> change:   user = "icinga2"
>>  password = "icinga2_password"
>>  host = "localhost"
>>  database = "icinga2"
systemctl enable icinga2 && systemctl start icinga2
tail -f /var/log/icinga2/icinga2.log #(check if it runs)
icinga2 feature enable command
icinga2 feature list # (to check)
systemctl restart icinga2
yum -y install icingaweb2 icingacli
grep icingaweb2 /etc/group #check if it's icingaweb2:x:990:apache
touch /var/www/html/index.html
chown apache /var/www/html/index.html
icingacli setup config directory --group icingaweb2
icingacli setup token create # get the token to the clipboard
icingacli setup token show # in case you missed it
systemctl restart httpd
# open a browser and type the IP address or FQDN of your server. That will be icinga.infokom.local for my case.
#next, next, you should see everything green

>authentication : database
>Database type: MySQL
>Host: localhost
>Database name: icingaweb2
>Username: myself
>Password: *********
>Character set: utf8
#rest of the web based setup detailed here with screenshots: 

#
#Now it's time to add your first node to your server.
#On the server, run: 

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specifiy the common name (CN) [icinga.infokom.local]: Press Enter
Checking for existing certificates for common name 'icinga.infokom.local'...
Certificates not yet generated. Running 'api setup' now.
information/cli: Generating new CA.
information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
information/cli: Generating new CSR in '/etc/icinga2/pki/icinga.infokom.local.csr'.
information/base: Writing private key to '/etc/icinga2/pki/icinga.infokom.local.key'.
information/base: Writing certificate signing request to '/etc/icinga2/pki/icinga.infokom.local.csr'.
information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/icinga.infokom.local.crt'.
information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
Generating master configuration for Icinga 2.
information/cli: Adding new ApiUser 'root' in '/etc/icinga2/conf.d/api-users.conf'.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
Please specify the API bind host/port (optional):Press Enter
Bind Host []: Press Enter
Bind Port []: Press Enter
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

# check the output if it's OK  

egrep 'NodeName|TicketSalt' /etc/icinga2/constants.conf 

mcedit /etc/icinga2/zones.conf 

# change the string NodeName to your FQDN, in my case:

cat /etc/icinga2/zones.conf
object Endpoint "icinga.infokom.local" {
}
object Zone ZoneName {
        endpoints = [ "icinga.infokom.local" ]
} 

systemctl restart icinga2.service

# to add my first client server named s2 i need a token 

icinga2 pki ticket --cn 's2.infokom.local'

# On the client server:

yum install -y epel-release
rpm --import http://packages.icinga.org/icinga.key
wget http://packages.icinga.org/epel/ICINGA-release.repo -O /etc/yum.repos.d/ICINGA-release.repo
yum makecache

yum install icinga2 mc

setenforce 0 # :( 

mcedit /etc/selinux/config
>> change enabled to SELINUX=disabled or SELINUX=permissive

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]:Enter 
Starting the Node setup routine...
Please specifiy the common name (CN) [s2.infokom.local]: Enter
Please specifiy the local zone name [s2.infokom.local]: Enter
Please specify the master endpoint(s) this node should connect to:Enter
Master Common Name (CN from your master setup): icinga.infokom.local
Do you want to establish a connection to the master from this node? [Y/n]: y
Please fill out the master connection information:Enter
Master endpoint host (Your master's IP address or FQDN): 192.168.183.235
Master endpoint port [5665]: Enter
Add more master endpoints? [y/N]:  Enter
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):Enter
Host [192.168.183.235]: Enter
Port [5665]: Enter
information/base: Writing private key to '/etc/icinga2/pki/s2.infokom.local.key'.
information/base: Writing X509 certificate to '/etc/icinga2/pki/s2.infokom.local.crt'.
information/cli: Generating self-signed certifiate:
information/cli: Fetching public certificate from master (192.168.183.235, 5665):

information/cli: Writing trusted certificate to file '/etc/icinga2/pki/trusted-master.crt'.
information/cli: Stored trusted master certificate in '/etc/icinga2/pki/trusted-master.crt'.

Please specify the request ticket generated on your Icinga 2 master.
 (Hint: # icinga2 pki ticket --cn 's2.infokom.local'): faaec3b98221622841cc437ee74b09a1f44b1ab
information/cli: Processing self-signed certificate request. Ticket 'faaec3b98221622841cc437ee74b09a1f44b1ab'.

information/cli: Created backup file '/etc/icinga2/pki/s2.infokom.local.crt.orig'.
information/cli: Writing signed certificate to file '/etc/icinga2/pki/s2.infokom.local.crt'.
information/cli: Writing CA certificate to file '/etc/icinga2/pki/ca.crt'.
Please specify the API bind host/port (optional):Enter
Bind Host []: Enter
Bind Port []: Enter
Accept config from master? [y/N]: y
Accept commands from master? [y/N]: y
information/cli: Disabling the Notification feature.
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Enabling the Apilistener feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Generating local zones.conf.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done. 

# to check

grep 's2' /etc/icinga2/constants.conf

mcedit /etc/icinga2/zones.conf 

# change NodeName to your local machine name, in my case it's FQDN

mcedit /etc/icinga2/zones.conf
object Endpoint "icinga.infokom.local" {
        host = "192.168.183.235"
        port = "5665"
}
object Zone "master" {
        endpoints = [ "icinga.infokom.local" ]
}
object Endpoint "s2.infokom.local" {
}
object Zone ZoneName {
        endpoints = [ "s2.infokom.local" ]
        parent = "master"
}

service icinga2 restart && service icinga2 enable

# wait a bit and back to the icinga server:

icinga2 node list 

# you SHOULD see your client server NOW

Node 's2.infokom.local' (last seen: Wed Jul 27 09:36:11 2016)
    * Host 's2.infokom.local'
        * Service 'apt'

[...]

 
icinga2 node update-config

systemctl reload icinga2.service 

Open your web GUI and see your new server, it's in PENDING state now. Wait a bit or click on CHECK NOW button in the 
CHECK EXECUTION section.

ntopng install on Debian Sqeeze

If you are careless enough to just follow a step-by-step tutorial like this being on a good old Squeeze you surely will end up with a failing and buggy ntopng. E.g. you won't be able to see your newly created users (users tab is totally empty: No Results Found)

Looks somewhat broken

or can not switch between your monitored interfaces. If you start ntopng from shell you may see something like this:
19/Aug/2015 13:28:28 [src/Redis.cpp:170] ERROR: ERR unknown command 'HSET' [HSET ntopng.host_labels ]
19/Aug/2015 13:28:28 [src/Redis.cpp:170] ERROR: ERR unknown command 'HSET' [HSET ntopng.host_labels ]
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:30 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'
19/Aug/2015 13:28:36 [src/Redis.cpp:148] ERROR: ERR unknown command 'HGET'

This whole thing is because your Redis installation is out of date. Another nice thing in Debian Squeeze is its repositories includes Version: 2:1.2.6-1 Redis. Simply fix that with:
echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> /etc/apt/sources.list
apt-get update
apt-get -t squeeze-backports install redis-server

Now it is:
redis-server                       2:2.4.15-1~bpo60+2    
How to reset your forgotten ntopng admin password.
You might don't want to bother with compiling ntopng-2.0 packages on a simple standard Squeeze. In that case here are the x64 and x86 versions. You're welcome.

How to assign a group of users to a group of alerts from a group of servers in Zabbix

Configuration - Host groups - Create host group (basic step)
Create your first custom group with a name. If you already have your hosts, here you can add them immediately.

Configuration - Hosts - Create host (basic steps)
Create your first host (if you don't have any ;)).
Host name: if you are using active checks (when zabbix agents connect to your server, less likely) this string has to be identical to that set in your zabbix_agent.conf.
Visible name: this can be any name you like.
Group: add your new host to your custom group.
Agent interfaces: set your host IP address. Don't do harm for yourself with DNS name.
Templates (second tab): Select and add a template to your host, e.g. "Linux servers".

 

Administration - Users - Create user
Set names, password, etc. Second tab: Media. Add an email address to your user.

 

Third tab: Permissions. This is one of the most annoying things in Zabbix: you can't set any of anything here. But there is a small hint on the bottom, see

 

So, create your second, third etc. user without adding permissions here.

Administration - Users - Create user group 

Group name: set a meaningful name. Add your users to the group. Second tab: Permissions. At long last, you can click on Add and link here your host group to your user group.

Configuration - Actions - Create action
Action is the cause why your users will receive emails. No. Wait. The cause is the trigger the action is linked to. No, wait.... The main cause is the the item that fires the trigger. Ahh, anyway... The default selection is "source: trigger" that is okay. Trust me, you don't want to know what the others are.

 

Action name: any meaningful name. Don't touch the default subject and message unless you t know what to do here. Second tab: Conditions. That's where the fun begins! 

In New condition Select host group and equals and your actual group.

New condition again. Trigger severity, equal or greater, warning. (modify this according to your needs.)
More filtering: (advanced!). For example: Trigger name, not like, [%string that matches that trigger's name that is high level enough to notify your users but for some reason you don't want to alert them of this certain cause.%]
Quick link to the manual. (See Escalations also, that a neat stuff.)

Third tab: Operations. Add. Operation type: send message. Select your user group. When everything is set, don't forget to click on Add on the left bottom of this page. Then, Save.
Happy zabbixing!

How to intall smokeping - the only way it works :)

What is smokeping? That's a powerful network monitoring tool which works mainly with tricky ICMP pings and also able to do special TCP and UDP port connection tests built-in. You can check your statistics on web based graphs.

How to install it? There are blogs that discusses the process but I strongly recommend not to follow them word for word because they suffer from serious errors that keep you from succeeding. Happily you are here, at the perfect place for the perfect tutorial!

What is a master and slave configuration?
Master is actually your central smokeping server. It periodically checks the hosts you configured to monitor. Nothing surprising, ehm ? Let's look at the slave(s) then. They check BACK to the master (or any other configured host) and send their results BACK to the master who process their data and displays the results together with its normal monitoring data.

Okay, let's install my master Debian/Ubuntu node first. I'm going to create two logical units inside my monitoring tree. I'll call the first "External hosts" and the second (guess what) "Internal hosts".

MASTER node
------

apt-get update
apt-get install smokeping

Check if /etc/default/smokeping has only ONE active line: "MODE=master"

cd /etc/smokeping/
touch slave-secrets

You define here your SLAVE servers individual passwords. In my case I will have two slave (also active checking) servers in my "External hosts", see later.

cat /etc/smokeping/slave-secrets

mywebserver:topsecr3t
myftpserver:topsecr3t

echo "topsecr3t" > slavesecrets.conf
chmod 660 slave-secrets
chmod 600 slavesecrets.conf
chown smokeping:www-data slave-secrets slavesecrets.conf

cd config.d/
cat Alerts

*** Alerts ***
to = me.admin@mydomain.com
from = smokeping@mydomain.com

[...others are remain the same default...]

cat Database

*** Database ***

step     = 200
pings    = 100

[...others are remain the same default...]
These two variables are changed because I want to check my hosts in every 200 seconds with 100 ping packets.

cat General

*** General ***

owner    = Me.Da.Admin
contact  = me.admin@mydomain.com
mailhost = localhost
# NOTE: do not put the Image Cache below cgi-bin
# since all files under cgi-bin will be executed ... this is not
# good for images.
cgiurl   = http://localhost/cgi-bin/smokeping.cgi
[...others are remain the same default...]

 cat Probes

*** Probes ***

+ FPing

binary = /usr/bin/fping
packetsize = 500
pings = 100
step = 200
timeout = 1.5
[...others are remain the same default...]
Several other parameters can be used, see later.

 cat Slaves

*** Slaves ***
secrets=/etc/smokeping/slave-secrets

+mywebserver
display_name=My Great webserver
color=ff0000

+myftpserver
display_name=My Super FTP server
color=00b7e2

I've defined here my slave servers. NOT those hosts I want to check. Don't be confused: these two categories are totaly different!

cat Targets

 *** Targets ***
probe = FPing

menu = Top
title = Network Latency Grapher
remark = Welcome to my little SmokePing website.

+ External
menu = External hosts
title = Ext

++ mywebserver
menu = My Superb Webserver
host = 10.243.43.6

++ myftpserver
menu = My gorgeos ftpserver
host = 172.16.29.253

++ mysmokeping
menu = this.server
host = 195.95.95.95
slaves = mywebserver myftpserver

+Internal
menu = Interal hosts
title = Gateways

++ MyGateway
menu = My Little Cisco Switch
host = 172.16.21.254

I've set the most important things here: my monitored hosts. Probe type is simple fping. Two units here: External and Internal. Their friendly name will shown in the web menu as "External hosts" and "Internal hosts". External has 3 hosts inside it: two external servers and the monitor server itself. mywebserver and myftpserver HAVE to be the same string as the servers identifies themselves! (as they answer to the "hostname" shell command) ++mysmokeping section MUST HAVE the "slaves = mywebserver myftpserver" line. If you don't have it, the slaves are going to reply with the unpleasant message
"ERROR: we did not get config from the master. Maybe we are not configured as a slave for any of the targets on the master ?"

/etc/init.d/smokeping restart

If you can't see any useful answer to this :) you may find this command profitable: journalctl -xn

Wait some minutes and point your browser to http://195.95.95.95/smokeping/smokeping.cgi

And! Here is the point for slaves: set your file rights according to the following:

/var/lib/smokeping# ls -sal

[...]
4 drwxrwx---  2 smokeping www-data  4096 Mar  6 13:05 External

cd smokeping/
chown smokeping:www-data *
chmod 755 *

This is a MUST to let the Slaves able to POST their data to apache running on your smokeping master.

SLAVE nodes
-----

apt-get install smokeping
cat /etc/default/smokeping

MODE=slave

MASTER_URL=http://195.95.95.95/cgi-bin/smokeping.cgi

SHARED_SECRET=/etc/smokeping/slavesecrets.conf

Note that this is considerably unsecure configuration. Use VPN connections, firewalls or other type of http authentication,in .htaccess for example. The above 3 lines you have to have, no more or less.

echo "topsecret" > /etc/smokeping/slavesecrets.conf
ls -sal /etc/smokeping/slavesecrets.conf

Set file rights as:
-r--r-----  1 smokeping root   13 Mar  6 07:41 slavesecrets.conf

All the other files are needless here. You can safely delete the whole config.d/ directory for example. Nice, huh?

/etc/init.d/smokeping restart

 Wait some minutes and watch your slave-driven data flow under your "External" session on your Master's webpage.
In case anything going wrong - or nothing, check your apache error log:

cat /var/log/apache/error.log

Check the online manual for further reference.