21 September 2010

Monitoring of distributed servers using Zabbix

Categories:  Server  Monitoring  Linux  Gentoo

How to configure and setup distributed monitoring system Zabbix on Gentoo Linux server.

Recently I was looking for a good and scalable monitoring solution which would simplify the management of my servers. My main concern was monitoring server performance, including performance of key server services like MySQL and Apache. I would like to have a monitoring system which would be able to present the performance statistics on meaningful graphs. Previously I was using MRTG but this solution was far from being perfect. After checking few alternatives my choice was quite obvious: distributed monitoring system Zabbix seems to be the best OpenSource monitoring system.

Zabbix is fully distributed system with lots of possible configuration options. I can truly say with this tool only your imagination and sky is the limit. With additional configuration and support for Snmp and OpenIMPI protocols this system is able to monitor anything you like on any computer. The system is build out of three components: server, proxy and agent. Server is the central monitoring node which will gather all the statistics and present them using simple to use and highly configurable Web interface. Proxy can gather statistics from number of monitored machines and push them to server. The agent is small program that runs on monitored machine and gathers statistics on this machine. The server software need to run on Posix type of systems but agent programs are available for almost all operating systems. Web interface require a web server with PHP support.

I will try to share how I configured this system on my servers. As you will notice I do not use proxy instance. Most of the information provided below are compilation of two sources I found on the Internet. On the bottom part of this article you will find full sources list. I want to document this configuration on my blog hoping that someone else will find this information useful. Most of the information provided below address Gentoo Linux servers but some of them should be applicable to any Linux or possix system running Zabbix.

Current latest version of Zabbix is 1.8.3 and fortunately this is also the version available in Gentoo Linux portage tree. First thing we need to do is installation of the Zabbix server. To install Zabbix on my server I used following command:

USE="-ldap -openipmi -oracle -postgres -proxy -sqlite3 mysql agent frontend jabber server ssh snmp" emerge -av net-analyzer/zabbix

This will of course pull all needed dependencies and check currently installed packages for compatibility. It's possible that you will be asked to recompile PHP to match configuration needed by frontend. As you can see I'm using mysql database server to store monitoring information (you can optionally use postgresql oracle or sqlite database). I disabled support for ldap authentication and openimpi protocol as I don't need it. I installed server and web fronted on the same machine and I included jabber instant messaging protocol support which will let me define notification events sent to my jabber communicator account. I additionally installed agent to monitor my Zabbix server machine.

On monitored Gentoo machines installation is easier. I used following command to install agents:

USE="-ldap -openipmi -oracle -postgres -proxy -sqlite3 -mysql  -frontend -jabber -server -ssh -snmp agent" emerge -av net-analyzer/zabbix

Before creating new database for Zabbix sever default Gentoo Linux MySql configuration will require additional tuning to prevent overloading database with information. According to Gentoo Linux Wiki Archive you need to change few database parameters relating to InnoDB storage engine. But beware if you were using the database server before Zabbix installation and you have some important data stored using InnoDB storage engine I would recommend backing up whole database before changing those parameters. You can follow this link to access my database backup script and use it to create backup of all databases. If you have got the backup it's safe to change the /etc/mysql/my.cnf file and make sure that following parameters are present in [mysqld] section.

expire_logs_days = 2
innodb_data_file_path = ibdata1:128M;ibdata2:50M:autoextend:max:12800M

This configuration will force the log rotation every second day and will enlarge maximum size of InnoDB data file. You can tune up this settings but make sure you will not keep the defaults. After changing this parameters you will have to enter /var/lib/mysql/ and delete all InnoDB related files from this catalog. Most likely this will include: ib_logfile* and ibdata* files. Without this the mysql database will not start. After restarting the database service make sure your old databases are accessible. If they are not drop them and restore them from previously made backup by running following command:

mysql -u User -pPassword database_name < /your/beckup/path/beckup_file.sql

After taking care of mysql configuration it is time to create the database for Zabbix. First we will create a database for Zabbix server then we will create a user with full access to this database:

mysql -u User_with_full_db_access -pPassword
create database zabbix;
grant all on zabbix.* to zabbix_user@localhost identified by 'password';
flush privileges;
quit;

Then we can create database structure and populate the needed database information using sql scripts provided with Zabbix:

cat /usr/share/zabbix/database/create/schema/mysql.sql | mysql -u zabbix_user pPassword zabbix
cat /usr/share/zabbix/database/create/data/data.sql | mysql -u zabbix_user pPassword zabbix
cat /usr/share/zabbix/database/create/data/images_mysql.sql | mysql -u zabbix_user pPassword zabbix

Now as the database is ready we can configure the Zabbix server. To do it we need to edit /etc/zabbix/zabbix_server.conf. This file is very well documented so it shouldn't been to hard to create a configuration that will match you needs. The most important part is database connection definition placed in the bottom part of the file. This configuration should look similar to this:

# Database host name
# Default is localhost
 
DBHost=localhost
 
# Database name
# SQLite3 note: path to database file must be provided. DBUser and DBPassword are ignored.
DBName=zabbix
 
# Database user
 
DBUser=zabbix_user
 
# Database password
# Comment this line if no password used
 
DBPassword=Password
 
# Connect to MySQL using Unix socket?
 
DBSocket=/var/run/mysqld/mysqld.sock

And that is it. We are ready to start the server by running following command:

/etc/init.d/zabbix-server start

and add the server service to default run level to make it start with every single boot:

rc-update add zabbix-server default

Next thing to do is installation of web frontend. I have Apache web server so this configuration is addressing this server. In Gentoo Linux we have a great webapp-config script which is simplifying the installation. I decided to install web front end in separate virtual host. To do it I had to run following command:

webapp-config -I -h zabbix zabbix 1.8.3

This command created a catalog for web virtual host in /var/www/zabbix and placed Zabbix frontend php files inside of /var/www/zabbix/htdocs. To make my zabbix frontend accessible I had to create a virtual host configuration files and place them inside of /etc/apache2/vhosts.d/. Below you can see my Zabbix virutal hosts configuration:

View the zabbix_vhost.conf
  1. <VirtualHost *:80>
  2. ServerAdmin admin@my_domain.org
  3. DocumentRoot /var/www/zabbix/htdocs/
  4. ServerName cname.my_domain.org
  5. RewriteEngine On
  6. RewriteCond %{HTTPS} !=on
  7. RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]
  8. ErrorLog /var/log/apache2/zabbix-error_log
  9. CustomLog /var/log/apache2/zabbix-access_log common
  10. </VirtualHost>

View the zabbix_ssl_vhost.conf
  1. <VirtualHost *:443>
  2. ServerAdmin admin@my_domain.org
  3. ServerName cname.my_domain.org
  4. UseCanonicalName On
  5. SSLEngine on
  6. SSLOptions StrictRequire
  7. SSLCertificateFile /etc/ssl/apache2/server.crt
  8. SSLCertificateKeyFile /etc/ssl/apache2/server.key
  9. SSLProtocol all -SSLv2
  10. DocumentRoot /var/www/zabbix/htdocs
  11. <Directory "/var/www/zabbix/htdocs">
  12. Options -Indexes FollowSymLinks
  13. AllowOverride All
  14. Order allow,deny
  15. Allow from all
  16. SSLRequireSSL
  17. </Directory>
  18. ErrorLog /var/log/apache2/zabbix-ssl-error_log
  19. CustomLog /var/log/apache2/zabbix-ssl-access_log "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %b"
  20. </VirtualHost>

As you can see this configuration will force connection to Zabbix interface using ssl protocol. I made sure that my FQDN Server Names are properly configured in my DNS system. Last thing to do is making sure that requirements of Zabbix frontend are fulfilled. You may be surprised to see that required PHP settings are quite weird. Therefor same as Gentoo Linux Wiki Archive I would not recommend placing this settings in main php.ini file. Instead crate a .htaccess file inside of /var/www/zabbix/htdocs catalog and place required configuration in this file. For Zabbix 1.8.3 this configuration should look like this:

php_value max_execution_time 600
php_value memory_limit 256M
php_value post_max_size 32M
php_value upload_max_filesize 16M
php_value max_input_time 600
php_value upload_max_filesize 200M

Now restart the apache server:

/etc/init.d/apache2 restart

and point your browser to your Zabbix address. This will fire up installation interface which will check your configuration and ask you to provide needed configuration options. In the end this interface will let you save configuration file. You should place this file in /var/zabbix/htdocs/conf catalog. And that is it you are ready to use Zabbix web interface.

Now we need some monitored computers. On every monitored server we should edit /etc/zabbix/zabbix_agentd.conf agent configuration file and change:

Server=127.0.0.1

line to point to your server IP. You should also change the:

Hostname=your_hostname

line to provide a unique hostname for your monitored server. This is required for active checks.

As Zabbix comes with a lot of default monitoring events this is all you have to do to make it work, but as I wrote on the very beginning I would like to monitor performance of Apache and MySql services on my servers. To do it I have to define additional user monitoring parameters. Sample configuration for MySQL database was available in /etc/zabbix/zabbix_agentd.conf I just had to uncomment some of the lines. I added some additional MySql monitoring options using:

SHOW GLOBAL STATUS;

mysql query. For Apache monitoring I used configuration proposed by Gentoo Linux Wiki Archive. Below you can find full list of user defined parameters coming from my monitored servers.

View the UserParamters configuration
  1. #MySql Uptime
  2. UserParameter=mysql.uptime,mysqladmin -uUser -pPassword status|cut -f2 -d":"|cut -f1 -d"T"
  3. #MySql number of queries sent to the server
  4. UserParameter=mysql.questions,mysqladmin -uUser -pPassword status|cut -f4 -d":"|cut -f1 -d"S"
  5. #MySql SlowQueries
  6. UserParameter=mysql.slowqueries,mysqladmin -uUser -pPassword status|cut -f5 -d":"|cut -f1 -d"O"
  7. #MySql Queries Per Second
  8. UserParameter=mysql.qps,mysqladmin -uUser -pPassword status|cut -f9 -d":"
  9. #Mysql Bytes recived
  10. UserParameter=mysql.bytes_received,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Bytes_received | awk '{ print $2; }'
  11. #MySql Bytes sent
  12. UserParameter=mysql.bytes_sent,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Bytes_sent | awk '{ print $2; }'
  13. #MySql Connections
  14. UserParameter=mysql.connections,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Connections | awk '{ print $2; }'
  15. #MySql Key Reads
  16. UserParameter=mysql.key_reads,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Key_reads | awk '{ print $2; }'
  17. #MySql Key Writes
  18. UserParameter=mysql.key_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Key_writes | awk '{ print $2; }'
  19. #MySql Threads Running
  20. UserParameter=mysql.threads_running,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Threads_running | awk '{ print $2; }'
  21. #MySql Threads Connected
  22. UserParameter=mysql.threads_connected,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Threads_connected | awk '{ print $2; }'
  23. #MySql InnoDB Buffer Pool Read
  24. UserParameter=mysql.innodb_pool_read,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_buffer_pool_read_requests | awk '{ print $2; }'
  25. #MySql InnoDB Buffer Pool Writes
  26. UserParameter=mysql.innodb_pool_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_buffer_pool_write_requests | awk '{ print $2; }'
  27. #MySql InnoDB Data Reads
  28. UserParameter=mysql.innodb_data_reads,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_data_reads | awk '{ print $2; }'
  29. #MySql InnoDB Data Writes
  30. UserParameter=mysql.innodb_data_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_data_writes | awk '{ print $2; }'
  31. #Apache monitoring
  32. #Total Accesses
  33. UserParameter=apache.accesses,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep Accesses | cut -d " " -f 3
  34. #Total kBytes
  35. UserParameter=apache.totalkb,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep kBytes | cut -d " " -f 3
  36. #CPU Load
  37. UserParameter=apache.cpuload,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep CPULoad | cut -d " " -f 2
  38. #Apache uptime
  39. UserParameter=apache.uptime,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep Uptime | cut -d " " -f 2
  40. #Requests per second
  41. UserParameter=apache.reqpersec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep ReqPerSec | cut -d " " -f 2
  42. #Bytes per second
  43. UserParameter=apache.bytespersec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BytesPerSec | cut -d " " -f 2
  44. #Bytes per request
  45. UserParameter=apache.bytesperrec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BytesPerReq | cut -d " " -f 2
  46. #Busy Workers
  47. UserParameter=apache.busywokers,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BusyWorkers | cut -d " " -f 2
  48. #Idle Workers
  49. UserParameter=apache.idleworkers,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep IdleWorkers | cut -d " " -f 2

This is all configuration I use, but as Zabbix is awesome tool I will surly add something to it in the feature. Last thing left to do is starting Zabbix agent on all monitored servers:

/etc/init.d/zabbix-agentd start

and add it to default run level to make it start with every system boot:

rc-update add zabbix-agentd default

I did one more thing. As I use Monit to control all running services on my servers I added a Zabbix server and agent service configuration to my /etc/monitrc configuration file. Server Monit configuration is looking like this:

###################################################
#Check zabbix server
###################################################
 
check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid
   group monitoring
   start program = "/etc/init.d/zabbix-server start"
   stop program = "/etc/init.d/zabbix-server stop"
   if failed host 127.0.0.1 port 10051 then restart
   if 5 restarts within 5 cycles then timeout

and agent Monit configuration is looking like this:

###################################################
#Check zabbix agent
###################################################
 
check process zabbix-agentd with pidfile /var/run/zabbix/zabbix_agentd.pid
   group monitoring
   start program = "/etc/init.d/zabbix-agentd start"
   stop program = "/etc/init.d/zabbix-agentd stop"
   if failed host 127.0.0.1 port 10050 then restart
   if 5 restarts within 5 cycles then timeout

Now you can start to monitor your servers and create a detailed monitoring configuration using Zabbix Web Interface. If you want to construct graphs from User Parameters you will have to create events for them providing UserParameter key as defined in /etc/zabbix/zabbix-agentd configuration. You can consult the screenshot below:

Zabbix Web FrontEnd

With Zabbix you can easily create graphs like the one presented below:

Zabbix Web FrontEnd

You can define the screens to present group of graphs one next to another like it's shown on the screenshots below:

Zabbix Web FrontEnd

You can also define slideshows, triggers and many more useful things with Zabbix. Make sure you will take a look at Zabbix documentation. You will find there more useful information about this great tool.

Sources:




Comments

If you have found something wrong with the information provided above or maybe you just want to speak your mind about it, feel free to leave a comment.
All comments will show up on page after being approved. Sorry for such policy but I want to make sure that my site will be free of abusive or vulgar content. I don't mind being criticized just do it using right words.

Leave a comment

  • Posted on:
  • Author:
  •  
  • Comment:
  • 12.03.2012
  • kardasa
  •  
  • "answer":
    I'm not sure what do you mean by template? The definition is preaty simple: UserParameter= key_used_in_zabbix_interface, command that will print some information
  • 21.02.2012
  • Ben
  •  
  • "Request":
    Can you provide a template with the items using the keys declared by your UserParameters ? thanks