Monitoring of distributed servers using Zabbix
How to configure and setup distributed monitoring system Zabbix on Gentoo Linux server.
Recently I was looking for a good and scalable monitoring solution which would simplify the management of my servers. My main concern was monitoring server performance, including performance of key server services like MySQL and Apache. I would like to have a monitoring system which would be able to present the performance statistics on meaningful graphs. Previously I was using MRTG but this solution was far from being perfect. After checking few alternatives my choice was quite obvious: distributed monitoring system Zabbix seems to be the best OpenSource monitoring system.
Zabbix is fully distributed system with lots of possible configuration options. I can truly say with this tool only your imagination and sky is the limit. With additional configuration and support for Snmp and OpenIMPI protocols this system is able to monitor anything you like on any computer. The system is build out of three components: server, proxy and agent. Server is the central monitoring node which will gather all the statistics and present them using simple to use and highly configurable Web interface. Proxy can gather statistics from number of monitored machines and push them to server. The agent is small program that runs on monitored machine and gathers statistics on this machine. The server software need to run on Posix type of systems but agent programs are available for almost all operating systems. Web interface require a web server with PHP support.
I will try to share how I configured this system on my servers. As you will notice I do not use proxy instance. Most of the information provided below are compilation of two sources I found on the Internet. On the bottom part of this article you will find full sources list. I want to document this configuration on my blog hoping that someone else will find this information useful. Most of the information provided below address Gentoo Linux servers but some of them should be applicable to any Linux or possix system running Zabbix.
Current latest version of Zabbix is 1.8.3 and fortunately this is also the version available in Gentoo Linux portage tree. First thing we need to do is installation of the Zabbix server. To install Zabbix on my server I used following command:
USE="-ldap -openipmi -oracle -postgres -proxy -sqlite3 mysql agent frontend jabber server ssh snmp" emerge -av net-analyzer/zabbix
This will of course pull all needed dependencies and check currently installed packages for compatibility. It's possible that you will be asked to recompile PHP to match configuration needed by frontend. As you can see I'm using mysql database server to store monitoring information (you can optionally use postgresql oracle or sqlite database). I disabled support for ldap authentication and openimpi protocol as I don't need it. I installed server and web fronted on the same machine and I included jabber instant messaging protocol support which will let me define notification events sent to my jabber communicator account. I additionally installed agent to monitor my Zabbix server machine.
On monitored Gentoo machines installation is easier. I used following command to install agents:
USE="-ldap -openipmi -oracle -postgres -proxy -sqlite3 -mysql -frontend -jabber -server -ssh -snmp agent" emerge -av net-analyzer/zabbix
Before creating new database for Zabbix sever default Gentoo Linux MySql configuration will require additional tuning to prevent overloading database with information. According to Gentoo Linux Wiki Archive you need to change few database parameters relating to InnoDB storage engine. But beware if you were using the database server before Zabbix installation and you have some important data stored using InnoDB storage engine I would recommend backing up whole database before changing those parameters. You can follow this link to access my database backup script and use it to create backup of all databases. If you have got the backup it's safe to change the /etc/mysql/my.cnf file and make sure that following parameters are present in [mysqld] section.
expire_logs_days = 2
innodb_data_file_path = ibdata1:128M;ibdata2:50M:autoextend:max:12800M
This configuration will force the log rotation every second day and will enlarge maximum size of InnoDB data file. You can tune up this settings but make sure you will not keep the defaults. After changing this parameters you will have to enter /var/lib/mysql/ and delete all InnoDB related files from this catalog. Most likely this will include: ib_logfile* and ibdata* files. Without this the mysql database will not start. After restarting the database service make sure your old databases are accessible. If they are not drop them and restore them from previously made backup by running following command:
mysql -u User -pPassword database_name < /your/beckup/path/beckup_file.sql
After taking care of mysql configuration it is time to create the database for Zabbix. First we will create a database for Zabbix server then we will create a user with full access to this database:
mysql -u User_with_full_db_access -pPassword create database zabbix; grant all on zabbix.* to zabbix_user@localhost identified by 'password'; flush privileges; quit;
Then we can create database structure and populate the needed database information using sql scripts provided with Zabbix:
cat /usr/share/zabbix/database/create/schema/mysql.sql | mysql -u zabbix_user pPassword zabbix cat /usr/share/zabbix/database/create/data/data.sql | mysql -u zabbix_user pPassword zabbix cat /usr/share/zabbix/database/create/data/images_mysql.sql | mysql -u zabbix_user pPassword zabbix
Now as the database is ready we can configure the Zabbix server. To do it we need to edit /etc/zabbix/zabbix_server.conf. This file is very well documented so it shouldn't been to hard to create a configuration that will match you needs. The most important part is database connection definition placed in the bottom part of the file. This configuration should look similar to this:
# Database host name # Default is localhost DBHost=localhost # Database name # SQLite3 note: path to database file must be provided. DBUser and DBPassword are ignored. DBName=zabbix # Database user DBUser=zabbix_user # Database password # Comment this line if no password used DBPassword=Password # Connect to MySQL using Unix socket? DBSocket=/var/run/mysqld/mysqld.sock
And that is it. We are ready to start the server by running following command:
/etc/init.d/zabbix-server start
and add the server service to default run level to make it start with every single boot:
rc-update add zabbix-server default
Next thing to do is installation of web frontend. I have Apache web server so this configuration is addressing this server. In Gentoo Linux we have a great webapp-config script which is simplifying the installation. I decided to install web front end in separate virtual host. To do it I had to run following command:
webapp-config -I -h zabbix zabbix 1.8.3
This command created a catalog for web virtual host in /var/www/zabbix and placed Zabbix frontend php files inside of /var/www/zabbix/htdocs. To make my zabbix frontend accessible I had to create a virtual host configuration files and place them inside of /etc/apache2/vhosts.d/. Below you can see my Zabbix virutal hosts configuration:
<VirtualHost *:80> ServerAdmin admin@my_domain.org DocumentRoot /var/www/zabbix/htdocs/ ServerName cname.my_domain.org RewriteEngine On RewriteCond %{HTTPS} !=on RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L] ErrorLog /var/log/apache2/zabbix-error_log CustomLog /var/log/apache2/zabbix-access_log common </VirtualHost>
<VirtualHost *:443> ServerAdmin admin@my_domain.org ServerName cname.my_domain.org UseCanonicalName On SSLEngine on SSLOptions StrictRequire SSLCertificateFile /etc/ssl/apache2/server.crt SSLCertificateKeyFile /etc/ssl/apache2/server.key SSLProtocol all -SSLv2 DocumentRoot /var/www/zabbix/htdocs <Directory "/var/www/zabbix/htdocs"> Options -Indexes FollowSymLinks AllowOverride All Order allow,deny Allow from all SSLRequireSSL </Directory> ErrorLog /var/log/apache2/zabbix-ssl-error_log CustomLog /var/log/apache2/zabbix-ssl-access_log "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %b" </VirtualHost>
As you can see this configuration will force connection to Zabbix interface using ssl protocol. I made sure that my FQDN Server Names are properly configured in my DNS system. Last thing to do is making sure that requirements of Zabbix frontend are fulfilled. You may be surprised to see that required PHP settings are quite weird. Therefor same as Gentoo Linux Wiki Archive I would not recommend placing this settings in main php.ini file. Instead crate a .htaccess file inside of /var/www/zabbix/htdocs catalog and place required configuration in this file. For Zabbix 1.8.3 this configuration should look like this:
php_value max_execution_time 600 php_value memory_limit 256M php_value post_max_size 32M php_value upload_max_filesize 16M php_value max_input_time 600 php_value upload_max_filesize 200M
Now restart the apache server:
/etc/init.d/apache2 restart
and point your browser to your Zabbix address. This will fire up installation interface which will check your configuration and ask you to provide needed configuration options. In the end this interface will let you save configuration file. You should place this file in /var/zabbix/htdocs/conf catalog. And that is it you are ready to use Zabbix web interface.
Now we need some monitored computers. On every monitored server we should edit /etc/zabbix/zabbix_agentd.conf agent configuration file and change:
Server=127.0.0.1
line to point to your server IP. You should also change the:
Hostname=your_hostname
line to provide a unique hostname for your monitored server. This is required for active checks.
As Zabbix comes with a lot of default monitoring events this is all you have to do to make it work, but as I wrote on the very beginning I would like to monitor performance of Apache and MySql services on my servers. To do it I have to define additional user monitoring parameters. Sample configuration for MySQL database was available in /etc/zabbix/zabbix_agentd.conf I just had to uncomment some of the lines. I added some additional MySql monitoring options using:
SHOW GLOBAL STATUS;
mysql query. For Apache monitoring I used configuration proposed by Gentoo Linux Wiki Archive. Below you can find full list of user defined parameters coming from my monitored servers.
#MySql Uptime UserParameter=mysql.uptime,mysqladmin -uUser -pPassword status|cut -f2 -d":"|cut -f1 -d"T" #MySql number of queries sent to the server UserParameter=mysql.questions,mysqladmin -uUser -pPassword status|cut -f4 -d":"|cut -f1 -d"S" #MySql SlowQueries UserParameter=mysql.slowqueries,mysqladmin -uUser -pPassword status|cut -f5 -d":"|cut -f1 -d"O" #MySql Queries Per Second UserParameter=mysql.qps,mysqladmin -uUser -pPassword status|cut -f9 -d":" #Mysql Bytes recived UserParameter=mysql.bytes_received,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Bytes_received | awk '{ print $2; }' #MySql Bytes sent UserParameter=mysql.bytes_sent,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Bytes_sent | awk '{ print $2; }' #MySql Connections UserParameter=mysql.connections,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Connections | awk '{ print $2; }' #MySql Key Reads UserParameter=mysql.key_reads,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Key_reads | awk '{ print $2; }' #MySql Key Writes UserParameter=mysql.key_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Key_writes | awk '{ print $2; }' #MySql Threads Running UserParameter=mysql.threads_running,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Threads_running | awk '{ print $2; }' #MySql Threads Connected UserParameter=mysql.threads_connected,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Threads_connected | awk '{ print $2; }' #MySql InnoDB Buffer Pool Read UserParameter=mysql.innodb_pool_read,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_buffer_pool_read_requests | awk '{ print $2; }' #MySql InnoDB Buffer Pool Writes UserParameter=mysql.innodb_pool_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_buffer_pool_write_requests | awk '{ print $2; }' #MySql InnoDB Data Reads UserParameter=mysql.innodb_data_reads,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_data_reads | awk '{ print $2; }' #MySql InnoDB Data Writes UserParameter=mysql.innodb_data_writes,echo 'SHOW GLOBAL STATUS;' | mysql -u User --password=Password | grep Innodb_data_writes | awk '{ print $2; }' #Apache monitoring #Total Accesses UserParameter=apache.accesses,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep Accesses | cut -d " " -f 3 #Total kBytes UserParameter=apache.totalkb,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep kBytes | cut -d " " -f 3 #CPU Load UserParameter=apache.cpuload,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep CPULoad | cut -d " " -f 2 #Apache uptime UserParameter=apache.uptime,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep Uptime | cut -d " " -f 2 #Requests per second UserParameter=apache.reqpersec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep ReqPerSec | cut -d " " -f 2 #Bytes per second UserParameter=apache.bytespersec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BytesPerSec | cut -d " " -f 2 #Bytes per request UserParameter=apache.bytesperrec,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BytesPerReq | cut -d " " -f 2 #Busy Workers UserParameter=apache.busywokers,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep BusyWorkers | cut -d " " -f 2 #Idle Workers UserParameter=apache.idleworkers,wget --quiet -O - http://localhost/server-status?auto | head -n 9 | grep IdleWorkers | cut -d " " -f 2
This is all configuration I use, but as Zabbix is awesome tool I will surly add something to it in the feature. Last thing left to do is starting Zabbix agent on all monitored servers:
/etc/init.d/zabbix-agentd start
and add it to default run level to make it start with every system boot:
rc-update add zabbix-agentd default
I did one more thing. As I use Monit to control all running services on my servers I added a Zabbix server and agent service configuration to my /etc/monitrc configuration file. Server Monit configuration is looking like this:
################################################### #Check zabbix server ################################################### check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid group monitoring start program = "/etc/init.d/zabbix-server start" stop program = "/etc/init.d/zabbix-server stop" if failed host 127.0.0.1 port 10051 then restart if 5 restarts within 5 cycles then timeout
and agent Monit configuration is looking like this:
################################################### #Check zabbix agent ################################################### check process zabbix-agentd with pidfile /var/run/zabbix/zabbix_agentd.pid group monitoring start program = "/etc/init.d/zabbix-agentd start" stop program = "/etc/init.d/zabbix-agentd stop" if failed host 127.0.0.1 port 10050 then restart if 5 restarts within 5 cycles then timeout
Now you can start to monitor your servers and create a detailed monitoring configuration using Zabbix Web Interface. If you want to construct graphs from User Parameters you will have to create events for them providing UserParameter key as defined in /etc/zabbix/zabbix-agentd configuration. You can consult the screenshot below:
![Zabbix Web FrontEnd](uploads/images/blog/Zabbix01.jpg)
With Zabbix you can easily create graphs like the one presented below:
![Zabbix Web FrontEnd](uploads/images/blog/Zabbix02.jpg)
You can define the screens to present group of graphs one next to another like it's shown on the screenshots below:
![Zabbix Web FrontEnd](uploads/images/blog/Zabbix03.jpg)
You can also define slideshows, triggers and many more useful things with Zabbix. Make sure you will take a look at Zabbix documentation. You will find there more useful information about this great tool.
Sources:
- Zabbix documentation
- Zabbix Gentoo Linux Wiki Archive.
GNU Free Documentation License or Creative Commons Share Alike
- Posted on:
- Comment:
- 12.03.2012
- "answer":I'm not sure what do you mean by template? The definition is preaty simple: UserParameter= key_used_in_zabbix_interface, command that will print some information
- 21.02.2012
- "Request":Can you provide a template with the items using the keys declared by your UserParameters ? thanks
If you have found something wrong with the information provided above or maybe you just want to speak your mind about it, feel free to leave a comment.
All comments will show up on page after being approved. Sorry for such policy but I want to make sure that my site will be free of abusive or vulgar content. I don't mind being criticized just do it using right words.
Processing a comment.