HOWTO devops log_analysis
To quickly analyze existing log files, mainly the access.log of the web server
and the production.log from the app server, we use a couple of command line tools that need to be launched from the Bash shell after having copied the log files locally.
Some tools:
- Pure Ruby request-log-analyzer gem
- Bash utility GoAccess (needs to be compiled and installed)
- Java utility LogStash download (needs Java 8)
request-log-analyzer is a gem dedicated to quickly analyze Rails environment log files.
GoAccess works almost instantly with the standard Apache2 access.log file and it's dedicated to the anaysis of web servers'access log files.
We haven't tested LogStash with our current configuration yet, although is highly customizable.
Copy the log files locally with:
$ scp root@<SERVER_IP>:/var/www/goggles.org/shared/log/*.log .
Requires the dev packages for ncurses and the geo-ip location library (which must be installed previously via package manager).
$ wget http://tar.goaccess.io/goaccess-1.1.1.tar.gz
$ tar -xzvf goaccess-1.1.1.tar.gz
$ cd goaccess-1.1.1/
$ ./configure --enable-geoip --enable-utf8
$ make
$ su
# make install
Execute from bash to get an interactive report:
$ goaccess -f access.log
Most of the times (depending on current locale) it is required to select the time format
at the beginning (first choice from the menu -> SPACE bar to select - > ENTER to start the log parsing)
The report is divided in sections and easily browsable using TAB or the cursor keys.
Install (out of the project bundle):
$ gem install request-log-analyzer
Execute and get a quick report on the Bash shell:
$ request-log-analyzer production.log
