Hello, when I wish to check the load of web pages in the server, I parse the apache logs like this: Code: # tail -f /var/www/clients/client*/web*/log/access.log # tail -f /var/log/apache2/other_vhosts_access.log although this is not very comfortable and clean. Do you know any tool or script designed to monitor the load of webpages in real time?
What do you want to monitor in realtime at all? If you want to view your webserver log files in realtime then I can only recommend to use GoAccess. Althouth ISPConfig has basic GoAccess support, the GoAccess with the realtime feature must be started individually from the command-line.
You can use logger. Code: ErrorLog "|/usr/bin/logger -S 32768 -plocal1.notice -tapache.err" CustomLog "|/usr/bin/logger -S 32768 -plocal1.notice -tapache.log" combined to transport log entries up to a size of 32k ( -S ) to your local syslog /etc/rsyslog.conf make sure you have it configured to receive this Code: $MaxMessageSize 56k module(load="imuxsock") # provides support for local system logging e.g. /etc/rsyslog.d/10-apache.conf Code: :syslogtag, isequal, "apache.err:" @127.0.0.1:1027 & stop :syslogtag, isequal, "apache.log:" @127.0.0.1:1028 & stop :syslogtag, isequal, "apache.forensic:" @127.0.0.1:1029 & stop which tells it to forward the logs to e.g. filebeat /etc/filebeat/filebeat.yml Code: - type: syslog enabled: true tags: ["www1-error"] max_message_size: 64KiB format: auto keep_null: true protocol.udp: host: "127.0.0.1:1027" - type: syslog enabled: true tags: ["www1-access"] max_message_size: 64KiB format: auto keep_null: true protocol.udp: host: "127.0.0.1:1028" - type: syslog enabled: true tags: ["www1-forensic"] max_message_size: 64KiB format: auto keep_null: true protocol.udp: host: "127.0.0.1:1029" .... output.logstash: # The Logstash hosts hosts: ["ip.......:2443", "ip....:2443"] have your /etc/logstash/conf.d/accesslog.conf e.g. Code: grok { # match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:level}\] \[%{DATA:module}\] \[(?:%{DATA:referrer}|-)\] (\[%{IP:clientip}:%{NUMBER:port}\] )?(\[%{DATA:forensic}\] ?)%{DATA:ah}:%{GREEDYDATA:error}" } match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:level}\] \[%{DATA:module}\] \[%{DATA:apros}\] \[%{DATA:referrer}\] \[((%{IP:clientip}:%{NUMBER:port})|-)\] \[%{NOTSPACE:forensic}\] %{DATA:ah}:%{GREEDYDATA:error}" } } if [clientip] { geoip { default_database_type => "City" source => "clientip" target => "client" ecs_compatibility => "v8" tag_on_failure => ["_city_geo_failure"] } those are just examples of course and you'd need to setup your logformat on apache, make your grok code as you please and configure your open or elasticsearch instance for rollover and stuff but that's a different topic. At least this is how I do this for a webserver-farm of about 50 apache-only instances serving about 400m api hits a day juste be aware, this is not using any authentication, but I do not have any users on my servers except one, so if he want to tamper with the logs, he maybe can but uhm highly unlikely =) not only is this "live" enough but it also reduces wear on nvme To make it more, but not entirely, secure, one could use "password" as a tag, though I'd rather not suppose this as a valid solution
the problem with GoAccess is that it is not possible (or I don't know) how to build my own format. I would need a simple thing, something similar to this able to be updated in real time: Code: IP Host lookup Page Method Country Time 4.4.4.4 ip.provider.what.net http://domain.com/page?dadas.php POST US 16:59:02 5.24.54.14 ip.provide3.what.com http://domain2.com/index.html GET NL 16:59:06 the issue is monitoring the page requests from the different websites in real time. It is a simple thing, although I cannot find something similar for the terminal. wow, that's awesome. Thanks a lot. My necessities are simpler, although if I cannot find some tool I will look your code with more time and maybe it could work to me. Thanks!
My solution would be rather complex involving an elasticsearch cluster with kibana, logstash and filebeat all with their own caveats. Grok is not the easiest scripting language but it can be done. Regarding to your mentiones format in #4 do you need the other access log format still? Otherwise you could sure enough create your own log format. Just be careful with ip / dns lookups, this can stall your webservers performance while it does the lookup for the log output.
thanks. ztkm.me. Your solution is good to keep in mind . Yes. Because the problems with real-time update and dns lookup, I was searching for a free tool for the terminal maybe made in C++ using threads or a similar thing. Although I cannot find some. I have found this PHP utility which could be executed in the terminal https://github.com/uuf6429/httpdmon Seems not enough fast although I will test this thx!
apachetop -d 1 -f /var/log/apache2/other_vhosts_access.log or apachetop -d 1 -f /var/www/<domainame>/log/access.log for a specific website.. you can probably skip the -d 1 part as well.. might even be a bit easier to use without that bit.
