How to monitor the load of webpages in real time?

Discussion in 'Server Operation' started by MaxT, May 31, 2024.

  1. MaxT

    MaxT Active Member HowtoForge Supporter

    Hello,
    when I wish to check the load of web pages in the server, I parse the apache logs like this:

    Code:
    # tail -f /var/www/clients/client*/web*/log/access.log
    # tail -f /var/log/apache2/other_vhosts_access.log
    although this is not very comfortable and clean.

    Do you know any tool or script designed to monitor the load of webpages in real time?
     
  2. michelangelo

    michelangelo Active Member

    What do you want to monitor in realtime at all?

    If you want to view your webserver log files in realtime then I can only recommend to use GoAccess.
    Althouth ISPConfig has basic GoAccess support, the GoAccess with the realtime feature must be started individually from the command-line.
     
    MaxT and ahrasis like this.
  3. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    You can use logger.
    Code:
            ErrorLog "|/usr/bin/logger -S 32768 -plocal1.notice -tapache.err"
            CustomLog "|/usr/bin/logger -S 32768 -plocal1.notice -tapache.log" combined
    
    to transport log entries up to a size of 32k ( -S ) to your local syslog

    /etc/rsyslog.conf
    make sure you have it configured to receive this
    Code:
    $MaxMessageSize 56k
    module(load="imuxsock") # provides support for local system logging
    
    e.g.
    /etc/rsyslog.d/10-apache.conf
    Code:
    :syslogtag, isequal, "apache.err:" @127.0.0.1:1027
    & stop
    
    :syslogtag, isequal, "apache.log:" @127.0.0.1:1028
    & stop
    
    :syslogtag, isequal, "apache.forensic:" @127.0.0.1:1029
    & stop
    
    which tells it to forward the logs to e.g. filebeat
    /etc/filebeat/filebeat.yml
    Code:
    - type: syslog
      enabled: true
      tags: ["www1-error"]
      max_message_size: 64KiB
      format: auto
      keep_null: true
      protocol.udp:
        host: "127.0.0.1:1027"
    
    
    
    - type: syslog
      enabled: true
      tags: ["www1-access"]
      max_message_size: 64KiB
      format: auto
      keep_null: true
      protocol.udp:
        host: "127.0.0.1:1028"
    
    
    
    - type: syslog
      enabled: true
      tags: ["www1-forensic"]
      max_message_size: 64KiB
      format: auto
      keep_null: true
      protocol.udp:
        host: "127.0.0.1:1029"
    
    ....
    
    output.logstash:
      # The Logstash hosts
      hosts: ["ip.......:2443", "ip....:2443"]
    
    
    have your
    /etc/logstash/conf.d/accesslog.conf
    e.g.
    Code:
      grok {
    #    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:level}\] \[%{DATA:module}\] \[(?:%{DATA:referrer}|-)\] (\[%{IP:clientip}:%{NUMBER:port}\] )?(\[%{DATA:forensic}\] ?)%{DATA:ah}:%{GREEDYDATA:error}" }
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:level}\] \[%{DATA:module}\] \[%{DATA:apros}\] \[%{DATA:referrer}\] \[((%{IP:clientip}:%{NUMBER:port})|-)\] \[%{NOTSPACE:forensic}\] %{DATA:ah}:%{GREEDYDATA:error}" }
      }
    
      if [clientip]  {
    
        geoip {
          default_database_type => "City"
          source => "clientip"
          target => "client"
          ecs_compatibility => "v8"
          tag_on_failure => ["_city_geo_failure"]
        }
    

    those are just examples of course and you'd need to setup your logformat on apache, make your grok code as you please and
    configure your open or elasticsearch instance for rollover and stuff but that's a different topic.
    At least this is how I do this for a webserver-farm of about 50 apache-only instances serving about 400m api hits a day ;)

    juste be aware, this is not using any authentication, but I do not have any users on my servers except one, so if he want to tamper with the logs, he maybe can but uhm highly unlikely =)
    not only is this "live" enough but it also reduces wear on nvme

    To make it more, but not entirely, secure, one could use "password" as a tag, though I'd rather not suppose this as a valid solution
     

    Attached Files:

    Last edited: May 31, 2024
    MaxT and ahrasis like this.
  4. MaxT

    MaxT Active Member HowtoForge Supporter

    the problem with GoAccess is that it is not possible (or I don't know) how to build my own format. I would need a simple thing, something similar to this able to be updated in real time:
    Code:
    IP                Host  lookup                 Page                                 Method     Country            Time
    4.4.4.4         ip.provider.what.net           http://domain.com/page?dadas.php      POST         US                 16:59:02
    5.24.54.14      ip.provide3.what.com        http://domain2.com/index.html             GET           NL                 16:59:06
    
    the issue is monitoring the page requests from the different websites in real time.
    It is a simple thing, although I cannot find something similar for the terminal.

    wow, that's awesome. Thanks a lot. :)
    My necessities are simpler, although if I cannot find some tool I will look your code with more time and maybe it could work to me.
    Thanks!
     
  5. ztk.me

    ztk.me Well-Known Member HowtoForge Supporter

    My solution would be rather complex involving an elasticsearch cluster with kibana, logstash and filebeat all with their own caveats.
    Grok is not the easiest scripting language but it can be done.

    Regarding to your mentiones format in #4 do you need the other access log format still? Otherwise you could sure enough create your own log format. Just be careful with ip / dns lookups, this can stall your webservers performance while it does the lookup for the log output.
     
    MaxT likes this.
  6. MaxT

    MaxT Active Member HowtoForge Supporter

    thanks. ztkm.me. Your solution is good to keep in mind .
    Yes. Because the problems with real-time update and dns lookup, I was searching for a free tool for the terminal maybe made in C++ using threads or a similar thing. Although I cannot find some.

    I have found this PHP utility which could be executed in the terminal https://github.com/uuf6429/httpdmon
    Seems not enough fast although I will test this

    thx!
     
    ztk.me likes this.
  7. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    apachetop -d 1 -f /var/log/apache2/other_vhosts_access.log

    or apachetop -d 1 -f /var/www/<domainame>/log/access.log for a specific website..

    you can probably skip the -d 1 part as well.. might even be a bit easier to use without that bit.
     

Share This Page