Munin in Deb 11: problems with 403 forbidden

Discussion in 'HOWTO-Related Questions' started by schwim, Mar 14, 2023.

  1. schwim

    schwim Member HowtoForge Supporter

    Hello there, all!

    I'm trying to implement Munin and monit via this Deb 10 tutorial but have run into an issue where the domain.com/munin results in a 403 forbidden. I've followed the instructions, adapting where necessary due to I guess changes between deb 10 and 11. My alterations to files are as follows:

    /etc/munin/munin.conf
    Code:
    # Example configuration file for Munin, generated by 'make build'
    
    # The next three variables specifies where the location of the RRD
    # databases, the HTML output, logs and the lock/pid files.  They all
    # must be writable by the user running munin-cron.  They are all
    # defaulted to the values you see here.
    #
    dbdir   /var/lib/munin
    htmldir /var/cache/munin/www
    logdir /var/log/munin
    rundir  /var/run/munin
    
    # Where to look for the HTML templates
    #
    tmpldir   /etc/munin/templates
    
    # Where to look for the static www files
    #
    #staticdir /etc/munin/static
    
    # temporary cgi files are here. note that it has to be writable by
    # the cgi user (usually nobody or httpd).
    #
    # cgitmpdir /var/lib/munin/cgi-tmp
    
    # (Exactly one) directory to include all files from.
    includedir /etc/munin/munin-conf.d
    
    # You can choose the time reference for "DERIVE" like graphs, and show
    # "per minute", "per hour" values instead of the default "per second"
    #
    #graph_period second
    
    # Graphics files are generated either via cron or by a CGI process.
    # See http://munin-monitoring.org/wiki/CgiHowto2 for more
    # documentation.
    # Since 2.0, munin-graph has been rewritten to use the cgi code.
    # It is single threaded *by design* now.
    #
    #graph_strategy cron
    
    # munin-cgi-graph is invoked by the web server up to very many times at the
    # same time.  This is not optimal since it results in high CPU and memory
    # consumption to the degree that the system can thrash.  Again the default is
    # 6.  Most likely the optimal number for max_cgi_graph_jobs is the same as
    # max_graph_jobs.
    #
    #munin_cgi_graph_jobs 6
    
    # If the automatic CGI url is wrong for your system override it here:
    #
    #cgiurl_graph /munin-cgi/munin-cgi-graph
    
    # max_size_x and max_size_y are the max size of images in pixel.
    # Default is 4000. Do not make it too large otherwise RRD might use all
    # RAM to generate the images.
    #
    #max_size_x 4000
    #max_size_y 4000
    
    # HTML files are normally generated by munin-html, no matter if the
    # files are used or not. You can change this to on-demand generation
    # by following the instructions in http://munin-monitoring.org/wiki/CgiHowto2
    #
    # Notes:
    # - moving to CGI for HTML means you cannot have graph generated by cron.
    # - cgi html has some bugs, mostly you still have to launch munin-html by hand
    #
    #html_strategy cron
    
    # munin-update runs in parallel.
    #
    # The default max number of processes is 16, and is probably ok for you.
    #
    # If set too high, it might hit some process/ram/filedesc limits.
    # If set too low, munin-update might take more than 5 min.
    #
    # If you want munin-update to not be parallel set it to 0.
    #
    #max_processes 16
    
    # RRD updates are per default, performed directly on the rrd files.
    # To reduce IO and enable the use of the rrdcached, uncomment it and set it to
    # the location of the socket that rrdcached uses.
    #
    #rrdcached_socket /var/run/rrdcached.sock
    
    # Drop [email protected] and [email protected] an email everytime
    # something changes (OK -> WARNING, CRITICAL -> OK, etc)
    #contact.someuser.command mail -s "Munin ${var:worst}: ${var:group}::${var:host}::${var:plugin}" [email protected]
    #contact.anotheruser.command mail -s "Munin ${var:worst}: ${var:group}::${var:host}::${var:plugin}" [email protected]
    #
    # For those with Nagios, the following might come in handy. In addition,
    # the services must be defined in the Nagios server as well.
    #contact.nagios.command /usr/bin/send_nsca nagios.host.comm -c /etc/nsca.conf
    
    # The maximum time the munin-update may take to get updates from all nodes,
    # this might be interesting when using munin-async in case of large transactions and/or backlog.
    # When using the munin protocol to connect to a node, then this value shouldn't be set higher than 240.
    # In case it's higher, gaps might be seen in the graphs.
    timeout_fetch_all_nodes 240
    
    # The maximum amount of time in seconds we may work on 1 node.
    # The value will be limited with timeout_fetch_all_nodes.
    timeout_fetch_one_node 180
    
    # a simple host tree
    [system.schwimserver.com]
        address 127.0.0.1
        use_node_name yes
    
    #
    # A more complex example of a host tree
    #
    ## First our "normal" host.
    # [fii.foo.com]
    #       address foo
    #
    ## Then our other host...
    # [fay.foo.com]
    #       address fay
    #
    ## IPv6 host. note that the ip adress has to be in brackets
    # [ip6.foo.com]
    #       address [2001::1234:1]
    #
    ## Then we want totals...
    # [foo.com;Totals] #Force it into the "foo.com"-domain...
    #       update no   # Turn off data-fetching for this "host".
    #
    #   # The graph "load1". We want to see the loads of both machines...
    #   # "fii=fii.foo.com:load.load" means "label=machine:graph.field"
    #       load1.graph_title Loads side by side
    #       load1.graph_order fii=fii.foo.com:load.load fay=fay.foo.com:load.load
    #
    #   # The graph "load2". Now we want them stacked on top of each other.
    #       load2.graph_title Loads on top of each other
    #       load2.dummy_field.stack fii=fii.foo.com:load.load fay=fay.foo.com:load.load
    #       load2.dummy_field.draw AREA # We want area instead the default LINE2.
    #       load2.dummy_field.label dummy # This is needed. Silly, really.
    #
    #   # The graph "load3". Now we want them summarised into one field
    #       load3.graph_title Loads summarised
    #       load3.combined_loads.sum fii.foo.com:load.load fay.foo.com:load.load
    #       load3.combined_loads.label Combined loads # Must be set, as this is
    #                                                 # not a dummy field!
    #
    ## ...and on a side note, I want them listen in another order (default is
    ## alphabetically)
    #
    # # Since [foo.com] would be interpreted as a host in the domain "com", we
    # # specify that this is a domain by adding a semicolon.
    # [foo.com;]
    #       node_order Totals fii.foo.com fay.foo.com
    #
    root@system:/etc/apache2/conf-enabled#
    
    Then during this:
    Code:
    cd /etc/apache2/conf-enabled/
    ln -s /etc/munin/apache24.conf munin.conf
    service apache2 restart
    
    I was told that munin.conf already existed so I checked that file:
    lrwxrwxrwx 1 root root 28 Mar 14 09:23 munin.conf -> ../conf-available/munin.conf
    cat munin.conf
    Code:
    # Munin configuration for apache2
    #
    # Apache can serve munin's files that are either statically generated
    # (in a cron job) or generated on demand (via cgi).
    # The relevant munin settings are "html_strategy" and "graph_strategy".
    # The related apache settings can be adjusted at the end of this file.
    #
    # See /usr/share/doc/munin/README.Debian for details.
    
    
    # ***** COMMON SETTINGS FOR ALL STRATEGIES *****
    
    ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
    Alias /munin/static/ /var/cache/munin/www/static/
    
    <Directory /var/cache/munin/www>
    #    Require local
        Options None
    </Directory>
    
    <Directory /usr/lib/munin/cgi>
     #   Require local
        <IfModule mod_fcgid.c>
            SetHandler fcgid-script
        </IfModule>
        <IfModule !mod_fcgid.c>
            SetHandler cgi-script
        </IfModule>
    </Directory>
    
    
    # ***** SETTINGS FOR CGI/CRON STRATEGIES *****
    
    # pick _one_ of the following lines depending on your "html_strategy"
    # html_strategy: cron (default)
    Alias /munin /var/cache/munin/www
    # html_strategy: cgi (requires the apache module "cgid" or "fcgid")
    #ScriptAlias /munin /usr/lib/munin/cgi/munin-cgi-html
    
    /etc/munin/apache24.conf:
    Code:
    # Munin configuration for apache2
    #
    # Apache can serve munin's files that are either statically generated
    # (in a cron job) or generated on demand (via cgi).
    # The relevant munin settings are "html_strategy" and "graph_strategy".
    # The related apache settings can be adjusted at the end of this file.
    #
    # See /usr/share/doc/munin/README.Debian for details.
    
    
    # ***** COMMON SETTINGS FOR ALL STRATEGIES *****
    
    ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
    Alias /munin/static/ /var/cache/munin/www/static/
    
    <Directory /var/cache/munin/www>
    #    Require local
        Options None
    </Directory>
    
    <Directory /usr/lib/munin/cgi>
     #   Require local
        <IfModule mod_fcgid.c>
            SetHandler fcgid-script
        </IfModule>
        <IfModule !mod_fcgid.c>
            SetHandler cgi-script
        </IfModule>
    </Directory>
    
    
    # ***** SETTINGS FOR CGI/CRON STRATEGIES *****
    
    # pick _one_ of the following lines depending on your "html_strategy"
    # html_strategy: cron (default)
    Alias /munin /var/cache/munin/www
    # html_strategy: cgi (requires the apache module "cgid" or "fcgid")
    #ScriptAlias /munin /usr/lib/munin/cgi/munin-cgi-html
    
    After which, service apache2 restart was run.

    Visiting domain.com/munin results in a 403. Does anyone see something I've missed in following the tut?

    Thanks for your time!
     
  2. schwim

    schwim Member HowtoForge Supporter

    I found this in my apache log after trying to visit the munin page:
    Code:
    ::1 - - [14/Mar/2023:16:25:01 -0500] "GET / HTTP/1.1" 200 10985 "-" "Mozilla/5.0 (ISPConfig monitor)"
    127.0.0.1 - - [14/Mar/2023:16:25:02 -0500] "GET /server-status?auto HTTP/1.1" 200 1683 "-" "munin/2.0.67 (libwww-perl/6.52)"
    127.0.0.1 - - [14/Mar/2023:16:25:04 -0500] "GET /server-status?auto HTTP/1.1" 200 1672 "-" "munin/2.0.67 (libwww-perl/6.52)"
    ::1 - - [14/Mar/2023:16:25:07 -0500] "GET / HTTP/1.1" 200 11022 "-" "Munin - http_loadtime"
    127.0.0.1 - - [14/Mar/2023:16:25:08 -0500] "GET /server-status?auto HTTP/1.1" 200 1683 "-" "munin/2.0.67 (libwww-perl/6.52)"
    185.156.72.30 - - [14/Mar/2023:16:28:45 -0500] "\x03" 400 392 "-" "-"
    root@system:/var/log/apache2#
    
    I doubt this is going to be useful in pinpointing my issue but I found this config log in the munin log dir:
    Code:
    root@system:/var/log/munin# cat munin-node-configure.log
    Mar 14 09:24:05 - Starting munin-node-configure --shell
    ln -s '/usr/share/munin/plugins/apache_accesses' '/etc/munin/plugins/apache_accesses'
    ln -s '/usr/share/munin/plugins/apache_processes' '/etc/munin/plugins/apache_processes'
    ln -s '/usr/share/munin/plugins/apache_volume' '/etc/munin/plugins/apache_volume'
    ln -s '/usr/share/munin/plugins/cpu' '/etc/munin/plugins/cpu'
    ln -s '/usr/share/munin/plugins/df' '/etc/munin/plugins/df'
    ln -s '/usr/share/munin/plugins/df_inode' '/etc/munin/plugins/df_inode'
    ln -s '/usr/share/munin/plugins/diskstats' '/etc/munin/plugins/diskstats'
    ln -s '/usr/share/munin/plugins/entropy' '/etc/munin/plugins/entropy'
    ln -s '/usr/share/munin/plugins/fail2ban' '/etc/munin/plugins/fail2ban'
    ln -s '/usr/share/munin/plugins/forks' '/etc/munin/plugins/forks'
    ln -s '/usr/share/munin/plugins/fw_conntrack' '/etc/munin/plugins/fw_conntrack'
    ln -s '/usr/share/munin/plugins/fw_forwarded_local' '/etc/munin/plugins/fw_forwarded_local'
    ln -s '/usr/share/munin/plugins/fw_packets' '/etc/munin/plugins/fw_packets'
    ln -s '/usr/share/munin/plugins/http_loadtime' '/etc/munin/plugins/http_loadtime'
    ln -s '/usr/share/munin/plugins/if_' '/etc/munin/plugins/if_eth0'
    ln -s '/usr/share/munin/plugins/if_err_' '/etc/munin/plugins/if_err_eth0'
    ln -s '/usr/share/munin/plugins/interrupts' '/etc/munin/plugins/interrupts'
    ln -s '/usr/share/munin/plugins/irqstats' '/etc/munin/plugins/irqstats'
    ln -s '/usr/share/munin/plugins/load' '/etc/munin/plugins/load'
    ln -s '/usr/share/munin/plugins/memory' '/etc/munin/plugins/memory'
    ln -s '/usr/share/munin/plugins/munin_stats' '/etc/munin/plugins/munin_stats'
    ln -s '/usr/share/munin/plugins/netstat' '/etc/munin/plugins/netstat'
    ln -s '/usr/share/munin/plugins/ntp_' '/etc/munin/plugins/ntp_12.167.151.1'
    ln -s '/usr/share/munin/plugins/ntp_' '/etc/munin/plugins/ntp_17.253.2.123'
    ln -s '/usr/share/munin/plugins/ntp_' '/etc/munin/plugins/ntp_178.16.23.50'
    ln -s '/usr/share/munin/plugins/ntp_' '/etc/munin/plugins/ntp_44.190.5.123'
    ln -s '/usr/share/munin/plugins/ntp_' '/etc/munin/plugins/ntp_74.6.168.72'
    ln -s '/usr/share/munin/plugins/ntp_kernel_err' '/etc/munin/plugins/ntp_kernel_err'
    ln -s '/usr/share/munin/plugins/ntp_kernel_pll_freq' '/etc/munin/plugins/ntp_kernel_pll_freq'
    ln -s '/usr/share/munin/plugins/ntp_kernel_pll_off' '/etc/munin/plugins/ntp_kernel_pll_off'
    ln -s '/usr/share/munin/plugins/ntp_offset' '/etc/munin/plugins/ntp_offset'
    ln -s '/usr/share/munin/plugins/ntp_states' '/etc/munin/plugins/ntp_states'
    ln -s '/usr/share/munin/plugins/open_files' '/etc/munin/plugins/open_files'
    ln -s '/usr/share/munin/plugins/open_inodes' '/etc/munin/plugins/open_inodes'
    ln -s '/usr/share/munin/plugins/postfix_mailqueue' '/etc/munin/plugins/postfix_mailqueue'
    ln -s '/usr/share/munin/plugins/postfix_mailvolume' '/etc/munin/plugins/postfix_mailvolume'
    ln -s '/usr/share/munin/plugins/proc_pri' '/etc/munin/plugins/proc_pri'
    ln -s '/usr/share/munin/plugins/processes' '/etc/munin/plugins/processes'
    ln -s '/usr/share/munin/plugins/swap' '/etc/munin/plugins/swap'
    ln -s '/usr/share/munin/plugins/threads' '/etc/munin/plugins/threads'
    ln -s '/usr/share/munin/plugins/uptime' '/etc/munin/plugins/uptime'
    ln -s '/usr/share/munin/plugins/users' '/etc/munin/plugins/users'
    ln -s '/usr/share/munin/plugins/vmstat' '/etc/munin/plugins/vmstat'
    The following errors were reported by munin-node-configure --shell
    # The following plugins caused errors:
    # postgres_autovacuum:
    #    Non-zero exit during autoconf (255)
    # postgres_bgwriter:
    #    Non-zero exit during autoconf (255)
    # postgres_cache_:
    #    Non-zero exit during autoconf (255)
    # postgres_checkpoints:
    #    Non-zero exit during autoconf (255)
    # postgres_connections_:
    #    Non-zero exit during autoconf (255)
    # postgres_connections_db:
    #    Non-zero exit during autoconf (255)
    # postgres_locks_:
    #    Non-zero exit during autoconf (255)
    # postgres_oldest_prepared_xact_:
    #    Non-zero exit during autoconf (255)
    # postgres_prepared_xacts_:
    #    Non-zero exit during autoconf (255)
    # postgres_querylength_:
    #    Non-zero exit during autoconf (255)
    # postgres_scans_:
    #    Non-zero exit during autoconf (255)
    # postgres_size_:
    #    Non-zero exit during autoconf (255)
    # postgres_transactions_:
    #    Non-zero exit during autoconf (255)
    # postgres_tuples_:
    #    Non-zero exit during autoconf (255)
    # postgres_users:
    #    Non-zero exit during autoconf (255)
    # postgres_xlog:
    #    Non-zero exit during autoconf (255)
    # sendmail_mailqueue:
    #    Non-zero exit during autoconf (255)
    # sendmail_mailstats:
    #    Non-zero exit during autoconf (255)
    # sendmail_mailtraffic:
    #    Non-zero exit during autoconf (255)
    root@system:/var/log/munin#
    
     
  3. schwim

    schwim Member HowtoForge Supporter

    I found this help doc on 403 with Munin and they stated:
    When I did this, Munin became accessible. I'm not sure if doing this put something else at risk, however.
     
    Th0m and ahrasis like this.
  4. Alex Mamatuik

    Alex Mamatuik Member

    Seems, there is also a password line missing.

    My configuration (but for Centos) - /etc/httpd/conf.d/munin.conf:
    Code:
    <directory /var/www/html/munin>
            AuthUserFile /etc/munin/munin-htpasswd
            AuthName "MUNIN"
            AuthType Basic
            require valid-user
    
            ExpiresActive On
            ExpiresDefault M310
    </directory>
     
    ahrasis likes this.
  5. Alex Mamatuik

    Alex Mamatuik Member

    Almost forgot, dear developers @Th0m , neither Munin, nor Monit is shown on the dashboard of the ISP Config.
    Monit.png
    Monit_access.png
     
  6. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    upload_2023-3-17_9-43-55.png
    I am not sure about munin but monit works fine in mine. There is a minor adjustment need to be made and I already shared the tricks / tips about it in my older posts. Note mine is especially in nginx.
     
    Last edited: Mar 19, 2023
  7. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Alex Mamatuik likes this.
  8. Alex Mamatuik

    Alex Mamatuik Member

  9. Alex Mamatuik

    Alex Mamatuik Member

    @ahrasis Hope, you have seen Mr Till's reply: https://forum.howtoforge.com/threads/ispc-monitor-munin-showing-blank-page.90360/#post-444737.

    However, looking below the white screen of Monit, we can see, that it passes data via iframe a such way:
    Code:
    <iframe src="https://<user_name>:<user_password>@<host>:<port>/" style="overflow:visible;" width="100%" height="1000" frameborder="0"></iframe>
    /usr/local/ispconfig/interface/web/monitor
    $monit_url = $monit_url_parts['scheme'].'://'.$auth_string.$monit_url_parts['host'].(isset($monit_url_parts['port']) ? ':' . $monit_url_parts['port'] : '').(isset($monit_url_parts['path']) ? $monit_url_parts['path'] : '').(isset($monit_url_parts['query']) ? '?' . $monit_url_parts['query'] : '').(isset($monit_url_parts['fragment']) ? '#' . $monit_url_parts['fragment'] : '');

    To overcome problem, for drupal websites i use:
    settings.php
    PHP:
    <?php
      
    // Turn off the X-Frame-Options header entirely, to restore the previous
      // behavior of allowing the site to be embedded in a frame on another site.
      
    $conf['x_frame_options'] = '';
     
      
    header_remove('X-Frame-Options');
    *
    https://www.php.net/manual/en/function.header-remove.php
    https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options


    It means, that to allow Monit either Munin to be rendered inside of ISP Config, X-Frame-Options policy should be changed.

    I have tried to add the mentioned strings to the top of (/usr/local/ispconfig/interface/web/monitor) show_data.php, show_monit.php and to '../../lib/config.inc.php' & '../../lib/app.inc.php', but vainly.

    Believe, there are some guys here, loving tinker all the stuff.
     
    schwim likes this.
  10. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Err... I didn't modify anything to the code for monit to works on my server ISPConfig UI.
     
  11. Alex Mamatuik

    Alex Mamatuik Member

    Which browser do you use?
     
  12. Alex Mamatuik

    Alex Mamatuik Member

    and how do you enable 4-tier servers: fahim.server.sch.my?
     
  13. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Firefox 111 (64 bit).
    As usual, same as any subdomain, only it is on the fourth tier, instead of the third.
     
  14. Alex Mamatuik

    Alex Mamatuik Member

    - really strange, hard to comprehend, why your monit works.

    - sorry, this is out of my abilities. have no idea of how to achieve.
     
  15. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    My guess is because of "use address 0.0.0.0" and "/usr/local/ispconfig/interface/ssl/ispserver.pem" which I already mentioned several times via several posts but I am not sure why others can't make it work like me. Strange indeed.
    Code:
    set ssl options {
        version: auto
        verify: enable
        selfsigned: reject
    }
    set httpd port 8082 and
        use address 0.0.0.0
        SSL ENABLE
        PEMFILE /usr/local/ispconfig/interface/ssl/ispserver.pem
        allow user:'password'
     
  16. Alex Mamatuik

    Alex Mamatuik Member

    Initially, my /etc/monitrc file was:
    Code:
    #set ssl options {
    #    version: auto
    #    verify: enable
    #    selfsigned: reject
    #}
    
    set httpd port 2812 and
       use address 0.0.0.0
       SSL ENABLE
    PEMFILE /var/certs/monit.pem  
    allow <user>:<pass>
    then i have changed for yours recommended setting:
    Code:
    set ssl options {
        version: auto
        verify: enable
       selfsigned: reject
    }
    
    set httpd port 2812 and
       use address 0.0.0.0
       SSL ENABLE
    PEMFILE /usr/local/ispconfig/interface/ssl/ispserver.pem
      allow <user>:<pass>
    and it has not brought any result.

    My question: which port your ispconfig is listened on?

    And your option of selfsigned: reject looks suspicious.
    How do you sign https://fahim.server.sch.my:8082?
     
  17. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    My port is as stated which is 8082 and this must be opened on your server side (nat router etc) and via ISPConfig opened firewall ports. For monit header try 'pass' with single quote. Monit user and this password must be add in server config too (obviously without the quote).

    I need this so that I know my ISPConfig was not using a self-signed certs. Unless you are using self-signed certs, I think you can take that off if you want.

    I'll sign as normal i.e. monit user name and password to open that page directly.
     
  18. Alex Mamatuik

    Alex Mamatuik Member

    Definitely, i use it without any quotes or other symbols. Through address bar of browser it goes smoothly.

    Am i clear, that your monit port is same as ISP Config port?

    Anyway, i have tried your configuration also.

    Will hope, that someone point out a proper path to add:
    PHP:
    <?php
      $conf
    ['x_frame_options'] = '';
       
    header_remove('X-Frame-Options');
     
  19. schwim

    schwim Member HowtoForge Supporter

    Would anyone happen to know the path/dir where I could add or edit .htaccess file for munin? I'm hoping to play with the csp directives a bit.
     
  20. till

    till Super Moderator Staff Member ISPConfig Developer

    Hmm, isn't munin running its own http server written in perl or python or so? I'm pretty sure it's not running under apache.
     

Share This Page