Multiplexing webservers with ISPConfig statistics

Discussion in 'Feature Requests' started by Serg, Sep 16, 2007.

  1. Serg

    Serg New Member

    Hello, everybody!
    I'm using ISPConfig some weeks and can say, that is nice software, congradulation to developement team.
    But there is a wider client wishes, and I must to solve some problems.
    Preface.
    There is standard ISPConfig installation.
    Also there is port-based hosting due to plone/zope service on rank of ports 2XXX-2X10
    Also there is another independend webserver with old feautures like MYSQL 4.0 and PHP4 (XAMPP powered) on port 85.
    And I can(and make) multiplexing to all servers via squid proxy.
    Main Apache2 now listen port 82.
    Control ISPConfig Apache1.3 listen port 81.
    Squid listen port 80 and work as transparent proxy according to simple acl rules.
    All works fine, but statistics still not.
    I reformat squid log outpot as ISPConfig log format.
    There is one difference(example)
    ISPConfig /var/log/httpd/ispconfig_access_2007_09_10 :
    ...GET /favicon.ico HTTP/...
    Squid(I point squid log to same place and make needed symlink to it) :
    ...GET http://test.domain.net/favicon.ico HTTP/...
    Then I point main apache2 log file to /dev/null
    Squid log utilization is made by my script in cron at 00.01 am, it unlink ispconfig_access_log from old log, delete two-days-old log, creates new day log, put symlink to it and then reconfigure(force reload) squid.
    But at morning I see two separate directory in /var/www with same name.(test.mydomain.com was created by ISPConfig panel)
    One is test.mydomain.net - symlink, as known to /var/www/webuserID with old log, generating when apache2 stored log directly and belonging to virtual host test.mydomain.net
    Second is real direcrory test.mydomain.net with new(not empty,right!) log, belonging to test site and generating from squid log.
    First directory is owned by testclient:webuserID
    Second is owned by root:root
    Directory creation time is 00:30
    Webalizer page operates with the old statistic, but all files in stat/ directory was rewritten in 04:00 am last day.
    1.What happen? Why there are two separate dirs with same name(but one is symlink)?
    And how I can tell script to write site log to correct dir/file(eq symlink)
    2.Which process is starter of scripts/shell/logs.php ? In 00.30 as I understand?
    It seems, it script run automatically by ... program
    3. Can I run(for test purpose, of course, not on production host) logs.php script to see changes with logfiles?
    testhost#php logs.php (or any parameters required?)
    Big thnks for reply
     
    Last edited: Sep 16, 2007
  2. falko

    falko Super Moderator Howtoforge Staff

    It is run by cron. See
    Code:
    crontab -l
    The correct command is
    Code:
    /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php
     
  3. Serg

    Serg New Member

    Thank you, Falko

    Yes! When I run this script with the analyze some variables (thanks to print_r() ) I catch an error - finishing SPACE in filenames. This was my (hidden :cool: ) bug in Logformat string. Shame to me!
    And now it fixed and all OK.
    Is squid-with-ISPconfig integration guide necessary for anyone?

    Great! We have now nice statistics of port-based hosting and (even!) of co-location traffic to another hosts with their web(ftp, etc)-servers. All clients can to operate statistics with ISPConfig client's web face and we can make new hosting plans and invoices.
    In this case there is a idea - to prepare a "tiny" client's interface or make a possibility to construct client ifaces by admin, depend of client's needs (eq with/without WEB-FTP, with/without database/co-domain/mail/user management, less menus items and so on)
    Any suggestions?
     
  4. the_spy

    the_spy New Member

    can you please explain how is working your installation of squid ? I don't understand how it works in squid and which configuration is needed to distribute requests to one port service or an other, and it could interest me if you can put a small tutorial here :) why not with ispconfig statistics also... :D Thanks
     
  5. Serg

    Serg New Member

    Well. This is not tutorial, but common issue, because I don't have such time to write full guide now(and in future week too)

    We assume, that our apache will listen on port 82, as one of any port-based services and we use Debian
    There are steps to prepare hosting service to work in complex:
    1. Stop main Apache.
    2. Install squid and squidguard by apt-get. (We don't need to use base squid from Debian distro, because it not responces all logformat parameters, discribed in squid tutorial :confused: , but we need squidguard)
    3. Install squid from source to /usr/local. You can install squidguard from source too and skip step 2 at all.
    4. Configure squid. Next code not whole, but touch only engine with ISPConfig
    Code:
    http_port IP.IP.IP.IP:80 transparent
    tcp_outgoing_address IP.IP.IP.IP
    emulate_httpd_log	off
    logformat squid %{Host}>h||||%st||||%>a %un %ul [%tl] "%rm %ru HTTP/%rv" %Hs %st "%{Referer}>h" "%{User-Agent}>h"
    access_log /var/log/httpd/ispconfig_access_log squid
    
    acl all src 0.0.0.0/0.0.0.0
    acl to_localhost dst 127.0.0.0/8
    acl our_host dst IP.IP.IP.IP
    
    http_access deny to_localhost
    http_access allow our_host
    http_access deny all
    
    follow_x_forwarded_for allow all
    log_access allow our_host
    log_access deny all
    logfile_rotate 0
    url_rewrite_program /usr/bin/squidGuard -c /etc/squid/squidGuard.conf
    url_rewrite_children 5
    url_rewrite_host_header on
    # via on
    via off #this inject warning in cache log, but works
    cache_effective_user proxy
    cache_effective_group proxy
    digest_generation off
    icp_port 0
    htcp_port 0
    forwarded_for on
    
    You can play with some options such as http_port (transparent, accel, vhost, vport) And we use external redirect program but it is possible use cache_peer directive and point to our apache server as parent cache on port 82 with some successful options, it works too.
    5. Configure squidguard
    Code:
    rew dmz {
    	s@://control.youdomain.com@://youdomain.com:81@i
    	s@://IP.IP.IP.IP@://sharedip.youdomain.com:82@i
    	s@://youdomain.com@://youdomain.com:82@i
    	s@://www.youdomain.com@://youdomain.com:82@i
    #and something else including all virtual hosts, webmail and redirects to another services
    	#Last string. If host exist in DNS but has no service on port. This prevent loop
    	s@://.*\.youdomain.com@://no-in-serve.youdomain.com:82@i
    }
    dest local {
    }
    acl {
    	default {
    		pass	 local any
    		rewrite	 dmz
    	}
    }
    
    6. chown -R /var/log/httpd to root:proxy
    chmod /var/log/httpd to 775
    make a file /var/log/httpd/ispconfig_access_log_2007_M_D
    where M and D - current month and day
    make a symlink to it with ispconfig_access_log
    7. start and testing squid.
    8. Patch /root(orwhereisyouinstallISP)/ispconfig/scripts/lib/config.lib.php
    Near string No 1264 write port :82 instead :80
    9. Patch apache's conf
    ports.conf
    Listen:82

    apache2.conf
    Comment (#)LogFormat and CustomLog
    Put new directives:
    SetEnv dontlog 1
    LogFormat "%v %b" combined
    CustomLog /var/log/something../access.log combined env=!dontlog
    10. Start apache. Go to URL youdomain.com:81 (to ISPConfig) and create new hosts with DNS records: control.youdomain.com shareip.youdomain.com and no-in-serve.youdomain.com, put to web directory on last two hosts nice pages with text.
    All you hosts in Vhost.conf will updated to port 82 automatilally.
    11. Now you can point you browser to control.youdomain.com and see login page of ISPConfig.

    Thats not at all. You must take care to rotate logs and to remake correct symlink and permissions in log/httpd, after logs rotation you must reload squid. This is may be done by our frend Cron Oliveira.
    Also we must update check_services.php about 82 port.

    Now all works, but there are some manual operation when you create or delete site - you must put(or delete) record in squidguard.conf
    Work to make this automatically is in progress.
    Also squid make different records in logfile about host which has aliases
    Another words, requests to domain.com and www.domain.com has separate records. This problem is half-solved(alpha version :D ) and I not describe jet.
     

Share This Page