No module named _weakrefset

Discussion in 'General' started by fyberoptik, Sep 15, 2022.

  1. fyberoptik

    fyberoptik New Member

    I'm having a very odd issue with my ispconfig setup. It is used as a student webhost to support modules each academic year, at the end of the year I archive and import ~500 user accounts (1200 in total).

    I have some scripts which use the PHP API scripts to add usernames, websites, database and these have generally worked fine up until now. When I add users now, it will completely break httpd and no sites will load until I manually remove the sym-link previously added /etc/httpd/conf/site-enabled. When the newly added username/website is added, I get the following in Apache's error log:

    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import UserDict
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import _abcoll
    import _abcoll
    import _abcoll
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    import _abcoll
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    import _abcoll
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
    from abc import ABCMeta, abstractmethod
    from abc import ABCMeta, abstractmethod
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    from abc import ABCMeta, abstractmethod
    from _weakrefset import WeakSet
    from _weakrefset import WeakSet
    from _weakrefset import WeakSet
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    ImportErrorImportErrorImportError: : : No module named _weakrefsetNo module named _weakrefsetfrom _weakrefset import WeakSet
    No module named _weakrefset

    ImportError: No module named _weakrefset
    from abc import ABCMeta, abstractmethod
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
    ImportError: No module named _weakrefset
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import _abcoll
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    from _weakrefset import WeakSet
    ImportError: No module named _weakrefset
    Traceback (most recent call last):
    File "/usr/lib64/python2.7/site.py", line 62, in <module>
    import os
    File "/usr/lib64/python2.7/os.py", line 398, in <module>
    import UserDict
    File "/usr/lib64/python2.7/UserDict.py", line 83, in <module>
    import _abcoll
    File "/usr/lib64/python2.7/_abcoll.py", line 11, in <module>
    from abc import ABCMeta, abstractmethod
    File "/usr/lib64/python2.7/abc.py", line 8, in <module>
    from _weakrefset import WeakSet​

    I can confirm the weakrefset is present and I can switch user to ispconfig, run python and import the module with no issue.

    I have tried checking over the userlist for hidden characters incase something wasn't getting sanitised properly but struggling to find a root cause. It's very specific websites being enabled which seems to cause the problem.

    I have also tried enabling the debug mode, this is the process of enabling of enabling the site.
    15.09.2022-11:51 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    15.09.2022-11:51 - DEBUG [server:177] - Found 1 changes, starting update process.
    15.09.2022-11:51 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    15.09.2022-11:51 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client3242/web3210' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client3242/web3210' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client3242/web3210'|awk 'END{print $2,$NF}' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: xfs_quota -x -c 'limit -u bsoft=500m bhard=501m web3210' '/' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: xfs_quota -x -c 'timer -bir -i 604800' '/' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client3242/web3210' - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/php-fcgi-scripts/web3210/.php-fcgi-starter' - return code: 0
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1602] - Creating fastcgi starter script: /var/www/php-fcgi-scripts/web3210/.php-fcgi-starter
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/php-fcgi-scripts/web3210/.php-fcgi-starter' - return code: 0
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1828] - Enable SSL for: cofh4.sci-project.ac.uk
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1875] - Writing the vhost file: /etc/httpd/conf/sites-available/cofh4.sci-project.ac.uk.vhost
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1910] - Creating symlink: /etc/httpd/conf/sites-enabled/100-cofh4.sci-project.ac.uk.vhost->/etc/httpd/conf/sites-available/cofh4.sci-project.ac.uk.vhost
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1993] - Apache status is: running
    15.09.2022-11:51 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'.
    15.09.2022-11:51 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
    15.09.2022-11:51 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'httpd' 2>&1 - return code: 0
    15.09.2022-11:51 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart httpd.service
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:1996] - Apache restart return value is: 0
    15.09.2022-11:51 - DEBUG [apache2 plugin.inc:2007] - Apache online status after restart is: running
    15.09.2022-11:51 - DEBUG [modules.inc:240] - Processed datalog_id 38337
    15.09.2022-11:51 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    15.09.2022-11:52 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    15.09.2022-11:52 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    15.09.2022-11:53 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    15.09.2022-11:53 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Thanks in advance
     
    Last edited: Oct 4, 2022
  2. fyberoptik

    fyberoptik New Member

    Just a little worried about this as there are some classes coming up that will be using this system. Any help would be fantastic.
     
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    So basically the sites added via API contain some wrong values or paths. Create a site within ISPConfig GUI similar to the ones you create via API, then compare in dbispconfig database the API and GUI created site in web_domain table to find out where they differ to adjust $params array in your API script.
     
  4. fyberoptik

    fyberoptik New Member

    Thanks for the suggestion, I have just taken a look at the web_domain table. The only difference I could see between a working site and broken site was minimal, I could only see a small difference between the SSL cert. Never the less, I experimented by copying the vhost file generated by ispconfig. I made a version which was just port 80 and one that was just port 443 so see if that made a difference. When that didn't work I tried to cutting out some of the PHP config and just cutting back to a basic static site. ISPconfig still stops loading. I also tried to get ISPconfig to create a very basic site, no SSL, PHP or dynamic content. No joy I'm afraid.

    The input file that works with the API scripts is really basic, it's a first name and a username... I used the cat command to print out hidden characters to try and find out whether it was passing something nasty, but couldn't really see anything. It's almost like it came to a certain number of users and config generated after that point fails. The API scripts create one specific type of user and all the config is the same for each user. It managed to import a good 200 usernames before it started to complain and fail.

    But many thanks again, I'll try anything to get it going.
     
  5. fyberoptik

    fyberoptik New Member

    I have just been through the process of adding a more up to date version of PHP and amending the api scripts to configure new websites with PHP 8.1 and fast-cgi interfacing. I removed the recent users from a few weeks back and started importing again. Everything was smooth sailing adding roughly ~270 websites before I began to have problems. I feel like I'm hitting a limit, I get to a certain number of sites and can add anymore. The httpd service reports that it has restarted with no issues, however ISPconfig is just dead repeating the same message around: ImportError: No module named _weakrefset.

    I have noticed that php-fpm isn't available from newer website config pages, this is only present for older sites that had used it in the past. New sites don't see this even though this is configured in the additional PHP versions.

    Worth noting, it doesn't really matter what creates a site at this point. A vanilla static page brings the whole system down.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    I just had a look at the ISPConfig source code and it does not contain the word _weakrefset anywhere. So this must be something that you either added in your code or that you are importing. ISPConfig has no limit in the range of 270 websites and that's not a large number of sites for an ISPConfig system anyway, so it might be that your import script just repeatedly imports some wrong data at that number of sites.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    And just to add that, ISPConfig is not using any python code, so no matter if python is installed or not, it will not affect ISPConfig GUI. ISPConfig is completely written in PHP.
     
  8. fyberoptik

    fyberoptik New Member

    Hi Till

    Thanks for confirming that, because the error wasn't pointing out a specific file I was struggling to see where the problem was. I (wrongly) assumed something in ISPconfig was using it for building the site config files or something as it seemed to be happening while adding websites.

    I just did a search on all the client website files and found 'weakrefset' in an Academic's website files.

    Sorry for bugging you, using grep to find weakrefset has done the trick and pointed out the location of where it was getting used. I'm still curious as to why it only started becoming a problem after a certain number of websites were added to the system, but the larger stress of adding students on ISPconfig.

    Thanks for the help and pointer.
    J
     
  9. fyberoptik

    fyberoptik New Member

    It turns out the python errors where a complete red herring, I had to remove all the python configuration from the webserver to show the actual issue which was down to a file ulimit httpd/apache was breaching. I took some steps from the following guide and this has appeared to solve the problem for now.

    https://support.cpanel.net/hc/en-us...-of-open-files-allowed-for-Apache-on-CentOS-7

    Thanks again for the help.
     
    till likes this.

Share This Page