Problems with groups/grpconv

Last night I decided to migrate all my users/sites to my ISPConfig server.
All appeared to be going well....

After adding maybe 10 sites and over 100 email accounts, I noticed a strange problem. Files that were usually owned by root root where now showing root 1
I looked at the /etc/group file and to my horror all my groups had disappeared and the only lines left in there where the last 5 groups added by ISPConfig.
I am uncertain why this happened but I am guessing it may be because my usernames had hyphens in them (-). Can anyone confirm that this may be true or is true ?

On further investigation, I found that each time I added/changed users in ISPConfig, a grpconv command was executed which ran at 100% cpu and 100% mem, and eventually was killed with an Out Of Memory message in the messages file.

After googling for a while, I found that grpconv can go into an endless loop until killed with an OOM.
I do not know whether the original corruption of /etc/group and /etc/gshadow was a result of the grpconv hanging or whether the grpconv hanging was a result of the corruption.
After manually fixing the group and gshadow files, I ran grpck which found some duplicate entries and some other small problems.
grpconv now executes fast and without problems.

One thing I do notice however, is that groups created by ISPConfig, for instance:

web1:x:10001:admispconfig,adminuser

When web1 has been deleted, this group remains and is not deleted. Is this normal or have I got another problem here ? If it IS normal, would it not be better for the group to be deleted also, once the web has been removed from the recycle bin ?

This brings me onto another issue..... after playing around with ISPConfig, and removing it and reinstalling it etc, ISPConfig seems to have no regard for already assigned UIDs. It seems to always start with 10001 regardless of whether this is already in use or not. .This causes immediate problems with /etc/passwd and /etc/group (and may have been the initial cause of my problems).
ISPConfig needs to check that UIDs are not in use before it decides to use them itself !!
My advice to anyone would be to make a backup of /etc/passwd /etc/shadow /etc/group and /etc/gshadow before installing ISPConfig just in case there are problems later, in fact after this experience I will be making backups of these 4 files on every server that I have.

Any comments Falko/Till ? My server seems to be running spot on now so I am happy again, but it's been a hell of a 24hrs

Edit:
If I add a new group manually for something, for instance:

groupadd testgroup

this group is assinged the next ID, which will be 1 higher than the last ispconfig site.
If you then add a new site in ISPConfig, it doesn't check for the next id, it just knows the last one it created itself so uses the same one that testgroup took.

 

Never heard of anything like that.

It should be deleted after you've deleted the web site also from the recycle bin.

Therefore you can set the starting user id and group id under Management -> Server -> Settings -> Other. You have to go sure yourself that these ids do not interfere with others you use on the system.

 

Thanks for the reply

OK. Can you tell me where in the ISPConfig code this group deletion takes place so that I may find out why it is not working on my system ?

Setting the starting user id and group id is all good and well, but if i do a manual groupadd for whatever reason, it will ALWAYS use the last used id + 1 and therefore the next group ISPConfig adds will be the same.
So basically, if this is not changed/fixed, the command groupadd can never be used by anyone or any application without specifying an id
Would it not make more sense for ISPConfig to check first ?

 

It's the function web_delete() in /root/ispconfig/scripts/lib/config.lib.php. But you can check the logfile first: /home/admispconfig/ispconfig/ispconfig.log.

See man groupadd. With groupadd -g gid group You can set the group id.

 

Ok the logfile looks fine, but it doesn't show creating/deleting users/groups.

im finding hard to understand the web_delete() function too, I can see where it backs up the website then removes it, and then backs up mysql and then deletes that, but I can't find exactly where it removes the group.
Can you help me further please ?

 

I'm sorry, it's not web_delete(), it's in the function user_delete().

 

Hehe OK

Well I am looking at this:

Code:

 $mod->system->delgroup("web".$web_doc_id);
    //$mod->log->caselog("groupdel web$web_doc_id &> /dev/null", $this->FILE, __LINE__);

what is the commented out line ? If I uncomment, will this log the action in the log file ?

Ughhh my head is hurting now, is there any quick way that I can log all actions by the function wf() in ispconfig_file_lib.php using your logging class ?

This may help me find why groups are not being deleted

 

You could use

PHP:

$mod->log->caselog("groupdel web$web_doc_id &> /dev/null", $this->FILE, __LINE__);

and comment out the other line.
That should log the action into the log file.
There's no quick way for logging every action. You'd have to add the code for logging to every single action manually...

 

OK Cool
Thanks Falko, will give it a try

 

I have experienced the group deletion as well. All except the groups of the 3 last users I edited were deleted.

Thankfully I keep backups of /etc/group, /etc/gshadow etc so restoration was not a big problem. But this issue is scary.

 

Did you have exactly the same problems as linuxfast (grpconv running at 100%, etc.)?
What were the gids of the deleted groups?

 

Well the groups deleted were the system groups.

groups from ids 0 - 511 were deleted.

In addition the VHosts_ispconfig.conf was also deleted.

I am using ISPConfig 2.1.1 . I would upgrade to 2.1.2 but I havent completed running it in our preproduction environment.

 

Then you definitely have a problem. Any warnings/errors in /home/admispconfig/ispconfig/ispconfig.log?

 

I got quite a few of the following:

Code:

WARNING: could not mv -f /etc/httpd/conf/vhosts/Vhosts_ispconfig.conf /etc/httpd/conf/vhosts/Vhosts_ispconfig.conf_22-12-05_12-00-41

 

What's the output of

Code:

ls -la /etc/httpd/conf/vhosts

?
Also post the output of

Code:

df -h

 

Hello Falko,

Sorry I was not able to get back to this thread sooner.

Here is the info on its current state

Code:

ls -la /etc/httpd/conf/vhosts:

$ ls -la /etc/httpd/conf/vhosts/
total 100
drwxr-xr-x    2 root     root         4096 Dec 22 12:29 .
drwxr-xr-x    8 root     root         4096 Dec  1 23:01 ..
-rw-r--r--    1 root     root        15338 Jan  3 09:58 Vhosts_ispconfig.conf
-rw-r--r--    1 root     root        15338 Jan  3 09:58 Vhosts_ispconfig.conf~
-rw-r--r--    1 root     root        12287 Dec 22 11:58 Vhosts_ispconfig.conf_22-12-05_11-58-56
-rw-r--r--    1 root     root        12386 Dec 22 12:00 Vhosts_ispconfig.conf_22-12-05_12-00-20
-rw-r--r--    1 root     root        12349 Dec 22 12:00 Vhosts_ispconfig.conf_22-12-05_12-00-40
-rw-r--r--    1 root     root        12384 Dec 22 12:01 Vhosts_ispconfig.conf_22-12-05_12-01-14

Code:

df -h:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda6             2.0G  538M  1.3G  29% /
/dev/sda1             145M   15M  123M  11% /boot
/dev/sdb1              74G   36M   70G   1% /export/data0
/dev/sda5             5.8G  756M  4.8G  14% /home
none                  503M     0  503M   0% /dev/shm
/dev/sda8             487M  8.2M  453M   2% /tmp
/dev/sda9              12G  3.0G  7.7G  29% /usr
/dev/sda3             7.7G  322M  7.0G   5% /var
/dev/sda2              46G  704M   43G   2% /sites

ISPConfig is working ok now. But earlier when it had removed those groups, it got a bit scary. Thankfully I had backups so I was able to get it back.

 

grpconv

Hi, after days of searching the error wer an server became totally unusable and plenty reboots I used top and saw grpconv command was executed which ran at 100% cpu and 100% mem, googled for it and voila back at ispc Howtoforge.

How did this happen on ubuntu 7.04 and ISPC 2.2.12

I found after deleting some webs that the user group was duplicated! causing this phenomena.
Hopefully all is fine now.

Some testing of these files by ISPC ? perhaps an integrity test after deletions.

 

Doubt

After you Post Till, I doubted myself because I have manipulated the password files 2 months ago as I transferred this ISPC from Mandriva to debian and the UID:GID's differ at installation. Actually it will change depend on the sequence of installing applications.: cool:
However every time changes was made by the admin on the users and domains the group entry "users" seem to be duplicated!
On the 4th I cleaned the group files and today the server crashed again and had to re-do it.

Code:

root@alpha:~# grpck
duplicate shadow group entry
delete line 'users:*::tang..<Snipped! more than 100 account names>..raining'? y
grpck: the files have been updated

Is there an ISPC process that manage "users" group? could it be DB data errors and or corruption. The test and repair procedures in ISPC show no DB errors.