Hello the Website Harddisk Quota is wrong in the main page of my ispconfig 3.0.5.3 installation. Can i use the Tools-->Rsync-->Resync Websites tool to fix it ? Thanks
solved Hello the Website Harddisk Quota was not wrong ispconfig uses the command repquota -au to display the quotas it displays the space used by every user for example web1 user you can run it as repquota -aus and check if the results are same with the ispconfig results you can also run the following command to get the same results find . -user web2 -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }' in my case i use a rsync script with keep file permissions option to backup websites in /var/www/backup/ folder so repquota counts also these files in results. thats why i thought the Website Harddisk Quota was wrong thanks
Very, very similar case here except I can't figure it out. ISPCFG3 reports: foodandchatter.co.za 4500.97 MB 5500 MB 5501 MB manually checking: root@alfred:/var/www/clients/client2# du -cksh ../client2/* 0 ../client2/foodandchatter.co.za 2.8G ../client2/web8 2.8G total Using above tools: root@alfred:/var/www/clients/client2# repquota -aus |grep web8 web8 -- 4501M 5500M 5501M 55418 0 0 root@alfred:/var/www/clients/client2# find . -user web8 -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }' Total used: 2817928 Where is the difference in used space?
Outside of the /var/www/clients/client2 folder (e.g. in /tmp, /var/tmp or somewhere else). What repquota and ispconfig show you is the usage of the linux user of the website. What you tested is the sum of file sizes in a specific directory.
OK, I get that, I was just wondering why the find command didn't find everything then realized I didn't run it from the root and tried again, getting ambiguous results, can you shed some light? I'd like to find and check where this space is being wasted. Why does this show 2 different results? Code: root@alfred:/# find . -user web8 -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }' find: `./sys/fs/cgroup/memory/lxc/100/memory.pressure_level': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/memory.force_empty': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/cgroup.event_control': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.allow': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.deny': Permission denied find: `./sys/fs/cgroup/blkio/lxc/100/blkio.reset_stats': Permission denied find: `./proc/29770/task/29770/fd/6': No such file or directory find: `./proc/29770/task/29770/fdinfo/6': No such file or directory find: `./proc/29770/fd/6': No such file or directory find: `./proc/29770/fdinfo/6': No such file or directory Total used: 4589548 root@alfred:/# Total used: 2817928
It shouldn't, and doesn't in my testing. As the second one is displayed after your shell prompt, I'm guessing you had one such command running in the background, and the other in the foreground, and you see the output from both.
I agree, I ran it again twice it it now shows up properly as expected. Code: root@alfred:/# find . -user web8 -type f -exec du -k {} \;|awk '{ s = s+$1 } END { print "Total used: ",s }' find: `./sys/fs/cgroup/memory/lxc/100/memory.pressure_level': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/memory.force_empty': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/cgroup.event_control': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.allow': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.deny': Permission denied find: `./sys/fs/cgroup/blkio/lxc/100/blkio.reset_stats': Permission denied find: `./proc/22969/task/22969/fd/6': No such file or directory find: `./proc/22969/task/22969/fdinfo/6': No such file or directory find: `./proc/22969/fd/6': No such file or directory find: `./proc/22969/fdinfo/6': No such file or directory Total used: 3580908 So there seems to be 3.58GB reported by find but only 1.8GB inside /var/www/clients/client2/web8 Code: root@alfred:/# du -cksh /var/www/clients/client2/* 0 /var/www/clients/client2/foodandchatter.co.za 1.8G /var/www/clients/client2/web8 1.8G total So I'm looking for the rest: Code: root@alfred:/# find . -user web8 -type f -exec du -k {} \; | grep -v web8 find: `./sys/fs/cgroup/memory/lxc/100/memory.pressure_level': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/memory.force_empty': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/cgroup.event_control': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.allow': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.deny': Permission denied find: `./sys/fs/cgroup/blkio/lxc/100/blkio.reset_stats': Permission denied find: `./proc/2364/task/2364/fd/6': No such file or directory find: `./proc/2364/task/2364/fdinfo/6': No such file or directory find: `./proc/2364/fd/6': No such file or directory find: `./proc/2364/fdinfo/6': No such file or directory find: `./proc/20762': No such file or directory root@alfred:/# Does this show there is not a single file owned by web8 outside the folder web8 ??? Sorry for insisting, this is not really important but I can't stand a mystery :-(
It shows there are no files owned by user web8 which do not have 'web8' in the pathname. Try: Code: find / -user web8 -type f -path /var/www/clients/client2 -prune -o -print0 | xargs -0 du -k
Isn't that exactly what I'm trying to establish? The first report says 3.6GB owned by web8, my second test says 1.8GB are inside the web8 folder so I did the last test to show me all files owned by web8 OUTSIDE of the web8 folder and the result was none so where are the other 1.8GB !? Am I having a blonde moment? Your command did show a lot of stuff, files all over the place not owned by web8 but thanks.
Sorry, it has to have explicit argument grouping or the precedence is wrong: Code: find / -user web8 -type f '(' -path /var/www/clients/client2 -prune -o -print0 ')' | xargs -0 du -k
OK that works but I don't get your point: everything it spits out is inside /var/www/clients/client2/web8 and if I add a "| grep -v web8" to your command it shows nothing again. So this means all 3.6GB of files owned by web8 are inside /var/www/clients/client2/web8 but "du" only reports that folder to be 1.8GB Have another look at my above findings please.
That should prune, ie. leave out, results matching /var/www/clients/client2; testing here seems to do what you want. Try your first command but change the grep: Code: find . -user web8 -type f -exec du -k {} \; | grep -v var/www/clients/client2
As expected, no files found. So my statement from above still stands, why is one command reporting 3.6GB and du only finds 1.8GB? Code: root@alfred:/# find . -user web8 -type f -exec du -k {} \; | grep -v "var/www/clients/client2" find: `./sys/fs/cgroup/memory/lxc/100/memory.pressure_level': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/memory.force_empty': Permission denied find: `./sys/fs/cgroup/memory/lxc/100/cgroup.event_control': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.allow': Permission denied find: `./sys/fs/cgroup/devices/lxc/100/devices.deny': Permission denied find: `./sys/fs/cgroup/blkio/lxc/100/blkio.reset_stats': Permission denied 0 ./proc/14341/task/14341/fdinfo/0 0 ./proc/14341/task/14341/fdinfo/1 0 ./proc/14341/task/14341/fdinfo/2 0 ./proc/14341/task/14341/fdinfo/3 0 ./proc/14341/task/14341/fdinfo/4 0 ./proc/14341/task/14341/fdinfo/5 0 ./proc/14341/task/14341/environ 0 ./proc/14341/task/14341/auxv 0 ./proc/14341/task/14341/status 0 ./proc/14341/task/14341/personality 0 ./proc/14341/task/14341/limits 0 ./proc/14341/task/14341/sched 0 ./proc/14341/task/14341/comm 0 ./proc/14341/task/14341/syscall 0 ./proc/14341/task/14341/cmdline 0 ./proc/14341/task/14341/stat 0 ./proc/14341/task/14341/statm 0 ./proc/14341/task/14341/maps 0 ./proc/14341/task/14341/children 0 ./proc/14341/task/14341/numa_maps 0 ./proc/14341/task/14341/mem 0 ./proc/14341/task/14341/mounts 0 ./proc/14341/task/14341/mountinfo 0 ./proc/14341/task/14341/clear_refs 0 ./proc/14341/task/14341/smaps 0 ./proc/14341/task/14341/pagemap 0 ./proc/14341/task/14341/attr/current 0 ./proc/14341/task/14341/attr/prev 0 ./proc/14341/task/14341/attr/exec 0 ./proc/14341/task/14341/attr/fscreate 0 ./proc/14341/task/14341/attr/keycreate 0 ./proc/14341/task/14341/attr/sockcreate 0 ./proc/14341/task/14341/wchan 0 ./proc/14341/task/14341/stack 0 ./proc/14341/task/14341/schedstat 0 ./proc/14341/task/14341/cpuset 0 ./proc/14341/task/14341/cgroup 0 ./proc/14341/task/14341/oom_score 0 ./proc/14341/task/14341/oom_adj 0 ./proc/14341/task/14341/oom_score_adj 0 ./proc/14341/task/14341/loginuid 0 ./proc/14341/task/14341/sessionid 0 ./proc/14341/task/14341/io 0 ./proc/14341/task/14341/uid_map 0 ./proc/14341/task/14341/gid_map 0 ./proc/14341/task/14341/projid_map 0 ./proc/14341/task/14341/setgroups 0 ./proc/14341/fdinfo/0 0 ./proc/14341/fdinfo/1 0 ./proc/14341/fdinfo/2 0 ./proc/14341/fdinfo/3 0 ./proc/14341/fdinfo/4 0 ./proc/14341/fdinfo/5 0 ./proc/14341/environ 0 ./proc/14341/auxv 0 ./proc/14341/status 0 ./proc/14341/personality 0 ./proc/14341/limits 0 ./proc/14341/sched 0 ./proc/14341/autogroup 0 ./proc/14341/comm 0 ./proc/14341/syscall 0 ./proc/14341/cmdline 0 ./proc/14341/stat 0 ./proc/14341/statm 0 ./proc/14341/maps 0 ./proc/14341/numa_maps 0 ./proc/14341/mem 0 ./proc/14341/mounts 0 ./proc/14341/mountinfo 0 ./proc/14341/mountstats 0 ./proc/14341/clear_refs 0 ./proc/14341/smaps 0 ./proc/14341/pagemap 0 ./proc/14341/attr/current 0 ./proc/14341/attr/prev 0 ./proc/14341/attr/exec 0 ./proc/14341/attr/fscreate 0 ./proc/14341/attr/keycreate 0 ./proc/14341/attr/sockcreate 0 ./proc/14341/wchan 0 ./proc/14341/stack 0 ./proc/14341/schedstat 0 ./proc/14341/cpuset 0 ./proc/14341/cgroup 0 ./proc/14341/oom_score 0 ./proc/14341/oom_adj 0 ./proc/14341/oom_score_adj 0 ./proc/14341/loginuid 0 ./proc/14341/sessionid 0 ./proc/14341/coredump_filter 0 ./proc/14341/io 0 ./proc/14341/uid_map 0 ./proc/14341/gid_map 0 ./proc/14341/projid_map 0 ./proc/14341/setgroups 0 ./proc/14341/timers find: `./proc/14456': No such file or directory find: `./proc/14457': No such file or directory find: `./proc/14458': No such file or directory find: `./proc/14459': No such file or directory find: `./proc/27936/task/27936/fd/6': No such file or directory find: `./proc/27936/task/27936/fdinfo/6': No such file or directory find: `./proc/27936/fd/6': No such file or directory find: `./proc/27936/fdinfo/6': No such file or directory root@alfred:/#
You might run quotacheck on your filesystem just to make sure nothing is out of sync (from memory, I think you're supposed to do that on readonly filesystem, or at least from single user mode). Other than that, the reasons find won't be showing files are either "-type f" doesn't match (ie. space used is not a plain file), or find can't find/see the files. Probably the latter. Two cases come to mind that could cause that, the most likely being you have files that a mount is making inaccessible. Eg. say you have a bunch of log files under the /var/www/clients/client2/web8/log, then you bind mount /var/log/ispconfig/httpd/whatever.com at that location (which is exactly what ispconfig normally does), find will only be able to see the files from /var/log; all the files under the web8/log directory (before it is mounted) are inaccessible to find/ls/etc. but your filesystem would still count them in repquota results. The second case, which probably isn't what you have happening, is that a running program is holding open a filehandle(s) to file(s) which has been removed. The directory entry is gone, so find won't see them, but they do still take up space in the filesystem until the open filehandle(s) are closed, eg. when the program exits. (A third case comes to mind, which is also unlikely, but can happen - a rootkit hiding the file info from find/ls/etc. I wouldn't expect files hidden by a rootkit to be owned by your web8 user though, it's most likely not what's going on.)
Thank you so much for all that useful information, I've learned a thing or two about find too I think its got to do with the log folders being mounted so I checked with mount and only 3 folders for webs are mounted in my /etc/fstab so I was wondering why not all? Shouldn't all web's log folders be mounted? Currently my fstab contains: Code: /var/log/ispconfig/httpd/domain1 /var/www/clients/client10/web39/log none bind,nobootwait,_netdev 0 0 /var/log/ispconfig/httpd/domain2 /var/www/clients/client1/web73/log none bind,nobootwait,_netdev 0 0 /var/log/ispconfig/httpd/domain3 /var/www/clients/client1/web76/log none bind,nobootwait,_netdev 0 0
Till that didn't change a thing. Nothing in the error logs, its an LXC with quota enabled and the unconfined apparmor profile. So mounting should work just fine.
Code: root@alfred:~# mount -a root@alfred:~# cat /etc/fstab # UNCONFIGURED FSTAB FOR BASE SYSTEM /var/log/ispconfig/httpd/xxx.sa.com /var/www/clients/client10/web39/log none bind,nobootwait,_netdev 0 0 /var/log/ispconfig/httpd/xxx.dj /var/www/clients/client1/web73/log none bind,nobootwait,_netdev 0 0 /var/log/ispconfig/httpd/xxx.dj /var/www/clients/client1/web76/log none bind,nobootwait,_netdev 0 0
