Hello Guys, We have over 300,000 registered users and they receive weekly newsletters, plus they are all on the forum which generates tons of emails. We run the website off 3 powerful frontend servers and so the site loads are not high. But all emails are sent out from one of the powerful servers. That machine has exim + cpanel and we see that it takes over 2 days for a successful run of 150K emails. As the site continues to grow, I wanted to understand how we could speed up this email sending and also how we could run email queues on all servers to distribute the connections if necessary. Any ideas on how we could architect this solution or if there are better solutions? I've seen most of the hosted solutions and they are too expensive to be considered
How big are the newletters? If not that big, sending this amount should be possible to do within a couple of hours. So the question is: where lies your problem? Is it the diskload, network load, bandwith, too many concurrent connections to the same destination servers, spam/virus-checking on outgoing email, etc, etc.. I managed to generate and consume (receive) about 300.000 dummy emails per hour on a 266 MHz Pentium 3 (no diskwrite, just deliver to /dev/null). Paul
Hi Paul, The newsletters are around 100kb. For sending and clearing up the queue for 100,000 odd emails, it takes just over 12 hours on our servers. I would just hope that we are able to find faster solutions. As of now, I don't think the servers are limited by processor or disk. The loads remain consistently low ~ 1 for a 8core server. Too many connections to the same destination server is a def. problem with respect to Y!. My goal is to figure out if solutions can bring down this sending time for the 12 hour odd to an hour. Can this be done using multiple servers (we have a bunch of webservers with spare capacity as well) - how do think we would need to architect such a system? Thanks
Hmm.. personalized newletter thus? If not, things would be a lot easier to get up to par.. Assuming personalized newletters, the first thing I would try, is spreading the load. Since sharing a mailspool over NFS is not something I would like for such a 'limited' problem, I would recommend splitting the generation of the newletters over three mailservers, each taking one third. If that is not possible, have the delivery script forward the mail to one of these three servers, assuming that they accept and forward email for the generating server. FWIW, I would like to optimize the generation; any server taking a long time, but not putting in 100% CPU or I/O is not fulfilling it's purpose.. ;-) Ideal situation: non-personalized newsletters,send out using standard mailinglist-software, such as MailMan (handles bounces too, but tends to get slow for lists this large). I had a client putting out 1+ mln personalized emails per month, not adhering to/taking advantage of SMTP-standards, not willing to put in the effort to analyze bounces, so getting blacklisted for spamming every couple of months.. *sigh* Governments.. ;-) Paul
Yahoo has a form that you should fill out: http://help.yahoo.com/l/us/yahoo/mail/postmaster/postmaster-22.html Afterwards, the problem should be gone.
I don't think you can do much about it. I use mailinglist software to send the email. Now it sends 5 emails every few seconds. So it takes a while . This is the only way to get all the email out without connection and spam problems. *What i was thinking about was a way to let the software send the mail every 1 or 2 seconds, but then only with hotmail and yahoo messages every 5 or 6 seconds for connection problems. 1st second: @hotmail @hotmail @hotmail @hotmail @hotmail 3nd second: @yahoo @yahoo @yahoo @yahoo @yahoo 5th second: @other @other @other @other @other 7th second: @hotmail @hotmail @hotmail @hotmail @hotmail etc.
