Hi guys, we are an online tutorial company with a list of 200k people (all active users & legally signed-up). We have 3 mailing ‘schemes’. The Newsletter that is sent to 200k people around 4 times a month, the system emails (logins, password requests, purchase confirmation etc…), and automated emails (birthdays, event reminders, forum posts etc…) mostly triggered by cron. For the longest time we have been using online services (mail chimp, sendinblue, marketo etc…) but we are at a point where it is just not working anymore. If only for the cost of it, the unsubscribe link management between the different schemes, the synchronization of the DB on a daily basis etc… It’s time to go solo. Yes we know it is not easy, we are willing to hire a pro just to do that. Setup: We are currently using a proxmox system on OVH servers to host the site, so we can pretty much create as many containers with dedicated IPs as we want. We are also considering moving on to Google cloud if need be. Using Php5.6 on apache2 with MySQL which contains our list of user. I gathered that given the amount of emails we send, we should use Postfix on different servers with a ‘master postfix server’ and X slaves if we want our news letter to be sent in a reasonable amount of time as well as handling all the system and cron emails. Questions: - We design our newsletter on our backend and wish to trigger its sending from there. Do we need to install postfix on our web server machine? or is it better to install a separate master postfix machine that also has access to our shared DB an iterate through our contacts? I just don’t know how cpu intensive being a ‘master postfix server’ is and if it requires it’s own dedicated machine or it can run along with apache. - How do we iterate ‘elegantly’ through this list? I understand we need to be careful about the rate we are sending at. Should I ‘just send’ the whole 200k email at once and rely on postfix pacing system? Or do I need to set a cron on the master postfix machine and handle the sending pace with php? - How many emails / seconds / machine do you recommend? - How do we handle the dispatching of the list through our X machines? do we need a load balancer to do that automatically? or can we do it with php in order initiate an smtp connection per batch / per postfix slave? (I would rather do it with php). - What about the system optimization? Should I just iterate through batches of 1000 emails and send them to mailerX.mycompany.com? What about latency in between those machines? do they need to be on the same network? What kind of connection / protocol between them - Do we need SMASL authentication? - What about SPF records? If we have a domain : mycompany.com and the mailing is sent by mailer1.mycompany.com, mailer2.mycompany.com etc… do we need SPF records for all those subdomains? or just for the mycompany.com? - What about bounce emails analysis? What do we get back those bounce email messages and analyze that with a php script (on which machine?) in order to unsubscribe those users? - I read about fail fallback techniques. If mailer1.mycompany.com fails at delivering an email, then it passes it on the mailer2.mycompany.com in order to keep mailer1.mycompany.com with the highest ranking. How does one do that? is it something that postfix automatically does? - How can we monitor the sending of the emails in real time? Thanks for your answers guys.
That's a lot of questions, and all very valid. Most of this stuff is something i have to look into myself. But i can comment on one thing at least, the option to move to google cloud. I currently have my servers running there (i just started setting up a new hosting environment there) but currently with only one single postfix server. The problem with google cloud is that port 25 outgoing is blocked to prevent spam abuse of their systems obviously, and you have to make use of a third-party service that has partnered with Google to send out e-mails. I currenly use sendgrid for and their current plans are Essentials 40K a month for $9.95/mo, Essentials 100K a month for $19.95/mo. Then there is the pro with for instance 300K a month for $199,95/mo. So you can at least have a look at that cost factor. Within sendgrid btw i can whitelabel all the domains i want and setup their appropriate SPF records in the DNS. In regard as to how to send out the e-mails. Basically i would setup Postfix to throttle that for you based off course on what the network can handle and what the underlying provider would consider as acceptable. There are plenty of mechanics available in postfix to do that. Wether or not to send it batchwise to postfix completely depends on your underlying system ofc, how it generates the e-mails etc etc.
