"You email isn't arriving at all now" - everyone.
I have spent far too long writing about email and how to set up vanity domains. This really should be easy and Just Work but ... well. Here is the third post. Why do I care? Well, given how important email is as part of our online identities I do believe in taking some ownership of it, hence using a vanity domain. By using my own domain instead of an @gmail.com address I could migrate away from Gmail in the future without losing access to everything in the world. While I don't intend to go anywhere any time soon, Google does have a habit of doing odd things with its services so I'd like to have some options (he says, using Blogger which is far more at risk than Gmail...).
With that in mind, I'd like to use a vanity domain. I'd also like my email to arrive. And I'd like people to be able to email me too. High requirements, I know.
The story so far
So this is the third post on this subject (sigh). In my first post I went into detail on my requirements and the underpinning bits of security apparatus required to make email happen. I set things up using SendGrid but lamented using a marketing company for email as well as a cap on my daily email usage.
In my second post I removed SendGrid as sending / receiving wasn't consistent and switched to using the Gmail mailservers. This removed the restrictions but also made it impossible to set up DKIM and DMARC properly. I helped my setup by setting p=none which is better than nothing, but not by a lot.
Guess what? Email didn't send / receive again. This appears to have gotten worse recently, or I'm noticing it more. When three email vanished over a couple of days I cracked - I can't live with inconsistent email. It's too important.
The problem
Reading around suggests that the problem is to do with how email forwarding works. No-frills forwarding essentially throws the email at the receiving server. The receiving server then figures out what to do with it. This is fine, until one factors in load - and that all spam needs forwarding in case of false positives. The system needs to decide what to do when it is overloaded, and it seems the Gmail servers drop email in this case. Then the forwarding service needs to decide what to do and the simplest approach is to also drop the email - else they are then storing email which has its own overheads and problems.
This is a crude explanation - here is an expert explaining it far more accurately.
Considering I've been using free options, I can see why they've taken this approach but it's not good enough for me.
The solution
The solution is to use something which holds incoming email temporarily and retries if the forwarding fails. There are a few ways to do this, including some approaches using scripting and free services but as noted above I'm really bored of fiddling with this ecosystem then gaslighting myself into thinking it's working when there are a few, but notable, errors. No scripts, time for something a bit more thorough.
Enter Gmailify. Apparently Tim O'Neill suggested this to me the first time around, but either I didn't note it or I got confused with the Google feature of exactly the same name. Either way, I am now giving it a go and the pricetag ($7 / year at time of writing) is very reasonable.
Gmailify works as a forwarding / mailbox service. It controls the incoming / outgoing mail on your domain and temporarily lets the email rest in a mailbox. Gmail then uses POP3 to pull from that mailbox which then erases all trace. It also enables all the DKIM / SPF / DMARC setup that was missing before.
Setup is really straightforward if you know how to edit DNS settings and tbh should be easy if you're just confident clicking around. It gives you exactly what you need at each step, and an option to verify each step has gone in properly. The interface for routing different addresses on your domain is really easy to use too, at least for a simple setup.
Couple of things that took me a moment of thought. First, you need to set up the primary email address then configure the catch-all email address if you're used to *@domain.com. This is easy in the Email Routing submenu. Second, Gmail doesn't automatically prompt for outgoing email any more (could be because I was migrating a config?) and when modifying an existing outgoing mail rule it doesn't perform a full validation which will likely create problems down the line. I got around this by deleting my existing outgoing mail rule and setting up from scratch again. Don't forget to reset your default outgoing email address if you do this!
Oh, and if you're migrating rather than setting this up for the first time don't forget to clean up your DNS config when you're done.
All done in less time than it took me to type this up. I sent some email to Tim's overly-fussy email account and it all got through which is a first. I also ran it through this awesome tool for learning and testing DMARC settings which is worth a play if only to see how education tools should be designed. All the tests now light up a pleasing green - another first.
I've had this set up a few days so I'm keeping my fingers crossed this is the last time I have to write about this...
