> Viva.com's outgoing verification emails lack a Message-ID header, a requirement that has been part of the Internet Message Format specification (RFC 5322) since 2008
> ...
> `Message-ID` is one of the most basic required headers in email.
+----------------+--------+------------+----------------------------+
| Field | Min | Max number | Notes |
| | number | | |
+----------------+--------+------------+----------------------------+
| | | | |
|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
... bla bla bla ...
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
| message-id | 0* | 1 | SHOULD be present - see |
| | | | 3.6.4 |
|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
... more bla bla ...
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
| optional-field | 0 | unlimited | |
+----------------+--------+------------+----------------------------+
and in section 3.6.4:
... every message SHOULD have a "Message-ID:" field.
That says SHOULD, not MUST, so how is it a requirement?
SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either.
IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.
I once had a job where reading standards documents was my bread and butter.
SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL.
My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justified it. Before our code could be released, everything was audited by a 3rd party auditor.
Maybe the standards documents you are used to differ from RFCs, but here is the official language:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.
I just don't understand how you get from the text you pasted to "required". Nowhere does it say that anything is effectively required. Words have meaning.
> the full implications must be understood and carefully weighed before choosing a different course.
In this case, the full implication is that your email might be undeliverable. "Should" indicates that the consequences for this fall on the entity that is deviating from the thing they "should" be doing.
But the RFC language clearly anticipates there are situations and good reasons leading to a message that does not include a message-id. Google therefore would be rejecting RFC-compliant emails, and they are the ones who have to justify themselves.
Theoretically, anyway, I expect in practice they'll just ignore the issue or have their own good reason. But they should accept emails with no message-id; there it does strain the imagination to see why lacking an ID would make a message unreadable or undeliverable.
There are indeed such situations. Two situations AFAICR, and neither of them apply to when you connect to someone else's MX.
Gmail rejects the vast majority of compliant messages, I think they've stated in public that they reject >99.9% of messages, and hearsay has it that the percentage for with minor errors like this is even higher.
There are good reasons why a message might be unreadable. For example, message-id is often used by the threading algorithms in MUAs and IMAP servers, and many don't test whether their threading code handles ID-less messages. I use one that deduplicates by ID, what do you think it does when the ID is empty or missing? I don't know, I haven't tested and I'm not going to.
- `MUST: This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification.`
- `SHOULD: This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.`
- `MAY: This word, or the adjective "OPTIONAL", mean that an item is truly optional.`
In practical terms:
- MUST: It's always a failure to do this. E.g. you MUST have some form of stored energy in your car for it to propel itself down the highway.
- SHOULD: If you don't do this, it's likely to cause failures unless you really know the situation is one of great exception and have thought about what else this change may affect. E.g. you SHOULD maintain a large distance between yourself and then next vehicle on the highway (an example of an exceptional case might be a standstill backup on the highway).
- MAY: This is something which is actually optional and has negligible impacts to successful operation if you do/don't. E.g. you MAY activate cruise control instead of always manually operating the accelerator.
For your car example as-is it'd probably best be MUST unless there is expectation one might reasonably consider their car exploding a valid scenario. In the real world where the car doesn't actually blow, it'd probably be that you MAY leave your keys in the ignition rather than SHOULD/MUST.
Is that what the spec says? Or is this something that Google decided, by making an optional feature a requirement when interoperating with their systems?
It is something Google decided. SHOULD means the other party should anticipate may not. The party examining a SHOULD and deciding not to do something is obviously not required to consider incompetence of other RFC readers as a reason to global replace SHOULD with SHALL before examining requirements.
You should wear sunscreen to the beach. Its recommended as a good way to prevent sunburn. However, the beach police aren't going to come get you for not wearing it. You just might get a sunburn if you don't plan accordingly with other sun countermeasures.
So if I send an email that lacks a feature that MUST be there, will the email police come get me? At a certain point, looking for an analogy stops making sense I think.
No, its not a "required"... It means someone may have reasons not to use something, and so spec implementors need to allow for circumstances where it is not present.
Those reasons can be anything. Legal, practical, technological, ideaological. You don't know. All you know is not using it is explicitly permitted.
"permitted" is a pretty empty word in the given context. Because dropping such emails is equally "permitted". Sure, there will be no arrests made, but there will be consequences. And those are what this article is about.
If that's your line, then I am equally permitted to send random binary blobs along the way. Not a crime, so totally permitted. They'll just drop the connection.
Buuut I don't think that is at all relevant to the discussion at hand.
I don't even know how you got to "used twice" tbh. Both your own comment AND the post you quoted from only have a single "must".
The only thing that text demands is understanding and carefully weighing the implications. If, having done that, you conclude that you don't want to then there is absolutely nothing in the spec stopping you. Maybe the spec would have been better off putting more stuff in SHALL and less in SHOULD, but as written that is definitely not the case.
Any time any document (standards or otherwise) says something is recommended, then of course you should think it through before going against the recommendation. Going from their verbiage to:
> SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.
Can you start actually reading what you quote? "The implications must be understood and weighed" does not mean "you are required to have a good reason not to do it". I can know of the Message-ID field, understand what it's used for, and carefully weigh the possible lack of interoperability against the effort required, concluding that I'm too lazy to do it right now. I have then understood and carefully weighed the implications before choosing a different course, but I don't have a good reason to.
Words mean things, especially in standard texts. You can't just carelessly rewrite sentences using not-quite-synonyms like you're doing here.
Oh, and that "must" is not a "MUST". Had the text said "The full implications MUST be understood ..." it would have been a proper requirement by the standard, but this lower-case "must" is just normal part of prose and not the magical "MUST" word which formally imposes a requirement.
This very clearly says that SHOULD is not effectively REQUIRED at all, and is fact nothing more than RECOMMENDED. Really not sure how you misinterpreted this so badly
It's not required but they need to understand the implications. In this case the implication is that Google drops the mail. So clearly they didn't understand the implications.
Google probably did parse these messages as well-formed before inspecting them and deciding to drop them based on the lack of this field. The RFC imposes no mandatory obligation to deliver messages just because they are well-formed.
Assuming we’re talking about RFC 2119, it’s important not to collapse the distinction between SHOULD and MAY, which is there for a reason. MAY elements are legitimately optional, SHOULD elements are there for a reason and are disregarded at one’s own risk.
> SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
To validly disregard a SHOULD, you need to (a) fully understand the implications, and (b) accept them.
Any time someone disregards a SHOULD and then complains about the result, they are necessarily in the wrong. Either they didn’t fully understand the implications, or they don’t actually accept them.
Agreed. We use RFC language at work for documents, and whenever I see someone using the word SHOULD, I always ask what is the MAY where you don't need to do the thing? If they can't think of one, then make it a MUST. SHOULD always means there are valid reasons to not do it.
Could was yeah nice to have.
Should, was yeah the system shall have it, but it's ok if not.
Shall : ffs we will hunt you and kill you if you don't implement it.
Yes, except there seems to be a move on the best words from SHALL to MUST and from SHOULD to MAY.
IANAL but I recall reading this in e.g. legal language guidance sites.
Pulling exact quotes out, SHOULD means "there may exist valid reasons in particular circumstances to ignore a particular item" while MAY means "an item is truly optional."
I don't think this can be interpreted as simply "should is optional".
Well… yes… no… SHOULD means that you must either do the thing or understand the consequences of not doing it. That's not simply optional, the two options are to ① spend time on writing code and ② spend time on learning the consequences. Either way you need to think hard and spend time. And that's why the definition of SHOULD includes the word "must".
> but the full implications must be understood and carefully weighed before choosing a different course.
In other words, you better have a damn good reason for deciding not to do it.
And (possibly even more to the point), you must decide not to do it, rather than simply throwing code at the wall until most emails make it through unscathed.
I think that is a bit to easy. MAY is described ar optional.
SHOULD - Should really be there. It's not MUST, you can ignore it but do not come crying if your email is not delivered to some of your customers !
you should have though about that before.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.
If the server has considered fully the implications of not having a Message-ID header, then it MAY continue processing.
In general, you will find most of the Internet specifications are labelled MUST if they are required for the protocol's own state-processing (i.e. as documented), while specifications are labelled SHOULD if they are required for application state-processing in some circumstances (i.e. other users of the protocol).
> If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.
That is not a rule.
In this situation the server can reject any message if it wants to, and not doing a SHOULD tests the server's patience, but it's still ultimately in the "server wanted to" category, not the "RFC was violated" category.
> But I'm not playing, because you're right: I don't know why you would be doing that.
This is the comment you originally replied to:
>> If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.
>> You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.
They're talking about how to categorize actions, just like I am.
So I thought you were playing on that same topic.
But if you weren't on that topic, and given that you quoted only the first one of those sentences, I have a guess.
I think you didn't realize how the second sentence affects the meaning of the first one, and you misunderstood what they were saying as trying to tell servers they can't reject. They were not trying to tell servers they can't reject.
If that's the case, then this whole line of conversation is pointless, because you were rebutting an argument that nobody made.
> It's a description of the behaviour of other Internet hosts, and it's a description of exactly what is happening in the linked article.
Taking a description of behavior and throwing an RFC-style "SHOULD" in front is only going to be correct if you get lucky. "is" and "SHOULD" are different things!
How does Google know whether or not the sender has a valid reason? They cannot know that so for them to reject an email for it means they would reject emails that have valid reasons as well.
How would the sender know the consequences of sending without the header? You shouldn’t assume anything here. As a sender, you should include it unless you’ve already worked out what the recipient is expecting or how it will be handled. Doing this with email is silly because the client is sennding to so many different servers they know nothing about so it’s basically a requirement to include it.
It is not required for the protocol of one SMTP client sending one message to one SMTP server, but it is required for many Internet Mail applications to function properly.
This one for example, is where if you want to send an email to some sites, you are going to need a Message-ID, so you SHOULD add one if you're the originating mail site.
> How does Google know whether or not the sender has a valid reason?
If the Sender has a valid reason, they would have responded to the RFC (Request For Comments) telling implementers what they SHOULD do, rather than do their own thing and hope for the best!
Google knows the meaning of the word SHOULD.
> it means they would reject emails that have valid reasons as well.
No shit! They reject spam for example. And there's more than a few RFC's about that. Here's one about spam that specifically talks about using Message-ID:
> If the server has considered fully the implications
The server "considers" nothing. The considerations are for the human implementers to make when building their software. And they can never presume to know why the software on the other side is working a certain way. Only that the RFC didn't make something mandatory.
The rejection isn't to be compliant with the RFC, it's a choice made by the server implementers.
Either the server must explicitly confirm to servers or the clients must accept everything. Otherwise message delivery is not guaranteed. In the context of an email protocol, this often is a silent failure which causes real-world problems.
I don’t care what the protocol rfc says, the client arbitrarily rejecting an email from the server for some missing unimportant header (for deduction detection?) is silly.
These changes MUST NOT be applied by an SMTP server that
provides an intermediate relay function.
That's Google in this situation.
> Stop hiding behind policy and think for yourself.
Sometimes you should think for yourself, but sometimes, and friend let me tell you this is one of those times, you should take some time to read all of the things that other people have thought about a subject, especially when that subject is as big and old as email.
There is no good reason viva couldn't make a Message-ID, but there's a good reason to believe they can't handle delivery status notifications, and if they can't do that, they are causing bigger problems than just this.
“MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional… An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)”
Note how it explicitly calls out interoperation with implementations that do or do not implement MAY. As a exception that proves the rule, we can reasonably assume that not interoperating with a system ignoring a SHOULD rule is a correct implementation and it is the fault of whoever is not implementing SHOULD.
Hearsay has it that the reason is spam. Spam messages are said to have massively higher chances of minor RFC violations when they arrive at the destination server.
Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.
Because in practice it showed up for a period of time as a common thing in spam-senders. They were trying to maximize throughput and minimize software maintenance costs, so they leave out things that the spec says are optional. But that makes "a commonly-implemented optional thing was left out" into a stronger spam signal.
Is it still a strong spam signal? Hard to say. Sources disagree. But as with laws, heuristics, once added, are often sticky.
CVE classify a lot of things that have nothing to do with security.
Not having a Message-ID can cause problems for loop-detection (especially on busy netnews and mailing lists), and with reliable delivery status notification.
Dealing with these things for clients who can't read the RFC wastes memory and time which can potentially deny legitimate users access to services
> It seems that Gmail is being pedantic for no reason
Well, gmail does not manage usenet groups and mailing lists. Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
> Well, [google] does not manage usenet groups and mailing lists.
They do. Sort of.
Google used to nntp, and manages the largest usenet archive; They still have one of the largest mailing list servers in the world, and they still perform distribution on those lists via SMTP email.
They still have all of the problems associated with it, as do lots of other mail/news/list sites still do that are a fraction of Google's size.
> Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Sure it does.
You consider them best-effort, but that doesn't follow that I should consider them best-effort. For a simple example: Consider spam.
In any event, if you keep sending me the same message without any evidence you can handle them, I'm not going to accept your messages either, because I don't know what else you aren't doing. That's part of the subtext of "SHOULD".
Most big sites take this policy because it is internal nodes that will generate the delivery notification, but the edge nodes that are tasked with preventing loops. If the edge node adds a Message-ID based on the content, it'll waste CPU and possibly deny service; If the edge node naively adds a Message-ID like an MSA, the origin won't recognise it, and forwarded messages can loop or (if sent to a mailing list) be amplified. There also are other specific documented requirements related to Internet Mail that edge nodes not do this (e.g. RFC2821 § 6.3).
However you seem to be assuming Google is blocking messages "for this case" which is a little presumptuous. Google is presumably trying to save themselves a headache of handling errors for people who aren't prepared to do anything about it, the most common of which is spam. And the use of Message-ID in this application is documented at least as early as RFC2635.
> Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Without knowing what Google does with the g-msgid header, you are making a mistake to assume it is equivalent to the Message-ID header just because it has a similar name. You have no reason to believe this is true.
> Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
I am sorry you are offended to not know things, but you do not know this thing, and your characterising my actions will make it very difficult for you to learn something new and not be so ignorant in the future.
Think hard exactly about what you want to happen here: Do you want Google (et al) to do something different? Do you want me to agree Google should? Who exactly are you trying to convince of what, and to what end?
I am trying to tell you how to interpret the documentation of the Internet, in this case to be successful sending email. That's it.
I am not likely to try and tell Google what to do in this case because of my own experiences running mail servers over the last 30 years, but listen: I am willing to be convinced. That's all I can do.
If it's something else, I'm sorry I just don't understand.
> If it's something else, I'm sorry I just don't understand.
I'm trying to explain to you that you're speaking very authoritatively about how things were done 30 years ago but things have changed since then, Gmail won't even send a message non-delivery notification to non-DKIM hosts.
So add a message id at the first stop, or hard ban the sender server version until they confirm. A midway point that involves a doom switch is not a good option.
As someone else who does System Engineering, when dealing with ancient protocols, "shall" is extremely difficult barrier to get over since there is always ancient stuff out there and there might be cases not to do it, esp if it's internal communication.
"SHOULD" is basically, if you control both sides of conversation, you can decide if it's required looking at your requirements. If you are talking between systems where you don't control both sides of conversation, you should do all "SHOULD" requirements with fail back in cases where other side won't understand you. If for reason you don't do "SHOULD" requirement, reason should be a blog article that people understand.
For example, "SHOULD" requirement would be "all deployable artifacts SHOULD be packaged in OCI container". There are cases where "SHOULD" doesn't work but those are well documented.
I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers.
The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software).
This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.
In a completely different field, navigating ships at sea, the Collision Regulations which define how people must conduct ships at sea, they use the words "Shall" and "May" to differentiate legal requirements and what may just be best practice. "Should" intuitively means something more like "May" to me
Note "the full implications must be understood and carefully weighed before choosing a different course". Gmail and the other big hosters have full-time spam teams who spend a lot of time weighing implications, so I assume the implications of this was weighed.
I feel like "SHOULD" and "SHOULD NOT" are redundant. You end up having to assume someone else treated them as a "MAY". If you control all the endpoints in a private implementation you can just deviate from the standard & not implement a MUST, it's your private implementation. There's thus no difference in public implementations between "SHOULD" and "MAY", and no difference in internal implementations between any of the words. They are therefore redundant, requirements are either mandatory or optional, there's no middle.
The original email RFC is also completely unaware of how bad spam is. Sure it might mention it but it's not really AWARE of the problem. The truth is, companies like Google, Microsoft and a few others have de-facto adjusted the minimum requirements for an email. Signing, anti-spam-agreements, etc.. are the true standard if you want an email to get from point a to b. (none of which are going to be REQUIRED in the RFC)
Given the amount of time that has passed, the big email companies have had many years to jointly update the RFC according to their experienced reality. They haven't done so and the way you describe how they effectively work together to adjust the minimum requirements (outside the RFC) is de-facto use of market power to the exclusion of other players.
If you read RFC 2119, you'll see that the definition of SHOULD includes the word must. Perhaps it's easier to understand if I rephrase it as: "If you understand the issue and its consequences, SHOULD means optional. If you don't, SHOULD is equivalent to MUST."
Message-ID has maybe half a dozen types of problem. If you know all of them, leaving it out is up to you.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
Not sure how the people at Google interpreted this about the message-id
You can argue that you not obligated to use message-id but if you don't use it you should blame only yourself that your messages are not accepted. In requiring message-id I would side with google (though in general I think they anti-spam is too aggressive and lacks ways to report false positives). Full RFC compliance (as in not only MUST but also SHOULD unless you have a very good reason) is the easiest part of making sure your emails will be delivered.
> if you don't use it you should blame only yourself that your messages are not accepted
I think it's a gray area
- If the receiver declines your message because "Message-id" is required - then I blame the receiver; because that's not true
- If the receiver declines your message because "most systems do include it, and it's lack of presence is highly correlated with spam email", then it's on the sender
I think it's the latter. But, in either case, you're right in that you get the same result.
Now, let's assume that if it is the latter (it's spam related), and Google were to accept the message, but then internally bin the message, it would be worse. At least in this case, they are bouncing the message. Because of this, the sender is at least aware that the message wasn't delivered.
Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.
My takeaway is that I think that Google is doing the least-wrong thing. And by being explicit in how they are handling it, it at least made the debugging for the author possible.
Also note: in a quick reading of RFC5321 (SMTP), rejecting messages for "policy reasons" is an acceptable outcome. I'm not sure if it applies completely here. The author should probably also be taking into account RFC5321 (SMTP) instead of just 5322 (message format).
> Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.
That's the annoying part to me.
An email is an email. By applying different rules for rejection on different mailboxes, gmail creates a system where it's harder for would-be implementers to test compliance.
If tomorrow gmail creates a new type of mailbox, will there be a third set of rules to have your message delivered?
There are dozens of spam and security settings that admins can change in the Google Workspace console, presumably because different businesses have different requirements. So in practice, there's not just two sets of rules in gmail -- there's probably thousands or millions (however many combinations of settings are actually in use).
In my experience, email is an unreliable way to communicate any time-bounded critical information. When I want to be sure an email was transmitted on either side, the only reliable way to ensure this is to use a distinct channel to validate reception and confirm content.
That is, when some hotline tell me that they just sent and email with the information, I ensure they hold the line until I got the actual email and checked it delivers the relevant information to fulfill the intended process. And when I want to make sure an email was received, I call the person and ask to check, waiting until confirmation.
It’s not that much SMTP/IMAP per se as the whole ecosystem. People can legitimately get fatigue of "is it in my junk directory", "it might be relayed but only after the overloaded spam/junk analyzer accept it", or whatever can go wrong in the MUA, MSA, MTA, MX, MDA chain. And of course people can simply lie, and pretend the email was sent/received when they couldn’t bother less with the actual delivery status.
There are of course many cases where emails is fantastic.
Email is an unreliable way to communicate any information, in the strictest sense of the word "reliable." The protocol does not guarantee that any email will be delivered, nor does it guarantee that failure will be detected. It's a good-faith effort. The bits could drop on the floor at any point and you might never know.
Does it even matter when in reality it's more likely that this is intentional anti-competitive behavior by Google?
They once made all emails from my very reputable small German email provider (a company that has existed and provided email services long before Google existed) go into a black whole - not bounce them back or anything like that, mind you, their servers accepted them and made them disappear forever. I was in contact with the technicians then to get the problem fixed and they told me it's very difficult for them to even reach anyone at Google. It took them several days to get the problem fixed.
Of course, no one will ever be able to prove an intention behind these kind of "technical glitches." Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it, so certainly nothing will happen when small issues with interoperability occur and drive more people to Gmail.
At scale, it's very hard to distinguish malicious intent from the simple consequence of being the largest operator in a space so any motion one makes makes waves.
For what it's worth: having seen some of how the sausage is made, Google isn't particularly interested in screwing over a small reputable German provider. But they also aren't particularly interested in supporting such a provider's desire to route messages to their users because the provider is small. At their scale, "I've forgotten how to count that low" is a real effect. And email, as a protocol, has been pretty busted for decades; it's not Google that made the protocol so open-ended and messy to implement in a way that providers will agree is correct.
> Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it
Nothing of significance outside Google. Inside, Google initiated a technical lift that turned their intranet into an untrusted-by-default ecosystem so that data was encrypted on the fiber (as well as between machines within a datacenter, to head off future compromised-employee attacks). That process took at least five years; I suppose there's a scenario where it was all smoke and mirrors, but being on the inside in the middle of the process, I watched several C-suite who are not particularly good actors be bloody pissed at the US government for putting itself into Google's "threat actor" box and making that much work for the system engineering teams.
Actors in a theater. I have no reason to believe they were pretending to be upset because the were actually spies and knew what was going on before Snowden's reveals came out. They were as surprised as everyone else; the NSA wiretapping was done off-prem in cable that was privately owned (but stretched hundreds of miles and was, therefore, practically undefendable against compromise).
Long time ago when I was managing ISP email relay and customers asked "Where is the message I've sent?" seeing in the logs message accepted by receiving SMTP server was the end of the debug for me: I just handed the customer the part of the log and suggested talking to the receiving side IT administrator.
Google is rejecting it to ensure incoming messages aren't spam. SHOULD means "you should do this unless you have a really, really good reason not to." Do they have a good reason not to? It doesn't seem so, meaning Viva is in the wrong here.
No, SHOULD is defined in the RFC, not by colloquial usage. Google is on the wrong, regardless of their "safety" intent.
After all, linguistics is full with examples of words that are spelled the same, but have different meaning in different cultures. I'm glad the RFC spelled it out it for everyone.
if Google's choices are protecting users, they can't be in the wrong. That's the reality of a shared communications infrastructure regardless of what the docs say.
When the docs disagree with the reality of threat-actor behavior, reality has to win because reality can't be fooled.
Did i miss the part of the RFC that says google must accept every message? Pretty sure the RFC allows email providers to reject any message they feel like.
RFC cannot force a mail server to accept spam. You may argue that requiring message-id is a bad anti-spam policy but it does reduce amount of spam. In my observations around a half or messages without message-id are spam. I would not use personally this as the only reason to reject a message but I understand why someone may choose to do.
Per RFC2119:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
So, it's fairly explicit that the sender should use message-id unless there's a good reason to not do so. The spec is quiet about the recipients behavior (unless there's another spec that calls it out).
Postel's law was a precept of the Internet of the 80's and 90's, when due to the primitive software engineering practices at that time, implementations couldn't be tested properly. That lead to many cases of poor interoperability, and it's no longer a good idea: for example, when HTML 5 was designed, they decide to put into the spec how to deal with the frequent errors like mismatched closing tags, etc... because all major implementations were "liberal" in what they accepted, but each in a different way.
(No seriously, I’m asking; are there examples of where it’s actually different from a MUST)?
Also this reminds me of something I read somewhere a long time ago: when specifying requirements don’t bother with SHOULD. Either you want it or you don’t. Because if it’s not a requirement, some people won’t implement it.
I guess the one time it’s good is if you want an optional feature or are moving towards requiring it. In this case Google has decided it’s not an optional feature.
Typically, MUST means that if you don't do that then something will break at the protocol level.
SHOULD means that if you don't that, bad things are likely to happen, but it will not immediately break at the protocol level and during discussion in the IETF some people thought there could be valid reasons to violate the SHOULD.
Typically, IETF standards track RFCs consider the immediate effects of the protocol but often do not consider operational reality very well.
Sometimes operational reality is that a MUST gets violated because the standard is just wrong. Sometimes a SHOULD becomes required, etc.
Certainly for email, there is a lot you have to do to get your email accepted that is not spelled out in the RFCs.
"SHOULD generally means: some people might require it."
No it absolutely does not mean that. It means, by explicit definition which is right here, that text is exactly that definition, that no one requires it. They can't require it, and still be conforming to the spec or rfc. That's the entire point of that text is to define that and remove all ambiguity about it.
It's not required by anyone.
The reason it's there at all, and has "should" is that it's useful and helpful and good to include, all else being equal.
But by the very definition itself, no people require it. No people are allowed to require it.
Incorrect. Not required is not required. You do not need to supply rationale or get agreement by anyone else that your reasons are good in their opinion and not just in your opinion.
Should just means the thing is preferred. It's something that is good and useful and helpful to do.
That is not "must unless you can convince me that you should be excused".
“ SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.”
I don't know if you thought you were validating or invalidating my argument, but this says exactly what I said.
Further, not only the presense of the words "should" and the doubling down with "recommended", also the absense of the words "must" or "required" or "shall" is pointed.
Because those words are used elsewhere. When something is actually required, those words are used, and so when they are not used, it is meaningful. It means no one is free to put their own words into the authors mouths and say "well they didn't say it but obviously they pretty much meant required" when they so pointedly chose not to say required.
And it's not a technicality. A thing either is or is not required. There are no other states. The spec does not require it, and so for Google to require it, Google is not adhering to the spec.
Since the spec does not require it, you can't say either that it falls under "full implications". "full implications" cannot include "Some servers might require it", because that would be circular logic. No server can require it in the first place.
I SHOULD have 8 hours of sleep every night. It's RECOMMENDED. However, there are times where it's best I don't (e.g. because of work, or travel, or needing to take someone to the hospital, etc.). It's definitely not that I MUST sleep 8 hours every night.
“When jump getting over a wall, you SHOULD use three points of contact.”
For most cases you should use three points of contact. However, there may be other situations for example if someone is giving you a leg up, or you can pole vault, where another solution is preferred.
You assume that internet standards are prescriptivist; that the document describes how it is to be implemented. In practice it's often descriptivist, with the standards documents playing catch-up with how things are actually going in practice.
Anyway, in general you can expect that doing unusual but technically valid things with email headers will very often get your messages rejected or filtered as spam.
Standards are definitely prescriptive. But just like a medical prescription, it doesn’t ensure that actors in the wild will conform to what’s prescribed. People will not follow prescriptions for whatever reason, willingly or otherwise. It doesn’t mean the document wasn’t prescriptive.
For producers, ignoring a SHOULD is riskier because it shifts the burden to every consumer.
For consumers, ignoring a SHOULD mostly affects their own robustness.
But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.
I think a mail 2.0 would be notify and pull based.... you notify a recipient's mail server that there's a message from <address> for them, then that server connects to the MX of record for the domain of <address> and retrieves <message-id> message.
Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.
Is the idea that you could decide from the envelope whether you want to even bother fetching the message? Besides that I'm not sure I see the advantage
You have to have a working mail server attached to a domain to be able to send mail... that's the big part. Right now, email can more or less come to anywhere from anywhere as anyone. There are extensions for signing connections, tls, etc... but in general SMTP at it's core is pretty open and there have been efforts to close this.
It would simply close the loop and push the burden of the messages onto the sender's system mostly.
And yes, you can decide from the envelope, and a higher chance of envelope validity.
jmap is the communication between a mail client and shared directory/mail services on a server. It does not include server to server communications (that I am aware of) for sending mail to other users/servers.
My take, as a postmaster for hosting company, who don't have any sympathy to gmail (that should be visible from my comments history):
Message-ID is absolutely MUST in production e-mails. You can send your test stuff without it, but real messages always have it. Not having Message-ID's causes lot of fun things. All somewhat competent software is capable to add Message-ID's, so lack of it is good indication of poorly made custom (usually spamming) solution.
Rspamd and spamassassin have missing MID check in their default rules, I am sure that most antispam software is same.
Your casual use of the word MUST is not the same as a standard document's use of the word MUST. Your real world experience is entirely irrelevant to the conversation about what the standard requires.
Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. One lesson surely most everyone learns in CS is that unique identifiers are maybe unique to the system generating them, but to rely on foreign generated IDs being unique globally is a terrible idea that will break within the minute.
So at that point the ID has no value to me except being obliged to carry it around with the message, so maybe the originating system can at some point make sense of it. But then there is obviously no reason to ever reject mail without it, it's an ID valid for the sender and the sender didn't care to include one, great, we save on storage.
>Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. [...] So at that point the ID has no value to me
Your framework of analysis is based on someone else's database key ids being irrelevant to you. That's true.
But another framework of analysis is tracking statistical correlations of what spam looks like. Lots of spam often don't have message ids. Therefore it's used as a heuristic in scoring it as potential spam. That's why other postmasters even without SpamAssassin independently arrive at the same answer of trying to block messages without a message id. Example: https://serverfault.com/questions/629923/blocking-messages-w...
MID-s are used by MUA-s for referring earlier messages, tracking answers and so on. So any software expecting dialog (messages coming back) needs to deal with MID-s correctly. Missing MID-s show that said communication is one direction, because broken dialog has not been problem.
An unrelated frustration of mine is that Message-ID really should not be overridden but SES for instance throws away your Message-ID and replaces it with another one :(
It is de-facto required and has been for many years.
Should in most RFCs also mean "do it as long as you don't have a very good technical reason not to do it". Like it's most times a "weak must". And in that case the only reason it isn't must is for backward compatibility with older mail system not used for sending automated mails.
And it is documented if you read any larger mail providers docs about "what to do that my automated mails don't get misclassified as spam". And spam rejection is a whole additional non-standardized layer on top of RFCs anyone working with mail should be aware of. In any decades old non centralized communication system without ever green standards having other "industry standard/de-factor" but not "standardized" requirements is pretty normal btw.
What's de-facto required and not is completely irrelevant here. The claim was that "Message-ID header [is a] a requirement that has been part of the Internet Message Format specification ... since 2008" (emphasis mine), which is false; it's a recommendation that has been part of the Internet Message Format specification since 2008
I would read this as a requirement for email to be 'legit' and not classified as spam.
Sure, you can send email with whatever headers you want, use weird combos, IP addresses, reply-to, and it might be still a technically valid email, but not something that should land in people's inboxes.
Also, a payment processor not testing their email on the most popular email provider in the world is quite ridiculous.
As indicated in the RFC, it uses another RFC[0] to define those words. Here's the relevant excerpt from that one:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
yeah, but that RFC isn't the only relevant document
Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known.
and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn.
In addition there are some de-facto standards not pinned down in any RFC, like e.g.:
- Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely).
- You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work...
- You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble.
- Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC.
- A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it.
so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice
As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly.
oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).
I know you're looking for "pedant points" but the specification generally take a backseat to implementation. If Message-ID is expected out here where the rubber meets the road, then you are the squeaky wheel in this scenario for not including it.
The reason that European tech sucks is that people in Europe are open to such arguments. If an engineer in the US started talking about SHOULD vs MUST, some PM would just give them that "what the fuck did I just listen to" face, spend the next few minutes gently trying to convince them that the customer experience matters more than the spec, and if they fail, escalate and get the decision they want.
For example, why does Google handle this differently for consumer and enterprise accounts? Well it's Google so the answer could always just be "they are disorganized" but there's a good chance that in both cases, it was the pragmatic choice given the slightly different priorities of these types of customers.
Not my PM (in the US). My PM would try to avoid anything that is not absolutely necessary and therefore ask developers not to develop anything that isn't a MUST. I know that we like making fun of Europe for their alleged lack of innovation but this isn't a Europe thing.
Your PM most definitely would not tell you to skip a feature that is needed for your emails to be delivered to Gmail accounts. What a preposterous thing to lie about.
Do bugs and bad implementations not exist in US software? If an US company did this, nobody would be bloviating about how it is a cultural issue or whatever.
> some PM would just give them that "what the fuck did I just listen to" face
Some people have become way too comfortable taking for granted that it is okay to treat others in an uncivil fashion. To those I say: keep it up, and we shall revert to the fundamentals of society where violence is an option and one day you are copping punches in the face, or getting shackled and thrown in a ditch somewhere for the ravens. Behave like an animal, get treated like one.
Standards are important, and the meaning assigned to words are important, but if your very important email does go through, because Google thinks you're wrong, you add the bloody message-id. I really do agree, I don't care about the linguistic/legal/standard/technological reason as to why Google might be wrong, if I can't deliver email to Gsuite customers, I add the message-id.
You're precisely right that customer experience matter, but I wouldn't put it past some conservative European company to go: Well Google is wrong, so they should fix that. Google doesn't care, you can't make them care, you can't even contact them. Just make it work for your customer.
Seriously, I hope that none of the posters upthread arguing about SHOULD v/s MUST in some standards body document are in charge of real world money making software.
Google and Microsoft's email practices define a pseudo-RFC in practice. As an engineer, I hate this. As a civic participant I can vote against it. But as a person that sells my software services for a living, I am going to implement the Google/Microsoft standards to the letter, not argue about definitions in an RFC.
And the definition of "SHOULD" (from RFC 2119) is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."
Having said that, I regret my original characterization of the Message-ID header as a "requirement" and have updated the blogpost to be fair to all sides.
1. SHOULD means, do it if you can/you have to have a really good reason if you don't do it. The only reason it's SHOULD and not MUST is backward compatibility. Mostly in context of "personal send mails", i.e. not automated mails. (For automated mail sending the expectations of you running somewhat up to date software is much higher).
2. You can't really implement mail stuff just based on RFCs:
- There docent overlapping RFCs which can sometimes influence each other and many of them obsolete older versions why others still relevant RFCs reference this older versions. This makes it hard to even know what actually is required/recommendation.
- Then you have a lot of "irrelevant" parts, which where standardized but are hardly supported/if at all. You probably should somewhat support them as recipient but should never produce them as sender today (mostly stuff related to pro-"everything is utf8" days). Like in general the ideas of "how mail should probably work" in old RFCs and "how it is done IRL today" are in some aspects _very_ far away.
- Lastly RFCs are not sufficient by themself. They don't cover large parts of the system for "spam detection/suspicious mail rejection". So it's a must have to go to the support pages of all large mail providers and read through what they expect of mails. And "automated mails need a message id" is a pretty common requirement. In addition you have to e.g. make sure the domain you use isn't black listed (e.g. due to behavior of a previous user), and that your servers IP addresses aren't black listed (they never should be black listed long term, but happens anyway, and e.g. MS has based on very questionable excuses "conveniently" black listed smaller local data center competition while also being one of the most widely used providers for commercial mail in that area).
The reason it’s recommended is that it’s useful for detecting when an email you receive is already in your mailbox, so that you don’t accumulate duplicates. Otherwise one would have to compare the complete email, which probably no MUA does. Another reason is that replies can include a reference to the original message, so that it can be properly threaded by MUAs.
So these are mostly quality-of-life reasons, it’s not a reason to reject an email.
> That says SHOULD, not MUST, so how is it a requirement?
Battle with spam has been for long part just trying to algorithmically fingerprint the scam bots and reject the message if it looks like it wasn't sent by "real" mail server/client.
So a lot of things that are optional like SPF/DKIM are basically "implement this else your mail have good chance of being put into spam automatically".
SHALL has been interpreted/clarified by US courts as not being a fancy MUST or REQUIRED that many people were taught it to mean, but SHOULD still has it's purposes, e.g., to provide contractual flexibility in development, i.e., a MUST/REQUIRED requirement was more challenging or complicated and took up more time/resources than anticipated, so SHOULDs can be trimmed due to contingencies.
Another example may be a lightweight implementation of a spec in a limited and/or narrow environment, which remains technically compliant with full implementations of a spec but interaction with such a limited/narrow environment comes with awareness about such limitations.
Worked on an ESP. We had a couple of server software we used on low-level for sending. None of them would accept the message without a Message-ID. But even if you have a super-custom, SMTP-injecting service built, how can you ignore all of these bounces from a provider thats likeliest to be the major one you are sending to? Unthinkable. I would not like to have business with such a payment provider.
This is the one that gets me - sometimes you're forced to work with systems that do annoying things that you have to accommodate. It's annoying, but it's more important to do the thing that prevents your users from having issues than it is to be theoretically right about whether something's required by a standard.
I've dealt with many worse cases than this, where the systems I was integrating with were doing things that weren't even close to reasonable, but they had the market power so I sucked it up and dealt with it for the sake of my users. Maybe Google's wrong here, but how do you not just implement the solution anyway?
> Maybe Google's wrong here, but how do you not just implement the solution anyway?
But they just did (make it work). The logical assumption is that most ppl did the same, just used another email provider. Why would viva care? (same as google, why would google care?).
Well, apparently it's not even an issue for gmail users:
To unblock myself, I switched to a personal @gmail.com address for the account. Gmail's own receiving infrastructure is apparently more lenient with messages, or perhaps routes them differently. The verification email came through.
So it's only an issue for people paying for Google's hosted email—a much smaller set!
> how can you ignore all of these bounces from a provider thats likeliest to be the major one you are sending to?
This is the major issue that most of the discussion is missing. It doesn't matter how you want to interpret the word SHOULD, if you want to send to google workspace, you MUST include a message-id. It's not like this is some fly-by-night server with 12 clients.
If you absolutely and completely don't want to include the message-id, then you need to have a warning that your service can't be used by Google Workspace customers. This used to be common practice, blocking communication to servers that behaved badly, and I sort of wish we'd bring it back.
> I would not like to have business with such a payment provider.
Chances are that the decision-makers in most companies don't care about the technicalities (i.e. which email you used for registration) - they want to get up and running.
The reason that Viva doesn't care, I assume, is the reason Google workspace doesn't care: they're both too big to care for 5% of their clients won't do the extra work. They know that their, usually much smaller clients, will "figure it out" by i.e. using another setup that works™. So why bother?
Since workspaces are probably tied to custom domains all they see if only cursorly looking are "random" businesses with bounces (as TFA mentions, Gmail addresses seem to work fine), sure if you look closer you'll notice the MX might be a google one but going from the cursory look to a closer one takes a bit of time.
For now, but with EU digital sovereignty efforts in full swing, it's possible this changes over time. More so if the EU uses regulation to dissuade the use of US Big Tech products and services.
This is round 5 or 6 of a " EU digital sovereignty efforts in full swing" maybe this time they will having a success or two but nothing seems to indicate they've changed enough to avoid failing again.
... which is irrelevant to the demonstrated and shocking incompetence of being unable to deliver either to the #1 or #2 inbox for businesses in the EU?
My pet peeve are services that go out of their way to include a text/plain alternative message part but send something useless, such as the message without the key link. One time I seriously ran into a service just send a short one-sentence note along the lines of "this is a plain text email" as the plain text part. If you don't want to support plain text, maybe just don't send the alternative part?
I find the ones that try to be cute the most frustrating because these appear on the new message notifications so I can't just delete them straight from the notification.
We'd love to share this exciting announcement but you'll a different email app.
Although I guess the argument will be that email clients should use AI to summarise the HTML into a plain text summary.
This might be me being old, but I still don't understand why html emails aren't the exception. If you want to do a fancy newsletter, trying to sell me crap, I can see why you'd need the images, the css and html. In most other cases, I don't really get the point.
- You are sending a receipt and want table alignment for items
- You want to put a logo of your company so that readers can recognize who's the email from
- You want to make unsubscribe link smaller and the "open the thing I'm notifying you about" link bigger, so that people would know which one is which without reading the url
Mostly those seems to be more about you as a sender wanting to do some branding or manipulation of the reader. I don't really see how it benefit the receiver, which should be the main concern of all communications.
I had one who sent me the booking details of another client in the plaintext part. I reported it to them nearly a year ago and they didn't reply, so screw anonymity, it was Avis.
I'm sorry, I did not properly read and comprehend your original post. I thought you were saying "they put sensitive details in the text/plain part", implying that those details somehow only belonged in the text/html part. What you actually said was "they put somebody else's sensitive details in the text/plain part".
So I'm wondering a bit here - I've seen an implementation where emails to send only have html versions, but as part of the sending process the html is run through a Lynx browser process with the -dump command to get the plain text, which is included as the text/plain part of the email.
Is there actual value to this? e.g. Is the output of Lynx's text dump better for plain-text email clients than whatever they'd display for html emails?
I've personally converted html to plaintext with beautifulsoup in python, and used that as the plaintext version. Did not have complaints, but I honestly don't know who actually reads the non-html version.
Epic Games tells me I don't support HTML formatted email.
Many emails are just the HTML version with HTML tags removed, leaving behind a bunch of ridiculously long image and link urls and little to no text telling you what they are. You might have better luck with a partial HTML implementation (pull out the title, alt, src, href attributes) than disabling it entirely.
I dont think people fully realise how novel Stripe actually is.
The idea of capturing payments via an API itself isn't unique, but the fact that as a company Stripe is actually on average pretty competent. I've never had an issue with stripe on the integration side because that just works.
I work with a massive variety of payment gateways, I'd imagine more than most people on this site ever will, 90% of them suck. The big ones all suck, Stripe's "competitors" suck. They all have effectively the same API but the issues are organisational.
Stripe is a technology company, its technology is good, its approach to the end user (Technical people implementing it) is good. The rest are finance companies and you get all the slow to move, impossible to get through to bureaucracy that comes with it. Of course they didn't respond to the actually issue and said "You've already solved it" and then ignored the actual problem. These payment providers sell their services to business not developers or anyone with technical chops.
They wont care about these issues because your boss has already been sold on the product, the issues are very very unlikely to change their mind and it is now your problem to solve.
> The rest are finance companies and you get all the slow to move, impossible to get through to bureaucracy that comes with it.
Look, I 100% get what you're saying here, but it's not (entirely) the other companies fault. The financial sector is incredibly regulated, and they're required to have policies and procedures for everything which leads to a terrible, no good, sub par user experience.
Stripe has (so far) managed to avoid this, but if (when) they end up in the regulators cross-hairs they too will become like many other financial companies. I mean, I want to be wrong on this but I don't think I am.
This is why a sane e-commerce platform should have routing through a bunch of different payment processors so they're not vulnerable to the regulator or regulatory officer anomalous behavior at any particular one.
How do you think Stripe had avoided the regulators crosshairs? Probably because the users are satisfied and not complaining. Which brings us back full circle to GP's comment on why users are happy.
I mean, Stripe are pretty young for a financial company. Assuming they keep growing, they'll eventually end up as the focus of regulation.
They do seem to be a very well run company, but if they end up as a bank, then they'd be globally systemic and at that point they'll be a focus for regulatory scrutiny
With fintech that surprises me not the slightest bit. Financial institutions are filled to the brim with unbelievably incompetent people. A large part of it is probably willful ignorance, too. It's often truly staggering that a financial company I interact with in day to day live is even able to exist. That's until I remember that all the others are just as incompetent.
"Major European Payment Processor" really just translates to "Major European Incompetence Center".
With a broad statement like this, I would usually just suggest this is inflammatory and surely overstated.
However, I've also worked at a financial institution which used core systems by Harland Financial Systems. Their "encryption" for data in transit from teller workstations to the core system was just a two byte XOR, and they sent the key at the beginning of the connection!
Was so unbelievable to be able to crack this in under a half-hour after noticing patterns in a PCAP. Wouldn't have believed it if I hadn't seen it with my own eyes.
That fraud was good enough for our regulators and theirs, so I have no doubt the industry is filled with rotten incompetence through and through.
The biggest disappointment in my 30 years of adulting has been how much absolute, shameless incompetence is out there in the workforce. When I was a kid, I naively thought that adults were smart and knew what they are doing. Then I got into industry and saw so many people just outright bluffing for 8 hours a day before going home, day in and day out.
I think that's actually an interesting feature of society as a macro system. It is very fault tolerant, which is frustrating for any power user but without which the system as a whole would not function at all.
Or without which the system would function much better because all poorly functioning systems would fail, forcing designers to focus on fixing the faults
There is plenty amount of incompetence in FAAMG. Notepad ....
Do Europe financial institutions have the same level of corruption as the USA? Such as a credit card company authorizing credit card transactions with incorrect expiration date to maximum profit, Bank of America? Or opening new accounts without consumer consent, Wells Fargo?
It's a broad question, but in many ways, very clearly yes, re. corruption of financial institutions, and to a far greater extent in many, albeit different ways.
Incompetence and corruption only slightly overlap in most cases, i.e., being competent at corruption is a very real thing. The incompetently corrupt, usually end up punished... and there are few and far between...as we all very well know.
The kind of schemes you mentioned are generally not going to be how "corruption" will manifest itself in European financial institutions, because although it is also difficult to speak in general across Europe since the EU has not yet subsumed democratic self-determination all across the continent yet, so there is wide variation; the competent corruption is largely in the form of money laundering and tax evasion, not lower level quantitative schemes that would quickly come to light because Europeans are also a lot more cognizant of money and value than Americans, so people are paying attention a lot more closely and will raise hell over a single cent, where Americans are known to have hundreds of dollars draining out of their pockets every month just alone on recurring payments for things they don't even use anymore and don't bother dealing with it.
What we all don't really seem to internalize as a human species, is the absolutely demonic type of pernicious nature of "banking", i.e., a kind of LotR, ring, that consumes you especially if you are weak... and human, or at least European civilization seems to frequently go through periods of immense weakness where things are going springily and everyone is dancing to the music the "bankers" are playing as they are pandering our pockets, and when they realize they could get away with that and all the pockets are plundered, they move on to plundering our homes, then our accounts, then they want to take our first born... "Banking" is like humanity's cocaine, the seemingly innocuous, feel good drug that will consume your soul if you do not rage and fight against that demon taking over aggressively. It's no coincidence that cocaine is so widely used by the most parasitic elements of European societies, especially in "banking"/finance in general.
it differs by country (EU gives countries some similar overlapping laws, but they still are distinct countries with a lot of different practices and different laws in many many ways.)
and it differs by severity
E.g. deutsche bank has become notorious to appear in most large scale bank scandals _in the US_ as a "German Representative". They also repeatedly get into trouble for relations to money laundering operations and other issues. At the same time they are the Bank which has to give you a (expensive but not completely absurd priced) bank account no matter your credit score, criminal history etc. and is the bank you mostly likely have a escrow account parking money if the person who gets it "if you mess up" is the state/local government (like e.g. in some long term visa context).
In general banks move very slow, including with software protocol updates/changes. And that isn't just the case in the EU AFIK. Actually some of the most crazy cases of "old software" I have seen in "the west" where as far as I remember in the UK and the issue was that it was deeply fundamentally impossible for them handle a lot of non English names correctly, like in 2017~2020 or so. Through at least they still could use the bank, just with a crippled/en-ified name. Funnily the same but worse (as in not working at all) I have heard about Japan banks, where they had(have?) really big issues if you have a middle name.).
I mean, since 2008 finance has been officially privatised profits, socialised losses. It's inevitable that an industry people are forced to use and cannot fail becomes this way.
I have some level of sympathy with Google here, which isn’t something I often say.
I recently switched from Gmail to Fastmail and by and large I’m happy with it. But I’ve been surprised by the amount of spam and (particularly) phishing emails I get in a regular basis. Google might be too strict in its filtering but it does serve a legitimate purpose.
Message-Id being basically required for _automated_ mails (very similar mail send to a lot of people) requiring Message-Id is a de-facto industry standard. Sure some providers don't care and some might make it just more likely that your mail ends up in spam. But this could have happened with pretty much any mail provider widely used by companies.
much more funny (/s) is if you start out in a startup and still use the default template for password reset/on-baording links of a widely used system (e.g. Keycloak) and it turns out multiple larger but "cheap" phishing campaigns did the same and now MS/Google and other suspect you are running a phishing operation
or when you use a local data center and can't send mails to MS/Outlook anymore because it turns out someone did some legal questionable things on them and MS wanted the personal information to be handed over _without court order or any ongoing legal proceeding against this people_ and they didn't hand it over (partially because they legally aren't allowed to...) and MS decided to retaliate by permanently blacklist the IPv4 range of that data center(s) which just happen to locally compete with Azure while self hosted mails competes with Outlooks...
Interesting that you mention this, as I also switched to Fastmail recently and got more spam than before, but after marking it as spam for some time it now died down I think. This may also be a symptom of changing providers, where the previous provider knew the kind of spam I tended to get from past years, while Fastmail needed some time to get up to speed.
Fingers crossed that the experience will be the same for you.
Long time Fastmail customer. Let it learn a bit about your inbox. I barely see any spam these days that’s not been caught. Except LinkedIn, that somehow goes through
Fastmail seems to go through periods where they're a little slower to adjust to new spam techniques, and they do rely on users filtering somewhat. About twice a year a few will slip through, but if I report them as spam they soon stop.
I've been a happy customer otherwise for years, for what its worth.
I don't think a single spam email has ever crossed into my Fastmail inbox. Granted, every service I sign up for gets it's own masked email. But while the @fastmail.nl email that I chuck on my website gets a fair amount of spam, it always gets categorized correctly.
Just to add another data point - I recently switched as well and have seen no uptick in spam at all. Could just be my emailing habits though? Not sure.
I switched from Fastmail to Gmail/Workspace a year ago. I think but cannot conclusively prove that Gmail drops Apple transaction emails on occasion ( like receipts ). But I also think Fastmail dropped other emails too.
The first thing that comes to my mind is: how come viva.com is unable to send emails to google workspaces and nobody at viva.com noticed before ? For how long has this going on ?
The second thing is, what email software are they using ? If it was any relatively used software I would not expect this problem to arise (maybe it is some commond software but misconfigured).
Third, while the header is not mandatory, I usually read SHOULD as a "if you don't implement it prepare for possible problems". SHOULD is not MAY.
Fourth, they should be thankful that Google bounced the messages with some appropriate error explaining how to solve it. I have plenty of issues in the past with both Google and Microsoft where they accept the message for then sending to /dev/null
I don't think either are. The payment processor should be sending it, but, at least according to the RFC, it is incorrect to reject an email that doesn't have it. I suspect the reason it is SHOULD, and not MUST is for backwards compatibility with software that predates the RFC that adds the message-id header.
Maybe there is a correlation between missing that header and being spam, but then it should go to the spam folder, not be outright rejected.
----------------------------
The experience with support is also similar to experiences I've had with support at many companies. I provide enough details that an engineer could probably easily fix the problem, but the support representative just dismisses it, and it is doubtful an engineer even hears about it.
Exactly. This minutiae is all so weird. Email as a formal specification does not work, and the industry as a whole has accepted that for decades now. It's not possible to filter spam from valid traffic without applying a truckload of heuristics and leveraging an ever growing set of auxiliary signals (SPF, DKIM, yada yada).
To wit: basically everything in this world is a "SHOULD", at best. The rules are a conversation.
If you have a client app that you think is reliably filtering spam, then to be blunt: you aren't receiving any spam, at least to first approximation. My stack of stuff on my three decade old personal account has a 95%+ hit rate (something I might naively be tempted to label as "reliably filtering spam"), and I see see more spam than signal in the inbox.
GMail, on the other hand, is damn close to 100%. And it does it through excruciating application of heuristics like "don't trust agents that don't set SHOULD headers".
They're variants of the same software stack. But that's sort of the thing: email filtering (the only thing that makes email useful) requires tuning and heuristics. Decisions that are effective for personal email may not be for the spam faced by corporate clients, and vice versa.
No one (at least no one serious) thinks you can just implement some RFCs and get a product that works and doesn't suck. Life just isn't like that.
I'm just speculating, but probably because you're on an email provider that isn't a big enough target to worry about the persistent threat-actor model.
Google is a big enough target to justify spending the resources on dedicated attacks against their infra. Other providers may simply get less spam because their email domain shows up less often in the sources attackers use to pick targets.
It's not necessarily the support person's fault. I worked in support way back when. There were some long-standing issues that needed only minor work to fix that engineering management didn't get, didn't care about or just wouldn't prioritize. Engineers weren't free to work on what wasn't prioritized. Acknowledging a problem and leaving the ticket open doomed to be unsolved forever negatively affected my performance rating. I had to respond with this sort of cheery everything's fine message and write it off as a customer issue, even though I didn't believe it.
They are testing it, every time someone signs up and it fails. We don't know that this wasn't something that changed on Google's side, so IMO it's a bigger indictment that no one is monitoring their live email deliverability
Maybe the whole thing was intentional, right at the footer of viva "Cloud services by Microsoft Azure" ; #1 I've never heard of viva before #2 I've never seen an azure logo at the footer of a website.
If I were to test an email delivery system, I would test Gmail. I probably wouldn't test Google Workspaces, because I'd (wrongly) assume that they work the same.
No, just over 6 million paying business customers.
But hey, if you're in a business domain where categorically leaving 6 million potential clients-who-are-demonstrated-to-spend-on-things isn't an issue? One fewer thing to worry about, right? ;)
"Email is tough", software development is tough, IT is tough, walking and talking at the same time is tough, mailing a letter is tough.
When orgs frame problems like this, it erodes trust in the message they try to convey. Email isn't a tough problem, but its a problem nobody wants to really deal with. Email is simple - its a text based protocol, that started out open, but now you need to add security to ensure your email is delivered.
The Gmail requirement is actually slightly different: the header must be present and unique. Gmail only keeps one copy of a message per user and message ID. Combined with a mail source that uses predictable message IDs (such as Github), you can abuse this to suppress delivery of certain messages to Gmail users.
They mean to send an email in advance, with a message ID that would later be used in the target email. First email gets ignored, moved to spam, or not read yet. Then the target email gets sent with the predicable message ID, and gets bounced.
Comments on issues use the format
<[OrgName]/[RepoName]/issues/[IssueNumber]/[CommentID]@github.com>
A mitigation to this would be to take the combination of message ID and the sending domain and use that as the unique value, because message ID is not guaranteed to actually contain a domain label that's owned by the sender.
For example SendGrid's message IDs are <[RandomValue]@geopod-ismtpd-[Integer]>.
> The reason Message-ID is SHOULD rather than MUST? Mail clients
> sometimes send messages without one to their submission server, which
> adds it on their behalf. As for why Google enforces it anyway:
> spam. Messages with minor RFC violations are far more likely to be
> spam, so rejecting them is a reasonable heuristic. In practice, Google
> and Microsoft have become the de-facto standards bodies for email —
> what the RFCs say matters less than what their servers accept.
Surely the problem is on Google's end? And a metaproblem is that we are allowing corporations to change or ignore standards for critical infrastructure?
The email landed in the spam folder. A bounced email means it didn't find the inbox. If it didn't find an inbox there would be no log for him to check. Technical knowledge of emails and what the terms mean out him instantly as a liar. The fact this is still up on the front page is an embarrassment for the tech community in my opinion.
Dude, you keep on just believing a blog post even though it makes no technical sense. That's why none of the hardcore nerds are even approaching this and why Viva are going to sue.
An email bouncing means it was rejected because it did not find an inbox or a recipient. You AI to question if that is true. So if the email service provider says we don't know who that belongs to, why would they provide logs.
And what email service provider is rejecting an email over a RFC thing when millions of emails a day because most devs don't care about the RFCs and only an idiot would have two different mail infrastructure and code versions running for gmail and workspaces. Especially on the sending and receiving part.
So are you going to apologies for being confidently wrong? Because I'm technically and confidently right. Again, which is why the hardcore nerds are leaving this the fuck alone because they saw me wreck the post in minutes of it being posted.
Typically I'm a DIY type who loves tinkering and building...
HOWEVER, I have learned the hard way to never apply that spirit to email.
In Europe you see this stuff all the time with old school "IT" (what old industrial companies call tech) people balking at the prices of commercial API-based senders and email marketing ESPs.
"Money to send emails in the cloud? HAH! Back at Siemens in 90s we were running millions of emails out of our servers just fine!"
Nobody understands that deliverability has gotten immensely harder these days, and trying to DIY it if its not your core business is just plain stupid. I would never in a million years try to roll my own email, it's nightmarish legacy cruft and footguns all the way down, in everything from IP/Domain Rep to something as simple as the HTML in the email templates themselves.
Microsoft Outlook and Gmail have the last word on everything in email, and their defacto duopoly (over B2B and consumer email respectively) means you play by the rules they set in 2008 and are too lazy to change or you don't get delivered. The protocol of email exists separately from the world of the actual inbox providers, which are locked down to insane degree given the security/spam concerns with email.
Google is at least less arbitrary than Microsoft. Microsoft will decide an email is spam today, and tomorrow the exact same email is perfectly fine. I think Google relies more and more on sending IP and domain reputation rather than content.
Deliverability to Microsoft famously took a dive a bit over a year ago due to random arbitrary failures within their infrastructure causing DMARC/DKIM problems which they clearly were having problems diagnosing.
Even with a six-figure email spend and weeks of troubleshooting the best response we could get from our mail provider was that they were having problems getting traction with Microsoft on the issue.
Worth mentioning is that there are several email umbrellas under Microsoft... including the newer office/365, the slightly older outlook.com hosting, the old corp hosting and hotmail and sub-properties... each with different rules and services to determine spam in inconsistent ways between them.
One of my main emails is still on a "free" outlook.com hosted with a personal domain that I never shifted to paid 365. I've also got an MTA server (mailu) of my own that I've been testing with... my own email under outlook.com is literally the only one of the MS systems I can't seem to deliver to, the rest work fine. Same for google.com for that matter... kinda wild.
This issue didn't seem to discriminate. I was seeing deliverability failures to Office 365 clients as well as consumer-facing brands like hotmail and msn
Okay... I was just adding my own experience as I was able to deliver to some without issue and not the one. It's entirely possible this has changed in the past couple years since I was actively testing.
Yeah, they're defining 'major' fairly generously, here, I think. Difficult to find figures, but seems to be transaction volumes of tens of billions a year. So not actually particularly big.
something the RFC standard recommend but doesn't require
but it being required is a de-facto industry standard for sending automated mails
and is clearly documented by support sides of large mail providers (like Google)
the mail standards only defines what parts you can put together, but widely fail to define how this parts can be interpreted, what are sensible combinations, etc.
and they don't cover spam/suspicious mail detection at all
so you can't just go by RFC, you need to read up on what all larger mail providers have as additional requirements (which mostly are the same, and Message-Id being the most common dominator) and then hope that another provider you didn't read up one doesn't have some other surprising rule (which doesn't tent to be the case if you don't do anything surprising, but it sucks anyway).
Message-Id being required for automated mails is a de-facto industry standard
while the consequences differ between mail provider, it missing will also make it much more likely for mail to be reject or put into the spam folder
It's also well known. Pretty much viva engineers fucked up doing proper research.
Now to be fair:
- it sucks that you can't just implement the RFC(s)
- the standards suck, docent of different RFCs overlapping and replacing each other and referencing often older versions of other RFCs, with docents of ways to do the same things of which only some can be used reliable in practice and a common gaps in the standards about edge cases or about the "higher level semantics" of constructs.
- so overall mail seems very simple at first but if you want to automated send mails reliable internationally it's a total pain and Message-Id is just the head of the iceberg.
I've often found when receiving a clueless support response like this, it can be effective to just follow up with a polite request to forward the ticket to an engineer or developer. Usually the front-line csr simply hasn't understood the issue. In this case I would say something like,
"Yes, I managed to work around the issue by switching to my personal email address, but this bug is still preventing me from using my work email domain. If you could please forward the error log I included to a developer, it should help them resolve the issue. Thank you."
10 percent of the effort in building software compatibility with open source file specifications is dealing with knowing the specifications. 90 percent of the effort is dealing with errors in generated files by less worthy software programs.
The RSS spec is one way. RSS readers do a fine job of interpreting files done the right way. Publishers don’t always do a good job with publishing error free RSS files. So RSS readers devs have to anticipate all sorts of errors and conduct error handling to ensure RSS items are properly handled.
This is why companies want to keep their file format proprietary. Other devs can really do damage to the ecosystem and ruin the experience
My personal fork of ttrss, from 2005, is a dodgy patchwork of fixes for badly formatted RSS. I can't imagine trying to host a service that deals with RSS feeds from random sites at scale.
One that always irks me to no end, is every time I see someone ham fisting csv handling by hand instead of using an established, well-sourced library. They almost always fail at commas or newlines in quoted text... It's one of the more annoying things.
Currently working on replacing a couple decades old system, and my csv output is using a library that isn't quoting all the strings that don't require quotes... so I'm forced to do it (for compatibility) with the other system this csv is going to. (sigh).
> This experience fits a pattern I keep running into with European business-facing APIs and services. Something is always a little bit broken. Documentation is incomplete, or packaged as a nasty PDF, edge cases are unhandled, error messages are misleading, and when you report issues, the support team doesn't have the technical depth to understand what you're telling them.
I can definitely confirm that this is a common thing. But I think this is a "small org"-problem more than a "European business"-problem. Apparently, the company has somewhere between 500 and 1000 employees (I couldn't find good data, sadly). With a size like this, the "support" is probably outsourced (meaning they don't know anything), there are maybe 100 engineers (probably less) and the mailing is either done via a third-party or set up by an Admin that left three years ago.
Without any basis, I will speculate that you will notice this more in Europe because there is simply no company at the size of Stripe or similar.
While there is some truth in while you saying, I have to say that from my European perspective all the big American companies feel enormously bloated. For example, I've recently learned thats Atlassian has 13000 employees, and I have to ask myself, what do all those people do?
> This experience fits a pattern I keep running into with European business-facing APIs and services
honestly, it's a pattern I have been running into with many start ups, fintech, banks and some other places _no matter where_.
It also often makes sense. For many large orgs (e.g. Banks) this APIs are often a side business, sometimes one they don't want but have to have for compliance (or market pressure). But for strip it's their live blood.
And many startups (like actual start ups, not 200 man companies running by investor money) often simply don't have the resources to prioritize a "very nice to use API".
Lastly API design which is both nice to use and stable for existing integrations is surprisingly hard to get right (if you don't have some senior engine prioritizing it very highly and forcing it being kept prioritized; Or a surprisingly "clean"/"clear" use case.).
> For viva.com's engineering team, in case this reaches you: add a Message-ID header to your outgoing transactional emails.
Don't know what they're using for sending emails, but that's something that should be handled by their email service provider, unless they're hosting their own email servers.
Interestingly the MX record of via.com points to Google, but their verification emails could come from anywhere of course. The IP address in the log is also a Google IP, although that could also be the receiving IP.
message-id is (or at least was ~5-10 years ago) only required for automated mails, i.e. if you send a mail which looks mostly the same to a lot of users
so if you (ab-)use protocols for mail clients to send automated mails they might very well not add Message-Id. In general many mail providers have both a way to send a mail where "they do some clever parts" (like adding Message-Id) and interfaces where they mostly just send what you give them. It's possible that they migrated from one solution which did automatically add Message-Id to one using interfaces where it doesn't happen without realizing that there is a mismatch in this solutions do implicitly add.
I went through this hell last year when trying to tell my customers I had to change payment processors.
In my case it turns out I was victim of Google having a beef with afraid.org years ago (the DNS authoritative record for my domain). 100% of my emails to gmail were going to spam, as soon as I switched NS record to fastmail (with same zone file) I hit the inbox.
I did not show up on any of the major blacklists with mxtoolbox, this took months to figure out and I felt like I was going insane with hardly anyone responding to my emails.
If a business like that doesn't get its emails delivered, it will slowly go out of business. Merchants will find another processor that is able to deliver emails to every inbox. That is, Google could be less picky, but the company with a problem at hand is Viva.
Thank you for the recommendation! That I couldn't sign up using a form and I had to "talk to their team" was a turn-off for my (extremely extroverted) self.
It used to be said that the reason the Internet evolved so well was because of the basic principle of "be strict when you send, but tolerant when you receive". Clearly Google has forgotten this.
That worked when the Internet participants were mostly benign. Nowadays you have to account for abusive participants (spammers, malware writers, etc., etc.).
Considering that Google Workspace is a major player and a place where your customer are, do you want your customers to succesfully use your product?
If the answer is no then yes go ahead and debate the RFC.
If you care about your customers and your product working broadly with a diverse set of mail services then please spend 15 minutes to write a patch to add a Message-ID header?
> For viva.com's engineering team, in case this reaches you: [...]
That's too kind of you, but on the other hand it really doesn't solve the issue of bad priorities and lack of overall Quality. Some engineer might log a couple hours fixing a Level 3 severity bug, emails will start working better, but the poor (or at the least, dubious) backwards technical stewardship (or lack of it) will keep going on inside the company, unnoticed from outside (until something bad eventually happens to some client)
Veeery interesting. I have a personal domain that forwards to a @gmail.com account. There are several companies I interact with, from banks to retail to just about anything, that send mail to my personal domain that never arrives in my Gmail account, but I can see on the hosted mailserver. For those companies I have to use my @gmail.com address instead for the mail to get through.
Maybe there are more companies than we think that send malformed emails?
It's amazing how long this has been on the front page. Are people upvoting as a "hit" against the EU wanting out of Visa/MasterCard? It's certainly interesting timing
edit: It seems Viva primarily is a neobank, not a payment processor. I'd never even heard of it until today. It's tiny compared to Revolut, Monzo etc. And tiny compared to an actual payment processor eg Worldline (French, 18,000 employees)
I vaguely remember hitting this message id issue in Google Workspace, and being able to work around it in mail routing configuration.
Saidly I don't remember the specifics, it was something along the lines of not all, but only specific routing features requiring it. Workspace settings are a moving target anyway, so the behavior probably changed more than once since.
I'm not saying it's a good idea to send emails without message id, but i'd also double-check that workspace configuration.
A quick search for their developer support revealed they accept submissions via GitHub issues for their API. Perhaps try there? https://developer.viva.com/get-support/
Sometimes you have to get creative to reach out to a company's engineering department...
Mamiride dot com email verifications are not delivered to Gmail from a self-hosted mail server and I wonder if this is the reason. We got around this by making email verification an optional step instead of mandatory.
I hope OP took more effort in making Viva understand the problem than the obvious zero effort it took writing this post, given it is completely AI generated.
> Their support team's response to my detailed bug report: "your account has a verified email, so there's no problem."
Sadly I doubt their system is xkcd806 compatible ether.
This isn't an engineering problem, it's an ITIL problem. To be fair 99% of these complaints will be dealt with by the flow chart. Sadly people on the front line are either not knowledgable enough or not empowered enough to bust out of that straightjacket.
I usually reply with snark dialed up about 50% asking if anyone had actually read the original message. And include detail about how it is not, in fact, "ok"
The other day, I literally had trouble signing into a website... then I tried filling the contact us form, about the bug... only to have that fail... call in, have the person on the other end schedule my appointment, then almost drop the call without actually logging my bug report/complaint about the whole issue that had me calling in the first place.
Sudden realization that one of my American banks must be having email problems with this too because I use a Google custom email and recently got an in-app notification from my bank saying "we're unable to email you" (and a letter) yet my email works perfectly fine... switching to consumer gmail worked.
Do you actually not receive their emails? Fidelity uses tracker dots to check email receipt, which drives me nuts, because like any sane person I don't allow emails to load images without damn good reason. My brokerage should not be sending me cute dog photos, thus I have no need of their images.
So they send me an email, send me another email saying they can't reach me by email, then mail me a letter with the same content as the original letter, and mail me an additional letter saying they can't reach me by email.
For some reason the announcement they sent me didn't show up in my email, which was odd. I've had other bank notifications come through unopened before, even after that announcement.
> This experience fits a pattern I keep running into with European business-facing APIs and services. Something is always a little bit broken.
I feel like this isn't just business services though.
American engineers are used to working for either big tech or "Silicon Valley inc." European engineers are used to working for Volkswagen, Ikea or Ryanair. Very different kinds of businesses who treat tech very differently.
Over here, competing on user experience and attracting users with a slick interface that people love to use isn't really something most companies think about (and so they get their lunch eaten by the Americans).
Nowhere is the European mentality more evident than in cybersecurity, where outdated beliefs still dominate. In this mentality, everybody is out to get you (and that notably incudes your vendors, your business partners and your customers), so all infrastructure has to be on prem, open source is free and hence suspicious by definition, obscurity is the best kind of security, encryption doesn't work so data should go over custom fiber, and if you have to expose an API on the public internet, an Authorization header isn't enough, it should also require MTLS behind a layer of IpSec.
And we're still getting passwords changed periodically or requiring a number, upper case letter, symbol...
I'm an European engineer and I can confirm that our tech is often broken and customer-reachable people are usually obtuse and hostile about it. We don't even seem to properly implement our own legal requirements. Sometimes, Americans implement the RGPD better than we do.
I offered to host a friends business email on my DO instance.
Works 99% of the time but every now and then emails just disappear only to find out that MS and Apple block DO IP addresses, sometimes.
Silently.
There is a war on small email providers it seems.
The biggest war on small providers is waged by other small providers. They can be ancient and outdated or simply extremely picky. Which makes everything Google or others require a piece of cake, which it actually kinda is.
Gripe only related to email in general: what annoys me to no end is that if my boss forwards me an email and asks me to reply to it (to everybody in the original email) then I have to type in or copy+paste all the addresses from the Fwd attachment (using Fastmail, but this problem exists everywhere). Instead, there should be a button to make that easy.
There's actually nothing that prevents that, if you craft the right headers you can reply to a thread you were not included in, and have it show up as a reply in the thread of common clients (tested Gmail and Outlook).
We added this feature at my $dayjob and I was quite surprised there is no authentication. But thinking about it, this is how mailing lists work (you aren't explicitly specified in "To:") so it makes sense you can do this.
I’m sorry but in the context of a 50 year old technology like email, 2008 was yesterday. Gmail is in the wrong, you don’t get to just update the standard for email like it’s TikTok content or a Roblox update or whatever.
Email was here long before Gmail and will be here long after Google abandons it.
>"We can see your account now has a verified email address, so there doesn't appear to be an issue."
There are still too many edge cases like this one that can't get fixed because of ignorant support not doing it's job. In my life, every company that escalates to an engineer instead of punting the ticket with some asinine 'but it works right now, goodbye' message gets rewarded via keeping my business. The ones that don't are immediately cancelled. Sometimes I even do a chargeback as extra punishment. Maybe I'm just old, but I have near zero tolerance for immature support playing games with my time.
Email deliverability is the reason I gave up on email entirely for my side project and built on Telegram instead. Setting up SPF, DKIM, DMARC, warming up a domain, monitoring reputation, dealing with bounces and complaints... all of that just to maybe land in someone's inbox.
With Telegram you send a message via the Bot API and it arrives. 100% deliverability. No spam filters. No authentication chain. The message just shows up with a notification on their phone.
Obviously Telegram has its own limitations (smaller user base in the US, less formal). But for anything where you need reliable message delivery to people who opted in, messaging platforms have a massive advantage over email in 2026.
Unless you are running a more complex setup SPF, DKIM and DMARC really aren't that complicated. They are annoying and additional checkboxes you have to go trough that are hard to fully automate because they require access to DNS, but they are more busywork than difficult.
Domain and IP reputation and all the other quirks of deliverability are much more of a headache. DMARC is setup, test and done. But deliverability in praxis is something you cannot just test and can break at any time. The second worst are email providers that do whitelisting for email and require you to go through their process to even be allowed to send emails to their customers. The worst are providers that randomly decide to drop your emails without informing you or giving you a proper way to appeal as a small sender. If you're not a large email provider they have no problem just dropping you and the fault is on you because your service is the only one that is not working.
And then there are so many more intricacies of the ancient email protocol. Compatibility with horrendously outdated and misconfigured mail infrastructure is particular frustrating to me. I'm running a modern, properly configured, state of the art email server, but get ghosted by large email providers, yet have to deal with the broken mess, much bigger providers than myself are, which of course have no trouble delivering their broken spoofable email just because they are large enough.
Until the platform owner bans for whatever reason and if communication by way of the platform was your only means of communication with your customer base, that's the platform owner having the power to destroy your business. No different that businesses that rely on the neverending goodwill of the mobile app store owners. One misstep and your business is gone with no recourse whatsoever. Protocols > Platforms. Always.
If Gmail rejects emails from your domain it is up to you to fix it. Google is not going to change, and enough of your users will be interacting with people on Gmail that you have to fix it. It doesn't help that Google has been pushing people away from running their own email and into Google's services by ever tightening what it accepts over the years. More than one person has given up on their email server because it was a constant battle with Google, Microsoft, and company to not have important emails disappear into the void.
My takeaway is there is no bug. My takeaway is that his test email bounced because he didn't have the reputation Viva does. Emails are handled on a reputation basis, this is why we use email service providers like Sendgrid, Mailgun, Postmark, etc.
It always amazes me how people can read a blog post like this one that has a clear description of the problem with a log excerpts demonstrating the problem, and then people will confidently make up a completely different scenario that was not mentioned at all and blame the problem on that.
WTF you talking about? Rene, this is defamation and I'm probably going to take action because honestly, enough is enough. I'm fed up of folk like you who lack basic technical knowledge or any knowledge making up bullshit. Your hourly rate makes me like you have money to take.
It amazes me people read that in this community and don't know for an email to bounce it means it didn't find an inbox. If it didn't find an inbox how did he check the logs?
TFA shows an excerpt from the email log for his google workspace account, showing the bounce of email sent from viva.com.
Then, TFA states that he switched "the account" (his viva.com account) from using his GWorkspace address to a personal @gmail.com address, and asked viva to send another verification email. That one arrived.
At no point does TFA describe the author themselves sending a test email.
I think that's a misunderstanding of the tale. Viva sent a "click here to verify your email" to OP. That email never arrived because Google rejected it for missing a header. OP tried to tell viva, but they don't wanna hear it because OP worked around it.
Yeah. I think email receiving is a game of exceptions… the email receivers (In the business world it’s essentially just MSFT and GOOG of course) answer to the addressees because they are the customer, and those customers will start to shriek if their inbox doesn’t receive “Important Messages.” But GOOG or MS have no leverage over the senders in this case so they just add an exception: “if IP range is just right and message fault ___ is present, fix message” (or otherwise allow)
Of course, they do have leverage over “marketing email” senders since they can block it and no one will complain, so those senders always have impeccable compliance with every year’s new “anti-spam standard.”
Apple is another major player in the email receiving game for consumers. And they are awful, by far the worst of all the big providers. They do not send dmarc reports and they make it very difficult to tell why they accept some email and not others.
If you read two paragraphs further than the Tl;Dr:
> To unblock myself, I switched to a personal @gmail.com address for the account. Gmail's own receiving infrastructure is apparently more lenient with messages, or perhaps routes them differently. The verification email came through.
> 1. Email didn’t arrive in his inbox of his Google workspace
> 2. He checked workspace email logs (with admin you can do this on gsuite)
But it didn't arrive, so how was it in his email logs!?!?!?! Are you tech literate? Isn't this the second time you've been asked or was I rate limited at that point?
It does seem unlikely that there are no customers on google workspace who have tried to use viva. I don't do payment processing, and my email is via zoho, so I've no idea how large either of those groups are.
I suspect that Google going out of their way to make this required had a very reasonable and thought-out process, while the sender's omission was on oversight, so I haven't contacted Google Workspace support.
What's truly iffy is that GMail doesn't have the same strict requirements, and there's no way (at least that I found) to turn it off for my Google Workspace domain.
> You don't have the right to complain to random websites without punishment. And nobody punished you. And "ability to redress" is something you still have. And there are no damages.
Well, you see Dang aka Daniel edited the settings on my account to stop me replying, or that's what I've heard, and pointing out the absolute lie that the blog post is. And that did hamper my ability to redress and people have been lying about me so there are damages. But even if it's just rate limiting that hampers my ability especially since it's been optimised heavily for high traffic so there is no technical reason for it since it's a file-based datastore.
> You brought it up. Though it's pretty obvious you're a liar.
I said someone there should know. I didn't say everyone should know. I clearly made it quite clear it was insider knowledge. This is what is known as a flex. I want to tell you my tagline because it's awesome but you can just call me a keyboard warrior.
I thought the problem was inability to edit, now it's inability to reply? Also you're wrong about the post and you don't know what damages are and your flex failed real bad. Good luck in life.
And nobody ever has to let you post on their private website. Right to redress is unrelated. If you take away anything from this conversation, please let it be that.
The bigger issue here is that Europe depends way too much on the USA in so many areas. This is not good - you can be constantly blackmailed when you have people such as Trump in charge. I don't think the EU can be fixed, but at the same time I also think the less Europeans depend on outside factors (in particular the USA) the better. Canada kind of showed how to do it. Granted, Canada is also dependent on the USA in numerous ways and most of this is hard to fix (most Canadians live in the south aka close to the USA and trade is primarily done via the USA; security has also been largely outsourced onto the USA and so forth). The sooner people in Canada and Europe get moving away towards more independence from the USA, the better. And more cooperation would not harm either.
The definition of SHOULD is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."
I suspect viva.com didn't consider the full implications, and I suspect Google did some hard math on hours saved for their customers
Postel’s Law would put the onus on Google to be forgiving in what it receives. Unsure how you could safely use a sender-created Message-Id for anything anyway.
Following Postel's law results in the normalisation and proliferation of defective implementations. The actual standard becomes irrelevant, and new implementations have to be coded against the defective ones.
My opinion is that Postel's law should be approached in the same way that Linus Torvalds did CVS when designing Git. If in doubt about an implementation decision, consider what Postel's law would recommend, and then do the exact opposite.
Yep. And even a world of perfect good faith, "forgiving in what you receive" has both costs and scaling problems - from researching what "spec" you'll need to design to, to customer service when the added complexity and permissiveness cause interesting stuff to happen.
> Viva.com's outgoing verification emails lack a Message-ID header, a requirement that has been part of the Internet Message Format specification (RFC 5322) since 2008
> ...
> `Message-ID` is one of the most basic required headers in email.
Section 3.6. of the RFC in question (https://www.rfc-editor.org/rfc/rfc5322.html) says:
and in section 3.6.4: That says SHOULD, not MUST, so how is it a requirement?SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either.
IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.
> SHOULD is a requirement.
I once had a job where reading standards documents was my bread and butter.
SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL.
My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justified it. Before our code could be released, everything was audited by a 3rd party auditor.
It's totally fine to ignore SHOULD.
Maybe the standards documents you are used to differ from RFCs, but here is the official language:
SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.I just don't understand how you get from the text you pasted to "required". Nowhere does it say that anything is effectively required. Words have meaning.
> the full implications must be understood and carefully weighed before choosing a different course.
In this case, the full implication is that your email might be undeliverable. "Should" indicates that the consequences for this fall on the entity that is deviating from the thing they "should" be doing.
But the RFC language clearly anticipates there are situations and good reasons leading to a message that does not include a message-id. Google therefore would be rejecting RFC-compliant emails, and they are the ones who have to justify themselves.
Theoretically, anyway, I expect in practice they'll just ignore the issue or have their own good reason. But they should accept emails with no message-id; there it does strain the imagination to see why lacking an ID would make a message unreadable or undeliverable.
There are indeed such situations. Two situations AFAICR, and neither of them apply to when you connect to someone else's MX.
Gmail rejects the vast majority of compliant messages, I think they've stated in public that they reject >99.9% of messages, and hearsay has it that the percentage for with minor errors like this is even higher.
There are good reasons why a message might be unreadable. For example, message-id is often used by the threading algorithms in MUAs and IMAP servers, and many don't test whether their threading code handles ID-less messages. I use one that deduplicates by ID, what do you think it does when the ID is empty or missing? I don't know, I haven't tested and I'm not going to.
if you apply that logic every recommendation becomes mandatory no?
if the alternative to [not doing a recommended thing] is [failure] then what's the difference between "should" and "must"?
"we recommend you leave the keys in the ignition while putting your car in park"
"or what?"
"or your car may blow up, killing everyone inside. just a recommendation though!"
It makes more sense when you read all of the definitions together https://datatracker.ietf.org/doc/html/rfc2119:
- `MUST: This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification.`
- `SHOULD: This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.`
- `MAY: This word, or the adjective "OPTIONAL", mean that an item is truly optional.`
In practical terms:
- MUST: It's always a failure to do this. E.g. you MUST have some form of stored energy in your car for it to propel itself down the highway.
- SHOULD: If you don't do this, it's likely to cause failures unless you really know the situation is one of great exception and have thought about what else this change may affect. E.g. you SHOULD maintain a large distance between yourself and then next vehicle on the highway (an example of an exceptional case might be a standstill backup on the highway).
- MAY: This is something which is actually optional and has negligible impacts to successful operation if you do/don't. E.g. you MAY activate cruise control instead of always manually operating the accelerator.
For your car example as-is it'd probably best be MUST unless there is expectation one might reasonably consider their car exploding a valid scenario. In the real world where the car doesn't actually blow, it'd probably be that you MAY leave your keys in the ignition rather than SHOULD/MUST.
ah! thank you for that excerpt. that does change the meaning a bit in my opinion, yeah
Is that what the spec says? Or is this something that Google decided, by making an optional feature a requirement when interoperating with their systems?
It is something Google decided. SHOULD means the other party should anticipate may not. The party examining a SHOULD and deciding not to do something is obviously not required to consider incompetence of other RFC readers as a reason to global replace SHOULD with SHALL before examining requirements.
You should wear sunscreen to the beach. Its recommended as a good way to prevent sunburn. However, the beach police aren't going to come get you for not wearing it. You just might get a sunburn if you don't plan accordingly with other sun countermeasures.
So if I send an email that lacks a feature that MUST be there, will the email police come get me? At a certain point, looking for an analogy stops making sense I think.
It sounds like Google won't deliver your mails. Presumably they read the spec and are aware of the consequences.
"there may exist valid reasons in particular circumstances" means effectively required in my common sense understanding of American English.
[flagged]
No, its not a "required"... It means someone may have reasons not to use something, and so spec implementors need to allow for circumstances where it is not present.
Those reasons can be anything. Legal, practical, technological, ideaological. You don't know. All you know is not using it is explicitly permitted.
> You don't know. All you know is not using it is explicitly permitted.
In theory, if they are truly following the specification, you know they thought hard about all the consequences.
I think the pushback in the comments comes from the commonsense feeling that this... didn't happen here.
"permitted" is a pretty empty word in the given context. Because dropping such emails is equally "permitted". Sure, there will be no arrests made, but there will be consequences. And those are what this article is about.
If that's your line, then I am equally permitted to send random binary blobs along the way. Not a crime, so totally permitted. They'll just drop the connection.
Buuut I don't think that is at all relevant to the discussion at hand.
I don't even know how you got to "used twice" tbh. Both your own comment AND the post you quoted from only have a single "must".
The only thing that text demands is understanding and carefully weighing the implications. If, having done that, you conclude that you don't want to then there is absolutely nothing in the spec stopping you. Maybe the spec would have been better off putting more stuff in SHALL and less in SHOULD, but as written that is definitely not the case.
Nope, it's exactly what it says: RECOMMENDED.
Any time any document (standards or otherwise) says something is recommended, then of course you should think it through before going against the recommendation. Going from their verbiage to:
> SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.
is a fairly big leap.
"The full implications must be understood and carefully weighed before choosing a different course." (emphasis mine)
i.e. you are required to have a good reason not to do it.
Can you start actually reading what you quote? "The implications must be understood and weighed" does not mean "you are required to have a good reason not to do it". I can know of the Message-ID field, understand what it's used for, and carefully weigh the possible lack of interoperability against the effort required, concluding that I'm too lazy to do it right now. I have then understood and carefully weighed the implications before choosing a different course, but I don't have a good reason to.
Words mean things, especially in standard texts. You can't just carelessly rewrite sentences using not-quite-synonyms like you're doing here.
Oh, and that "must" is not a "MUST". Had the text said "The full implications MUST be understood ..." it would have been a proper requirement by the standard, but this lower-case "must" is just normal part of prose and not the magical "MUST" word which formally imposes a requirement.
This very clearly says that SHOULD is not effectively REQUIRED at all, and is fact nothing more than RECOMMENDED. Really not sure how you misinterpreted this so badly
You should read it again.
Are you recommending they read it again or requiring it?
Well, at least someone understood.
It's not required but they need to understand the implications. In this case the implication is that Google drops the mail. So clearly they didn't understand the implications.
Yes, but this might be googles fault for not respecting/missinterpretating the spec.
Google probably did parse these messages as well-formed before inspecting them and deciding to drop them based on the lack of this field. The RFC imposes no mandatory obligation to deliver messages just because they are well-formed.
So you agree that calling it a requirement is wrong?
required means it must exist, not that it may or may not exist depending on the reason
Assuming we’re talking about RFC 2119, it’s important not to collapse the distinction between SHOULD and MAY, which is there for a reason. MAY elements are legitimately optional, SHOULD elements are there for a reason and are disregarded at one’s own risk.
> SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
To validly disregard a SHOULD, you need to (a) fully understand the implications, and (b) accept them.
Any time someone disregards a SHOULD and then complains about the result, they are necessarily in the wrong. Either they didn’t fully understand the implications, or they don’t actually accept them.
Agreed. We use RFC language at work for documents, and whenever I see someone using the word SHOULD, I always ask what is the MAY where you don't need to do the thing? If they can't think of one, then make it a MUST. SHOULD always means there are valid reasons to not do it.
FYI slightly over 10% of RFCs contain at least one occurrence of SHALL. Even that small minority uses other words for most requirements.
Email is about standards like browsers were about standards in 2017...
Browsers are still heavily standardized. Processes and organizations have evolved, but they still develop plenty of specifications and standards.
Could was yeah nice to have. Should, was yeah the system shall have it, but it's ok if not. Shall : ffs we will hunt you and kill you if you don't implement it.
Yes, except there seems to be a move on the best words from SHALL to MUST and from SHOULD to MAY. IANAL but I recall reading this in e.g. legal language guidance sites.
RFC language is expmicltly defined in 2119[0]. Any other interpretation is incorrect.
[0] https://www.rfc-editor.org/rfc/rfc2119
Thank you for that. So should is optional, people!
Pulling exact quotes out, SHOULD means "there may exist valid reasons in particular circumstances to ignore a particular item" while MAY means "an item is truly optional."
I don't think this can be interpreted as simply "should is optional".
They're both optional, but one comes attached to a big warning message.
Well… yes… no… SHOULD means that you must either do the thing or understand the consequences of not doing it. That's not simply optional, the two options are to ① spend time on writing code and ② spend time on learning the consequences. Either way you need to think hard and spend time. And that's why the definition of SHOULD includes the word "must".
> So should is optional, people!
Only once you have satisfied:
> but the full implications must be understood and carefully weighed before choosing a different course.
In other words, you better have a damn good reason for deciding not to do it.
And (possibly even more to the point), you must decide not to do it, rather than simply throwing code at the wall until most emails make it through unscathed.
I think that is a bit to easy. MAY is described ar optional.
SHOULD - Should really be there. It's not MUST, you can ignore it but do not come crying if your email is not delivered to some of your customers ! you should have though about that before.
> "I don't want to" is not a valid excuse
for the client. If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.
You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.
> isn't a valid excuse to reject a client either.
Yes it absolutely is: https://www.rfc-editor.org/rfc/rfc2119 is quite clear.
If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.If the server has considered fully the implications of not having a Message-ID header, then it MAY continue processing.
In general, you will find most of the Internet specifications are labelled MUST if they are required for the protocol's own state-processing (i.e. as documented), while specifications are labelled SHOULD if they are required for application state-processing in some circumstances (i.e. other users of the protocol).
> If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.
That is not a rule.
In this situation the server can reject any message if it wants to, and not doing a SHOULD tests the server's patience, but it's still ultimately in the "server wanted to" category, not the "RFC was violated" category.
[flagged]
You are confused about what I'm doing. I'm not telling anyone what to do. I'm saying what category their actions fall into.
And the line of yours I quoted is still not supported by anything.
> You are confused about what I'm doing.
Absolutely.
And if you're not confused by what I said, that's not obvious in the slightest.
> I'm not telling anyone what to do.
So you say.
> I'm saying what category their actions fall into.
I think that's pretty weird that you think you get to decide that all by yourself.
But I'm not playing, because you're right: I don't know why you would be doing that.
> And the line of yours I quoted is still not supported by anything.
Yes it is. It's a description of the behaviour of other Internet hosts, and it's a description of exactly what is happening in the linked article.
> But I'm not playing, because you're right: I don't know why you would be doing that.
This is the comment you originally replied to:
>> If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.
>> You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.
They're talking about how to categorize actions, just like I am.
So I thought you were playing on that same topic.
But if you weren't on that topic, and given that you quoted only the first one of those sentences, I have a guess.
I think you didn't realize how the second sentence affects the meaning of the first one, and you misunderstood what they were saying as trying to tell servers they can't reject. They were not trying to tell servers they can't reject.
If that's the case, then this whole line of conversation is pointless, because you were rebutting an argument that nobody made.
> It's a description of the behaviour of other Internet hosts, and it's a description of exactly what is happening in the linked article.
Taking a description of behavior and throwing an RFC-style "SHOULD" in front is only going to be correct if you get lucky. "is" and "SHOULD" are different things!
That clearly means it’s not required.
How does Google know whether or not the sender has a valid reason? They cannot know that so for them to reject an email for it means they would reject emails that have valid reasons as well.
How would the sender know the consequences of sending without the header? You shouldn’t assume anything here. As a sender, you should include it unless you’ve already worked out what the recipient is expecting or how it will be handled. Doing this with email is silly because the client is sennding to so many different servers they know nothing about so it’s basically a requirement to include it.
> That clearly means it’s not required.
You and I have different definitions of "clearly"
It is not required for the protocol of one SMTP client sending one message to one SMTP server, but it is required for many Internet Mail applications to function properly.
This one for example, is where if you want to send an email to some sites, you are going to need a Message-ID, so you SHOULD add one if you're the originating mail site.
> How does Google know whether or not the sender has a valid reason?
If the Sender has a valid reason, they would have responded to the RFC (Request For Comments) telling implementers what they SHOULD do, rather than do their own thing and hope for the best!
Google knows the meaning of the word SHOULD.
> it means they would reject emails that have valid reasons as well.
No shit! They reject spam for example. And there's more than a few RFC's about that. Here's one about spam that specifically talks about using Message-ID:
https://datatracker.ietf.org/doc/html/rfc2635
> If the server has considered fully the implications
The server "considers" nothing. The considerations are for the human implementers to make when building their software. And they can never presume to know why the software on the other side is working a certain way. Only that the RFC didn't make something mandatory.
The rejection isn't to be compliant with the RFC, it's a choice made by the server implementers.
Either the server must explicitly confirm to servers or the clients must accept everything. Otherwise message delivery is not guaranteed. In the context of an email protocol, this often is a silent failure which causes real-world problems.
I don’t care what the protocol rfc says, the client arbitrarily rejecting an email from the server for some missing unimportant header (for deduction detection?) is silly.
If it was unimportant it would be MAY.
Is the server somehow unable to inject an ID if the sender did not send one? Stop hiding behind policy and think for yourself.
> Is the server somehow unable to inject an ID if the sender did not send one?
Yes. https://www.rfc-editor.org/rfc/rfc2821#section-6.3 refers to servers that do this and says very clearly:
That's Google in this situation.> Stop hiding behind policy and think for yourself.
Sometimes you should think for yourself, but sometimes, and friend let me tell you this is one of those times, you should take some time to read all of the things that other people have thought about a subject, especially when that subject is as big and old as email.
There is no good reason viva couldn't make a Message-ID, but there's a good reason to believe they can't handle delivery status notifications, and if they can't do that, they are causing bigger problems than just this.
You want me to think for myself when writing an email server that interoperates with other email servers? Are you just clueless?
You are describing MAY.
“MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional… An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)”
Note how it explicitly calls out interoperation with implementations that do or do not implement MAY. As a exception that proves the rule, we can reasonably assume that not interoperating with a system ignoring a SHOULD rule is a correct implementation and it is the fault of whoever is not implementing SHOULD.
Hearsay has it that the reason is spam. Spam messages are said to have massively higher chances of minor RFC violations when they arrive at the destination server.
Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.
How is not having a message-id a security risk? It seems that Gmail is being pedantic for no reason
Because in practice it showed up for a period of time as a common thing in spam-senders. They were trying to maximize throughput and minimize software maintenance costs, so they leave out things that the spec says are optional. But that makes "a commonly-implemented optional thing was left out" into a stronger spam signal.
Is it still a strong spam signal? Hard to say. Sources disagree. But as with laws, heuristics, once added, are often sticky.
> How is not having a message-id a security risk?
CVE classify a lot of things that have nothing to do with security.
Not having a Message-ID can cause problems for loop-detection (especially on busy netnews and mailing lists), and with reliable delivery status notification.
Dealing with these things for clients who can't read the RFC wastes memory and time which can potentially deny legitimate users access to services
> It seems that Gmail is being pedantic for no reason
Now you know that feeling is just ignorance.
Well, gmail does not manage usenet groups and mailing lists. Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
> Well, [google] does not manage usenet groups and mailing lists.
They do. Sort of.
Google used to nntp, and manages the largest usenet archive; They still have one of the largest mailing list servers in the world, and they still perform distribution on those lists via SMTP email.
They still have all of the problems associated with it, as do lots of other mail/news/list sites still do that are a fraction of Google's size.
> Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Sure it does.
You consider them best-effort, but that doesn't follow that I should consider them best-effort. For a simple example: Consider spam.
In any event, if you keep sending me the same message without any evidence you can handle them, I'm not going to accept your messages either, because I don't know what else you aren't doing. That's part of the subtext of "SHOULD".
Most big sites take this policy because it is internal nodes that will generate the delivery notification, but the edge nodes that are tasked with preventing loops. If the edge node adds a Message-ID based on the content, it'll waste CPU and possibly deny service; If the edge node naively adds a Message-ID like an MSA, the origin won't recognise it, and forwarded messages can loop or (if sent to a mailing list) be amplified. There also are other specific documented requirements related to Internet Mail that edge nodes not do this (e.g. RFC2821 § 6.3).
However you seem to be assuming Google is blocking messages "for this case" which is a little presumptuous. Google is presumably trying to save themselves a headache of handling errors for people who aren't prepared to do anything about it, the most common of which is spam. And the use of Message-ID in this application is documented at least as early as RFC2635.
> Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Without knowing what Google does with the g-msgid header, you are making a mistake to assume it is equivalent to the Message-ID header just because it has a similar name. You have no reason to believe this is true.
> Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
I am sorry you are offended to not know things, but you do not know this thing, and your characterising my actions will make it very difficult for you to learn something new and not be so ignorant in the future.
Think hard exactly about what you want to happen here: Do you want Google (et al) to do something different? Do you want me to agree Google should? Who exactly are you trying to convince of what, and to what end?
I am trying to tell you how to interpret the documentation of the Internet, in this case to be successful sending email. That's it.
I am not likely to try and tell Google what to do in this case because of my own experiences running mail servers over the last 30 years, but listen: I am willing to be convinced. That's all I can do.
If it's something else, I'm sorry I just don't understand.
> If it's something else, I'm sorry I just don't understand.
I'm trying to explain to you that you're speaking very authoritatively about how things were done 30 years ago but things have changed since then, Gmail won't even send a message non-delivery notification to non-DKIM hosts.
Same thing with the g-msgid I'm telling you about – google documented it explicitly as a unique identifier, see here for example: https://developers.google.com/workspace/gmail/imap/imap-exte...
So yeah, things changed.
[flagged]
sic transit gloria mundi..
So add a message id at the first stop, or hard ban the sender server version until they confirm. A midway point that involves a doom switch is not a good option.
> So add a message id at the first stop
That should have already happened. Google is not the "first stop".
> hard ban the sender server version until they confirm
SMTP clients do not announce their version.
Also I don't work for you, stop telling me what to do.
> A midway point that involves a doom switch is not a good option.
No shit. That's almost certainly a big part of why Google blocks messages from being transited without a Message-ID.
As someone who does systems engineering, the only valid requirements include the word "shall".
As someone else who does System Engineering, when dealing with ancient protocols, "shall" is extremely difficult barrier to get over since there is always ancient stuff out there and there might be cases not to do it, esp if it's internal communication.
"SHOULD" is basically, if you control both sides of conversation, you can decide if it's required looking at your requirements. If you are talking between systems where you don't control both sides of conversation, you should do all "SHOULD" requirements with fail back in cases where other side won't understand you. If for reason you don't do "SHOULD" requirement, reason should be a blog article that people understand.
For example, "SHOULD" requirement would be "all deployable artifacts SHOULD be packaged in OCI container". There are cases where "SHOULD" doesn't work but those are well documented.
> … when dealing with ancient protocols
I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers.
The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software).
This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.
In a completely different field, navigating ships at sea, the Collision Regulations which define how people must conduct ships at sea, they use the words "Shall" and "May" to differentiate legal requirements and what may just be best practice. "Should" intuitively means something more like "May" to me
Happily, the meanings in RFCs are clearly specified, see https://www.rfc-editor.org/rfc/rfc2119.
Note "the full implications must be understood and carefully weighed before choosing a different course". Gmail and the other big hosters have full-time spam teams who spend a lot of time weighing implications, so I assume the implications of this was weighed.
I feel like "SHOULD" and "SHOULD NOT" are redundant. You end up having to assume someone else treated them as a "MAY". If you control all the endpoints in a private implementation you can just deviate from the standard & not implement a MUST, it's your private implementation. There's thus no difference in public implementations between "SHOULD" and "MAY", and no difference in internal implementations between any of the words. They are therefore redundant, requirements are either mandatory or optional, there's no middle.
And EVERY rfc has a paragraph talking about rfc 2119 in the preamble.
I guess that's why nobody reads it. /s
"shall" and "must"
The original email RFC is also completely unaware of how bad spam is. Sure it might mention it but it's not really AWARE of the problem. The truth is, companies like Google, Microsoft and a few others have de-facto adjusted the minimum requirements for an email. Signing, anti-spam-agreements, etc.. are the true standard if you want an email to get from point a to b. (none of which are going to be REQUIRED in the RFC)
Given the amount of time that has passed, the big email companies have had many years to jointly update the RFC according to their experienced reality. They haven't done so and the way you describe how they effectively work together to adjust the minimum requirements (outside the RFC) is de-facto use of market power to the exclusion of other players.
That's not what the word "should" means, though.
"Should" is a lot closer to "better do it this way" than "you must do it this way". While "must" implies a mandatory-ness, "should" does not.
Or take it from perplexity:
"Must" normally expresses strong obligation/necessity: something is required, with little or no choice.
"Should" is softer and usually expresses recommendation, expectation, or what is right/appropriate, not a strict requirement.
It isn't a requirement. SHOULD is conditional. MUST is _not_ conditional.
Sure, you can argue, if you require that the email reach their destination, it is required to set this. ;-)
But I am totally with the OP here. SHOULD was never a requirement, just a recommendation that is maybe better to follow.
>It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case.
But that means a valid reason could exist and Google would block those mails too.
Should = internal target
Must = external requirement
I cannot fathom how you think should* would act as a requirement in any sense of the world.
If you read RFC 2119, you'll see that the definition of SHOULD includes the word must. Perhaps it's easier to understand if I rephrase it as: "If you understand the issue and its consequences, SHOULD means optional. If you don't, SHOULD is equivalent to MUST."
Message-ID has maybe half a dozen types of problem. If you know all of them, leaving it out is up to you.
SHOULD is not MUST
The official definition of SHOULD per RFC2119:
Not sure how the people at Google interpreted this about the message-idYou can argue that you not obligated to use message-id but if you don't use it you should blame only yourself that your messages are not accepted. In requiring message-id I would side with google (though in general I think they anti-spam is too aggressive and lacks ways to report false positives). Full RFC compliance (as in not only MUST but also SHOULD unless you have a very good reason) is the easiest part of making sure your emails will be delivered.
> if you don't use it you should blame only yourself that your messages are not accepted
I think it's a gray area
- If the receiver declines your message because "Message-id" is required - then I blame the receiver; because that's not true
- If the receiver declines your message because "most systems do include it, and it's lack of presence is highly correlated with spam email", then it's on the sender
Admittedly, the end result is the same.
I think it's the latter. But, in either case, you're right in that you get the same result.
Now, let's assume that if it is the latter (it's spam related), and Google were to accept the message, but then internally bin the message, it would be worse. At least in this case, they are bouncing the message. Because of this, the sender is at least aware that the message wasn't delivered.
Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.
My takeaway is that I think that Google is doing the least-wrong thing. And by being explicit in how they are handling it, it at least made the debugging for the author possible.
Also note: in a quick reading of RFC5321 (SMTP), rejecting messages for "policy reasons" is an acceptable outcome. I'm not sure if it applies completely here. The author should probably also be taking into account RFC5321 (SMTP) instead of just 5322 (message format).
> Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.
That's the annoying part to me.
An email is an email. By applying different rules for rejection on different mailboxes, gmail creates a system where it's harder for would-be implementers to test compliance.
If tomorrow gmail creates a new type of mailbox, will there be a third set of rules to have your message delivered?
There are dozens of spam and security settings that admins can change in the Google Workspace console, presumably because different businesses have different requirements. So in practice, there's not just two sets of rules in gmail -- there's probably thousands or millions (however many combinations of settings are actually in use).
Other anti-spam implementations also punish the lack of Message-ID. There are tools online that highlight this as an issue.
This here is a trivial case of simply not testing deliverability at all.
In my experience, email is an unreliable way to communicate any time-bounded critical information. When I want to be sure an email was transmitted on either side, the only reliable way to ensure this is to use a distinct channel to validate reception and confirm content.
That is, when some hotline tell me that they just sent and email with the information, I ensure they hold the line until I got the actual email and checked it delivers the relevant information to fulfill the intended process. And when I want to make sure an email was received, I call the person and ask to check, waiting until confirmation.
It’s not that much SMTP/IMAP per se as the whole ecosystem. People can legitimately get fatigue of "is it in my junk directory", "it might be relayed but only after the overloaded spam/junk analyzer accept it", or whatever can go wrong in the MUA, MSA, MTA, MX, MDA chain. And of course people can simply lie, and pretend the email was sent/received when they couldn’t bother less with the actual delivery status.
There are of course many cases where emails is fantastic.
Email is an unreliable way to communicate any information, in the strictest sense of the word "reliable." The protocol does not guarantee that any email will be delivered, nor does it guarantee that failure will be detected. It's a good-faith effort. The bits could drop on the floor at any point and you might never know.
Does it even matter when in reality it's more likely that this is intentional anti-competitive behavior by Google?
They once made all emails from my very reputable small German email provider (a company that has existed and provided email services long before Google existed) go into a black whole - not bounce them back or anything like that, mind you, their servers accepted them and made them disappear forever. I was in contact with the technicians then to get the problem fixed and they told me it's very difficult for them to even reach anyone at Google. It took them several days to get the problem fixed.
Of course, no one will ever be able to prove an intention behind these kind of "technical glitches." Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it, so certainly nothing will happen when small issues with interoperability occur and drive more people to Gmail.
At scale, it's very hard to distinguish malicious intent from the simple consequence of being the largest operator in a space so any motion one makes makes waves.
For what it's worth: having seen some of how the sausage is made, Google isn't particularly interested in screwing over a small reputable German provider. But they also aren't particularly interested in supporting such a provider's desire to route messages to their users because the provider is small. At their scale, "I've forgotten how to count that low" is a real effect. And email, as a protocol, has been pretty busted for decades; it's not Google that made the protocol so open-ended and messy to implement in a way that providers will agree is correct.
> Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it
Nothing of significance outside Google. Inside, Google initiated a technical lift that turned their intranet into an untrusted-by-default ecosystem so that data was encrypted on the fiber (as well as between machines within a datacenter, to head off future compromised-employee attacks). That process took at least five years; I suppose there's a scenario where it was all smoke and mirrors, but being on the inside in the middle of the process, I watched several C-suite who are not particularly good actors be bloody pissed at the US government for putting itself into Google's "threat actor" box and making that much work for the system engineering teams.
Also, an engineer at Google then made an end-to-end email crypto plugin for Chrome, including a flag that was a nod-and-middle-finger to the information revealed in the Snowden documents. https://techcrunch.com/2014/06/04/nsa-mocking-easter-egg-fou...
Thanks for this reply full of interesting information! These kind of comments are what I like about HN.
What does “not particularly good actors” mean in this context? Actors as in theater? Or “Bad actors” plotting evil things?
Actors in a theater. I have no reason to believe they were pretending to be upset because the were actually spies and knew what was going on before Snowden's reveals came out. They were as surprised as everyone else; the NSA wiretapping was done off-prem in cable that was privately owned (but stretched hundreds of miles and was, therefore, practically undefendable against compromise).
Long time ago when I was managing ISP email relay and customers asked "Where is the message I've sent?" seeing in the logs message accepted by receiving SMTP server was the end of the debug for me: I just handed the customer the part of the log and suggested talking to the receiving side IT administrator.
On the other hand, by erroneously treating a SHOULD as a MUST, I would say that Google is the one who's not RFC-compliant
Google is rejecting it to ensure incoming messages aren't spam. SHOULD means "you should do this unless you have a really, really good reason not to." Do they have a good reason not to? It doesn't seem so, meaning Viva is in the wrong here.
No, SHOULD is defined in the RFC, not by colloquial usage. Google is on the wrong, regardless of their "safety" intent.
After all, linguistics is full with examples of words that are spelled the same, but have different meaning in different cultures. I'm glad the RFC spelled it out it for everyone.
The RFC says a SHOULD is to be treated like a MUST, but well-justified exceptions are allowed.
RFC speak requires you to think for a while about skipping a SHOULD. It doesn't require strong justification.
When producing a message, it SHOULD have the id. With or withot it is compliant.
On the other end, we may receive messages with or without. Both are valid. We MUST therefore accept both variations.
The second one is a consequence of the former. So yes Google is the violating party.
No it doesnt lmao. It's quoted all over this thread and clearly is not in any way like a MUST
if Google's choices are protecting users, they can't be in the wrong. That's the reality of a shared communications infrastructure regardless of what the docs say.
When the docs disagree with the reality of threat-actor behavior, reality has to win because reality can't be fooled.
Spam senders don’t have pseudorandom number generators?
They're more likely to put in the least amount of effort or care the least about the reasons how the header is used later on.
Did i miss the part of the RFC that says google must accept every message? Pretty sure the RFC allows email providers to reject any message they feel like.
RFC cannot force a mail server to accept spam. You may argue that requiring message-id is a bad anti-spam policy but it does reduce amount of spam. In my observations around a half or messages without message-id are spam. I would not use personally this as the only reason to reject a message but I understand why someone may choose to do.
The RFC says a SHOULD is to be treated like a MUST, but well-justified exceptions are allowed.
Per RFC2119: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
So, it's fairly explicit that the sender should use message-id unless there's a good reason to not do so. The spec is quiet about the recipients behavior (unless there's another spec that calls it out).
Not a specification but "Be liberal in what you accept?" comes to mind. (which I always personally hated but i'm just one shoveler).
Postel's law was a precept of the Internet of the 80's and 90's, when due to the primitive software engineering practices at that time, implementations couldn't be tested properly. That lead to many cases of poor interoperability, and it's no longer a good idea: for example, when HTML 5 was designed, they decide to put into the spec how to deal with the frequent errors like mismatched closing tags, etc... because all major implementations were "liberal" in what they accepted, but each in a different way.
What is the point of SHOULD then?
(No seriously, I’m asking; are there examples of where it’s actually different from a MUST)?
Also this reminds me of something I read somewhere a long time ago: when specifying requirements don’t bother with SHOULD. Either you want it or you don’t. Because if it’s not a requirement, some people won’t implement it.
I guess the one time it’s good is if you want an optional feature or are moving towards requiring it. In this case Google has decided it’s not an optional feature.
Typically, MUST means that if you don't do that then something will break at the protocol level.
SHOULD means that if you don't that, bad things are likely to happen, but it will not immediately break at the protocol level and during discussion in the IETF some people thought there could be valid reasons to violate the SHOULD.
Typically, IETF standards track RFCs consider the immediate effects of the protocol but often do not consider operational reality very well.
Sometimes operational reality is that a MUST gets violated because the standard is just wrong. Sometimes a SHOULD becomes required, etc.
Certainly for email, there is a lot you have to do to get your email accepted that is not spelled out in the RFCs.
SHOULD generally means: some people might require it. implement it for best results
backward compatibility makes it hard to add MUST. using SHOULD is a good alternative
"SHOULD generally means: some people might require it."
No it absolutely does not mean that. It means, by explicit definition which is right here, that text is exactly that definition, that no one requires it. They can't require it, and still be conforming to the spec or rfc. That's the entire point of that text is to define that and remove all ambiguity about it.
It's not required by anyone.
The reason it's there at all, and has "should" is that it's useful and helpful and good to include, all else being equal.
But by the very definition itself, no people require it. No people are allowed to require it.
Any that do, are simply violating the spec.
the spec doesn't include an obligation to deliver your mail. it parses fine, and is rejected by the larger system
> the full implications must be understood and carefully weighed
MUST means omission is unacceptable. SHOULD means MUST unless you have a good, well-reasoned excuse.
Incorrect. Not required is not required. You do not need to supply rationale or get agreement by anyone else that your reasons are good in their opinion and not just in your opinion.
Should just means the thing is preferred. It's something that is good and useful and helpful to do.
That is not "must unless you can convince me that you should be excused".
RFC2119
“ SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.”
I don't know if you thought you were validating or invalidating my argument, but this says exactly what I said.
Further, not only the presense of the words "should" and the doubling down with "recommended", also the absense of the words "must" or "required" or "shall" is pointed.
Because those words are used elsewhere. When something is actually required, those words are used, and so when they are not used, it is meaningful. It means no one is free to put their own words into the authors mouths and say "well they didn't say it but obviously they pretty much meant required" when they so pointedly chose not to say required.
And it's not a technicality. A thing either is or is not required. There are no other states. The spec does not require it, and so for Google to require it, Google is not adhering to the spec.
Since the spec does not require it, you can't say either that it falls under "full implications". "full implications" cannot include "Some servers might require it", because that would be circular logic. No server can require it in the first place.
I SHOULD have 8 hours of sleep every night. It's RECOMMENDED. However, there are times where it's best I don't (e.g. because of work, or travel, or needing to take someone to the hospital, etc.). It's definitely not that I MUST sleep 8 hours every night.
Yeah, and sometimes if you pull an all nighter bad things happen as a result of lack of sleep. So it is an apt metaphor.
“When jump getting over a wall, you SHOULD use three points of contact.”
For most cases you should use three points of contact. However, there may be other situations for example if someone is giving you a leg up, or you can pole vault, where another solution is preferred.
You assume that internet standards are prescriptivist; that the document describes how it is to be implemented. In practice it's often descriptivist, with the standards documents playing catch-up with how things are actually going in practice.
Anyway, in general you can expect that doing unusual but technically valid things with email headers will very often get your messages rejected or filtered as spam.
Standards are definitely prescriptive. But just like a medical prescription, it doesn’t ensure that actors in the wild will conform to what’s prescribed. People will not follow prescriptions for whatever reason, willingly or otherwise. It doesn’t mean the document wasn’t prescriptive.
For producers, ignoring a SHOULD is riskier because it shifts the burden to every consumer.
For consumers, ignoring a SHOULD mostly affects their own robustness.
But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.
> maybe the scale of spam is enough to justify it.
This is 100 percent the case, and why these things are this way.
If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
> This is 100 percent the case, and why these things are this way.
But gmail accepts emails without message-id on personal mailboxes apparently.
I think a mail 2.0 would be notify and pull based.... you notify a recipient's mail server that there's a message from <address> for them, then that server connects to the MX of record for the domain of <address> and retrieves <message-id> message.
Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.
Is the idea that you could decide from the envelope whether you want to even bother fetching the message? Besides that I'm not sure I see the advantage
You have to have a working mail server attached to a domain to be able to send mail... that's the big part. Right now, email can more or less come to anywhere from anywhere as anyone. There are extensions for signing connections, tls, etc... but in general SMTP at it's core is pretty open and there have been efforts to close this.
It would simply close the loop and push the burden of the messages onto the sender's system mostly.
And yes, you can decide from the envelope, and a higher chance of envelope validity.
Like it proves you have the ability to receive mail at the domain you're sending from? I feel like SPF/DKIM already does this
https://jmap.io
jmap is the communication between a mail client and shared directory/mail services on a server. It does not include server to server communications (that I am aware of) for sending mail to other users/servers.
Couldn’t resist replying to:
> If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
Google interpreted it that way because it drives more people to use gmail.
My take, as a postmaster for hosting company, who don't have any sympathy to gmail (that should be visible from my comments history): Message-ID is absolutely MUST in production e-mails. You can send your test stuff without it, but real messages always have it. Not having Message-ID's causes lot of fun things. All somewhat competent software is capable to add Message-ID's, so lack of it is good indication of poorly made custom (usually spamming) solution.
Rspamd and spamassassin have missing MID check in their default rules, I am sure that most antispam software is same.
Your casual use of the word MUST is not the same as a standard document's use of the word MUST. Your real world experience is entirely irrelevant to the conversation about what the standard requires.
Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. One lesson surely most everyone learns in CS is that unique identifiers are maybe unique to the system generating them, but to rely on foreign generated IDs being unique globally is a terrible idea that will break within the minute.
So at that point the ID has no value to me except being obliged to carry it around with the message, so maybe the originating system can at some point make sense of it. But then there is obviously no reason to ever reject mail without it, it's an ID valid for the sender and the sender didn't care to include one, great, we save on storage.
>Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. [...] So at that point the ID has no value to me
Your framework of analysis is based on someone else's database key ids being irrelevant to you. That's true.
But another framework of analysis is tracking statistical correlations of what spam looks like. Lots of spam often don't have message ids. Therefore it's used as a heuristic in scoring it as potential spam. That's why other postmasters even without SpamAssassin independently arrive at the same answer of trying to block messages without a message id. Example: https://serverfault.com/questions/629923/blocking-messages-w...
Ah, so it's just the evil bit, or lack of.
MID-s are used by MUA-s for referring earlier messages, tracking answers and so on. So any software expecting dialog (messages coming back) needs to deal with MID-s correctly. Missing MID-s show that said communication is one direction, because broken dialog has not been problem.
Exactly. Message-ID is not required.
An unrelated frustration of mine is that Message-ID really should not be overridden but SES for instance throws away your Message-ID and replaces it with another one :(
It is de-facto required and has been for many years.
Should in most RFCs also mean "do it as long as you don't have a very good technical reason not to do it". Like it's most times a "weak must". And in that case the only reason it isn't must is for backward compatibility with older mail system not used for sending automated mails.
And it is documented if you read any larger mail providers docs about "what to do that my automated mails don't get misclassified as spam". And spam rejection is a whole additional non-standardized layer on top of RFCs anyone working with mail should be aware of. In any decades old non centralized communication system without ever green standards having other "industry standard/de-factor" but not "standardized" requirements is pretty normal btw.
What's de-facto required and not is completely irrelevant here. The claim was that "Message-ID header [is a] a requirement that has been part of the Internet Message Format specification ... since 2008" (emphasis mine), which is false; it's a recommendation that has been part of the Internet Message Format specification since 2008
I would read this as a requirement for email to be 'legit' and not classified as spam.
Sure, you can send email with whatever headers you want, use weird combos, IP addresses, reply-to, and it might be still a technically valid email, but not something that should land in people's inboxes.
Also, a payment processor not testing their email on the most popular email provider in the world is quite ridiculous.
As indicated in the RFC, it uses another RFC[0] to define those words. Here's the relevant excerpt from that one:
[0] https://www.rfc-editor.org/rfc/rfc2119yeah, but that RFC isn't the only relevant document
Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known.
and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn.
In addition there are some de-facto standards not pinned down in any RFC, like e.g.:
- Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely).
- You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work...
- You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble.
- Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC.
- A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it.
so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice
As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly.
oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).
I think the question for the sender is: do I have a good enough reason to reject the SHOULD requirement? If not, then I should implement it.
I know you're looking for "pedant points" but the specification generally take a backseat to implementation. If Message-ID is expected out here where the rubber meets the road, then you are the squeaky wheel in this scenario for not including it.
> the specification generally take a backseat to implementation.
And we should be raising hell for it. Should never happen. Using your popularity to violate protocol should be not be tolerated
Per the RFC. "should" means "you better have a good reason for not implementing {thing}".
The only messages I receive without one are spam/phishing. I check because they're not recognised by notmuch, so I don't see them otherwise.
The reason that European tech sucks is that people in Europe are open to such arguments. If an engineer in the US started talking about SHOULD vs MUST, some PM would just give them that "what the fuck did I just listen to" face, spend the next few minutes gently trying to convince them that the customer experience matters more than the spec, and if they fail, escalate and get the decision they want.
For example, why does Google handle this differently for consumer and enterprise accounts? Well it's Google so the answer could always just be "they are disorganized" but there's a good chance that in both cases, it was the pragmatic choice given the slightly different priorities of these types of customers.
Not my PM (in the US). My PM would try to avoid anything that is not absolutely necessary and therefore ask developers not to develop anything that isn't a MUST. I know that we like making fun of Europe for their alleged lack of innovation but this isn't a Europe thing.
Implementing a 20 years old needed RFC header is the cutting edge of innovation
Your PM most definitely would not tell you to skip a feature that is needed for your emails to be delivered to Gmail accounts. What a preposterous thing to lie about.
The PM wouldn't know about that gmail behavior at that point in the development cycle.
Well the current US Administration would agree - the law doesn't matter, we need to be "pragmatic" and do what we think is right. Rules be damned.
Once you deviate a bit from the standard, you're down a slippery slope. Its not that difficult to use pragmatism to justify wrongdoing.
Do bugs and bad implementations not exist in US software? If an US company did this, nobody would be bloviating about how it is a cultural issue or whatever.
> some PM would just give them that "what the fuck did I just listen to" face
Some people have become way too comfortable taking for granted that it is okay to treat others in an uncivil fashion. To those I say: keep it up, and we shall revert to the fundamentals of society where violence is an option and one day you are copping punches in the face, or getting shackled and thrown in a ditch somewhere for the ravens. Behave like an animal, get treated like one.
Standards are important, and the meaning assigned to words are important, but if your very important email does go through, because Google thinks you're wrong, you add the bloody message-id. I really do agree, I don't care about the linguistic/legal/standard/technological reason as to why Google might be wrong, if I can't deliver email to Gsuite customers, I add the message-id.
You're precisely right that customer experience matter, but I wouldn't put it past some conservative European company to go: Well Google is wrong, so they should fix that. Google doesn't care, you can't make them care, you can't even contact them. Just make it work for your customer.
Seriously, I hope that none of the posters upthread arguing about SHOULD v/s MUST in some standards body document are in charge of real world money making software.
Google and Microsoft's email practices define a pseudo-RFC in practice. As an engineer, I hate this. As a civic participant I can vote against it. But as a person that sells my software services for a living, I am going to implement the Google/Microsoft standards to the letter, not argue about definitions in an RFC.
[dead]
You SHOULD follow the wording of the RFC, you MUST follow Google's interpretation of the RFC.
That is the difference.
evidently they must not
And the definition of "SHOULD" (from RFC 2119) is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."
Having said that, I regret my original characterization of the Message-ID header as a "requirement" and have updated the blogpost to be fair to all sides.
Thank you for bringing this up.
GMail SHOULD handle your messages, not MUST.
The HTTP User-Agent header is also optional, but if you omit it, something like half of all endpoints will respond with a 500 error code.
1. SHOULD means, do it if you can/you have to have a really good reason if you don't do it. The only reason it's SHOULD and not MUST is backward compatibility. Mostly in context of "personal send mails", i.e. not automated mails. (For automated mail sending the expectations of you running somewhat up to date software is much higher).
2. You can't really implement mail stuff just based on RFCs:
- There docent overlapping RFCs which can sometimes influence each other and many of them obsolete older versions why others still relevant RFCs reference this older versions. This makes it hard to even know what actually is required/recommendation.
- Then you have a lot of "irrelevant" parts, which where standardized but are hardly supported/if at all. You probably should somewhat support them as recipient but should never produce them as sender today (mostly stuff related to pro-"everything is utf8" days). Like in general the ideas of "how mail should probably work" in old RFCs and "how it is done IRL today" are in some aspects _very_ far away.
- Lastly RFCs are not sufficient by themself. They don't cover large parts of the system for "spam detection/suspicious mail rejection". So it's a must have to go to the support pages of all large mail providers and read through what they expect of mails. And "automated mails need a message id" is a pretty common requirement. In addition you have to e.g. make sure the domain you use isn't black listed (e.g. due to behavior of a previous user), and that your servers IP addresses aren't black listed (they never should be black listed long term, but happens anyway, and e.g. MS has based on very questionable excuses "conveniently" black listed smaller local data center competition while also being one of the most widely used providers for commercial mail in that area).
SHOULD = You are strongly recommended to do this, but it’s not absolutely required.
- In most cases, you are expected to follow it.
- You can choose not to follow it, but you must have a very good reason.
For example, RFC 7231 say that there should be DATE header but some embedded devices have no real-time clock so it ok not to implement.
The reason it’s recommended is that it’s useful for detecting when an email you receive is already in your mailbox, so that you don’t accumulate duplicates. Otherwise one would have to compare the complete email, which probably no MUA does. Another reason is that replies can include a reference to the original message, so that it can be properly threaded by MUAs.
So these are mostly quality-of-life reasons, it’s not a reason to reject an email.
> That says SHOULD, not MUST, so how is it a requirement?
Battle with spam has been for long part just trying to algorithmically fingerprint the scam bots and reject the message if it looks like it wasn't sent by "real" mail server/client.
So a lot of things that are optional like SPF/DKIM are basically "implement this else your mail have good chance of being put into spam automatically".
You're totally right. I've updated the blog to reflect this. Thank you!
Also email as a protocol (SMTP) predates RFC5322 by 25 years or so.
Standard RFC naming, optional field.
Avoid SHALL, SHOULD and all other crap, use Elon MUST.
SHALL has been interpreted/clarified by US courts as not being a fancy MUST or REQUIRED that many people were taught it to mean, but SHOULD still has it's purposes, e.g., to provide contractual flexibility in development, i.e., a MUST/REQUIRED requirement was more challenging or complicated and took up more time/resources than anticipated, so SHOULDs can be trimmed due to contingencies.
Another example may be a lightweight implementation of a spec in a limited and/or narrow environment, which remains technically compliant with full implementations of a spec but interaction with such a limited/narrow environment comes with awareness about such limitations.
Worked on an ESP. We had a couple of server software we used on low-level for sending. None of them would accept the message without a Message-ID. But even if you have a super-custom, SMTP-injecting service built, how can you ignore all of these bounces from a provider thats likeliest to be the major one you are sending to? Unthinkable. I would not like to have business with such a payment provider.
This is the one that gets me - sometimes you're forced to work with systems that do annoying things that you have to accommodate. It's annoying, but it's more important to do the thing that prevents your users from having issues than it is to be theoretically right about whether something's required by a standard.
I've dealt with many worse cases than this, where the systems I was integrating with were doing things that weren't even close to reasonable, but they had the market power so I sucked it up and dealt with it for the sake of my users. Maybe Google's wrong here, but how do you not just implement the solution anyway?
> Maybe Google's wrong here, but how do you not just implement the solution anyway?
But they just did (make it work). The logical assumption is that most ppl did the same, just used another email provider. Why would viva care? (same as google, why would google care?).
[dead]
Well, apparently it's not even an issue for gmail users:
So it's only an issue for people paying for Google's hosted email—a much smaller set!> how can you ignore all of these bounces from a provider thats likeliest to be the major one you are sending to?
This is the major issue that most of the discussion is missing. It doesn't matter how you want to interpret the word SHOULD, if you want to send to google workspace, you MUST include a message-id. It's not like this is some fly-by-night server with 12 clients.
If you absolutely and completely don't want to include the message-id, then you need to have a warning that your service can't be used by Google Workspace customers. This used to be common practice, blocking communication to servers that behaved badly, and I sort of wish we'd bring it back.
> I would not like to have business with such a payment provider.
Chances are that the decision-makers in most companies don't care about the technicalities (i.e. which email you used for registration) - they want to get up and running.
The reason that Viva doesn't care, I assume, is the reason Google workspace doesn't care: they're both too big to care for 5% of their clients won't do the extra work. They know that their, usually much smaller clients, will "figure it out" by i.e. using another setup that works™. So why bother?
Since workspaces are probably tied to custom domains all they see if only cursorly looking are "random" businesses with bounces (as TFA mentions, Gmail addresses seem to work fine), sure if you look closer you'll notice the MX might be a google one but going from the cursory look to a closer one takes a bit of time.
I doubt Google Workspace is going to be the major provider for European businesses
...and you'd be wrong.
For now, but with EU digital sovereignty efforts in full swing, it's possible this changes over time. More so if the EU uses regulation to dissuade the use of US Big Tech products and services.
This is round 5 or 6 of a " EU digital sovereignty efforts in full swing" maybe this time they will having a success or two but nothing seems to indicate they've changed enough to avoid failing again.
... which is irrelevant to the demonstrated and shocking incompetence of being unable to deliver either to the #1 or #2 inbox for businesses in the EU?
Helps justify moving away from them I suppose.
My pet peeve are services that go out of their way to include a text/plain alternative message part but send something useless, such as the message without the key link. One time I seriously ran into a service just send a short one-sentence note along the lines of "this is a plain text email" as the plain text part. If you don't want to support plain text, maybe just don't send the alternative part?
I find the ones that try to be cute the most frustrating because these appear on the new message notifications so I can't just delete them straight from the notification.
We'd love to share this exciting announcement but you'll a different email app.
Although I guess the argument will be that email clients should use AI to summarise the HTML into a plain text summary.
> Although I guess the argument will be that email clients should use AI to summarise the HTML into a plain text summary.
Or you could pass it through ~5,000 lines of C [1] and you will have it done in milliseconds even on hardware that would be old enough to drink.
[1]: https://codemadness.org/webdump.html
This might be me being old, but I still don't understand why html emails aren't the exception. If you want to do a fancy newsletter, trying to sell me crap, I can see why you'd need the images, the css and html. In most other cases, I don't really get the point.
What comes to mind
- You are sending a receipt and want table alignment for items
- You want to put a logo of your company so that readers can recognize who's the email from
- You want to make unsubscribe link smaller and the "open the thing I'm notifying you about" link bigger, so that people would know which one is which without reading the url
- You want to add a header
Mostly those seems to be more about you as a sender wanting to do some branding or manipulation of the reader. I don't really see how it benefit the receiver, which should be the main concern of all communications.
You don't see how a table can communicate things more clearly and benefit both the reader and sender?
I had one who sent me the booking details of another client in the plaintext part. I reported it to them nearly a year ago and they didn't reply, so screw anonymity, it was Avis.
If you're in EU or California, you should probably email the local data privacy official's offices about that.
text/plain != plaintext
This is about media types, not encryption.
Do you think I was talking about encryption, or is it not more likely I meant text/plain given the context?
I'm sorry, I did not properly read and comprehend your original post. I thought you were saying "they put sensitive details in the text/plain part", implying that those details somehow only belonged in the text/html part. What you actually said was "they put somebody else's sensitive details in the text/plain part".
Then report it to your government authority in charge of GDPR Enforcement. They suddenly will care very much about it
So I'm wondering a bit here - I've seen an implementation where emails to send only have html versions, but as part of the sending process the html is run through a Lynx browser process with the -dump command to get the plain text, which is included as the text/plain part of the email.
Is there actual value to this? e.g. Is the output of Lynx's text dump better for plain-text email clients than whatever they'd display for html emails?
I've personally converted html to plaintext with beautifulsoup in python, and used that as the plaintext version. Did not have complaints, but I honestly don't know who actually reads the non-html version.
Some (old?) spam filters may be triggered by html only emails.
Epic Games tells me I don't support HTML formatted email. Many emails are just the HTML version with HTML tags removed, leaving behind a bunch of ridiculously long image and link urls and little to no text telling you what they are. You might have better luck with a partial HTML implementation (pull out the title, alt, src, href attributes) than disabling it entirely.
the best is when some put the same payload in the text/plain part as in the text/html part. yes. the html source. as text/plain.
My favorite is when the plain text version is a bunch of css and html.
I dont think people fully realise how novel Stripe actually is.
The idea of capturing payments via an API itself isn't unique, but the fact that as a company Stripe is actually on average pretty competent. I've never had an issue with stripe on the integration side because that just works.
I work with a massive variety of payment gateways, I'd imagine more than most people on this site ever will, 90% of them suck. The big ones all suck, Stripe's "competitors" suck. They all have effectively the same API but the issues are organisational.
Stripe is a technology company, its technology is good, its approach to the end user (Technical people implementing it) is good. The rest are finance companies and you get all the slow to move, impossible to get through to bureaucracy that comes with it. Of course they didn't respond to the actually issue and said "You've already solved it" and then ignored the actual problem. These payment providers sell their services to business not developers or anyone with technical chops.
They wont care about these issues because your boss has already been sold on the product, the issues are very very unlikely to change their mind and it is now your problem to solve.
> The rest are finance companies and you get all the slow to move, impossible to get through to bureaucracy that comes with it.
Look, I 100% get what you're saying here, but it's not (entirely) the other companies fault. The financial sector is incredibly regulated, and they're required to have policies and procedures for everything which leads to a terrible, no good, sub par user experience.
Stripe has (so far) managed to avoid this, but if (when) they end up in the regulators cross-hairs they too will become like many other financial companies. I mean, I want to be wrong on this but I don't think I am.
This is why a sane e-commerce platform should have routing through a bunch of different payment processors so they're not vulnerable to the regulator or regulatory officer anomalous behavior at any particular one.
Yeah, that's sensible risk management for something as important as payments. That kind of redundancy is expensive and prone to bugs, though.
How do you think Stripe had avoided the regulators crosshairs? Probably because the users are satisfied and not complaining. Which brings us back full circle to GP's comment on why users are happy.
I mean, Stripe are pretty young for a financial company. Assuming they keep growing, they'll eventually end up as the focus of regulation.
They do seem to be a very well run company, but if they end up as a bank, then they'd be globally systemic and at that point they'll be a focus for regulatory scrutiny
With fintech that surprises me not the slightest bit. Financial institutions are filled to the brim with unbelievably incompetent people. A large part of it is probably willful ignorance, too. It's often truly staggering that a financial company I interact with in day to day live is even able to exist. That's until I remember that all the others are just as incompetent.
"Major European Payment Processor" really just translates to "Major European Incompetence Center".
With a broad statement like this, I would usually just suggest this is inflammatory and surely overstated.
However, I've also worked at a financial institution which used core systems by Harland Financial Systems. Their "encryption" for data in transit from teller workstations to the core system was just a two byte XOR, and they sent the key at the beginning of the connection!
Was so unbelievable to be able to crack this in under a half-hour after noticing patterns in a PCAP. Wouldn't have believed it if I hadn't seen it with my own eyes.
That fraud was good enough for our regulators and theirs, so I have no doubt the industry is filled with rotten incompetence through and through.
The biggest disappointment in my 30 years of adulting has been how much absolute, shameless incompetence is out there in the workforce. When I was a kid, I naively thought that adults were smart and knew what they are doing. Then I got into industry and saw so many people just outright bluffing for 8 hours a day before going home, day in and day out.
It's amazing that society even functions at all.
I think that's actually an interesting feature of society as a macro system. It is very fault tolerant, which is frustrating for any power user but without which the system as a whole would not function at all.
Or without which the system would function much better because all poorly functioning systems would fail, forcing designers to focus on fixing the faults
I think of it as a plane that cannot land. Allow it to fail in flight and you won't have a plane to fix or designers to fix it.
At least when you realize it, you are cured from any imposter syndrome you might have.
Until you actively start looking for genuine personals from the flood of incompetents
There is plenty amount of incompetence in FAAMG. Notepad ....
Do Europe financial institutions have the same level of corruption as the USA? Such as a credit card company authorizing credit card transactions with incorrect expiration date to maximum profit, Bank of America? Or opening new accounts without consumer consent, Wells Fargo?
It's a broad question, but in many ways, very clearly yes, re. corruption of financial institutions, and to a far greater extent in many, albeit different ways.
Incompetence and corruption only slightly overlap in most cases, i.e., being competent at corruption is a very real thing. The incompetently corrupt, usually end up punished... and there are few and far between...as we all very well know.
The kind of schemes you mentioned are generally not going to be how "corruption" will manifest itself in European financial institutions, because although it is also difficult to speak in general across Europe since the EU has not yet subsumed democratic self-determination all across the continent yet, so there is wide variation; the competent corruption is largely in the form of money laundering and tax evasion, not lower level quantitative schemes that would quickly come to light because Europeans are also a lot more cognizant of money and value than Americans, so people are paying attention a lot more closely and will raise hell over a single cent, where Americans are known to have hundreds of dollars draining out of their pockets every month just alone on recurring payments for things they don't even use anymore and don't bother dealing with it.
What we all don't really seem to internalize as a human species, is the absolutely demonic type of pernicious nature of "banking", i.e., a kind of LotR, ring, that consumes you especially if you are weak... and human, or at least European civilization seems to frequently go through periods of immense weakness where things are going springily and everyone is dancing to the music the "bankers" are playing as they are pandering our pockets, and when they realize they could get away with that and all the pockets are plundered, they move on to plundering our homes, then our accounts, then they want to take our first born... "Banking" is like humanity's cocaine, the seemingly innocuous, feel good drug that will consume your soul if you do not rage and fight against that demon taking over aggressively. It's no coincidence that cocaine is so widely used by the most parasitic elements of European societies, especially in "banking"/finance in general.
somewhat, but the question is to broad
it differs by country (EU gives countries some similar overlapping laws, but they still are distinct countries with a lot of different practices and different laws in many many ways.)
and it differs by severity
E.g. deutsche bank has become notorious to appear in most large scale bank scandals _in the US_ as a "German Representative". They also repeatedly get into trouble for relations to money laundering operations and other issues. At the same time they are the Bank which has to give you a (expensive but not completely absurd priced) bank account no matter your credit score, criminal history etc. and is the bank you mostly likely have a escrow account parking money if the person who gets it "if you mess up" is the state/local government (like e.g. in some long term visa context).
In general banks move very slow, including with software protocol updates/changes. And that isn't just the case in the EU AFIK. Actually some of the most crazy cases of "old software" I have seen in "the west" where as far as I remember in the UK and the issue was that it was deeply fundamentally impossible for them handle a lot of non English names correctly, like in 2017~2020 or so. Through at least they still could use the bank, just with a crippled/en-ified name. Funnily the same but worse (as in not working at all) I have heard about Japan banks, where they had(have?) really big issues if you have a middle name.).
> Do Europe financial institutions have the same level of corruption as the USA?
lol... how much time do you have? Here's just one to start with off the top of my head: https://en.wikipedia.org/wiki/HSBC#Controversies
https://en.wikipedia.org/wiki/Wirecard_scandal
I mean, since 2008 finance has been officially privatised profits, socialised losses. It's inevitable that an industry people are forced to use and cannot fail becomes this way.
I have some level of sympathy with Google here, which isn’t something I often say.
I recently switched from Gmail to Fastmail and by and large I’m happy with it. But I’ve been surprised by the amount of spam and (particularly) phishing emails I get in a regular basis. Google might be too strict in its filtering but it does serve a legitimate purpose.
> Google might be too strict in its filtering
they are, but not in this case
Message-Id being basically required for _automated_ mails (very similar mail send to a lot of people) requiring Message-Id is a de-facto industry standard. Sure some providers don't care and some might make it just more likely that your mail ends up in spam. But this could have happened with pretty much any mail provider widely used by companies.
much more funny (/s) is if you start out in a startup and still use the default template for password reset/on-baording links of a widely used system (e.g. Keycloak) and it turns out multiple larger but "cheap" phishing campaigns did the same and now MS/Google and other suspect you are running a phishing operation
or when you use a local data center and can't send mails to MS/Outlook anymore because it turns out someone did some legal questionable things on them and MS wanted the personal information to be handed over _without court order or any ongoing legal proceeding against this people_ and they didn't hand it over (partially because they legally aren't allowed to...) and MS decided to retaliate by permanently blacklist the IPv4 range of that data center(s) which just happen to locally compete with Azure while self hosted mails competes with Outlooks...
In Europe you can sue Microsoft for not accepting your emails that aren't spam.
>basically required
doesn't sound like a hard requirement
Interesting that you mention this, as I also switched to Fastmail recently and got more spam than before, but after marking it as spam for some time it now died down I think. This may also be a symptom of changing providers, where the previous provider knew the kind of spam I tended to get from past years, while Fastmail needed some time to get up to speed.
Fingers crossed that the experience will be the same for you.
Long time Fastmail customer. Let it learn a bit about your inbox. I barely see any spam these days that’s not been caught. Except LinkedIn, that somehow goes through
Fastmail seems to go through periods where they're a little slower to adjust to new spam techniques, and they do rely on users filtering somewhat. About twice a year a few will slip through, but if I report them as spam they soon stop.
I've been a happy customer otherwise for years, for what its worth.
I don't think a single spam email has ever crossed into my Fastmail inbox. Granted, every service I sign up for gets it's own masked email. But while the @fastmail.nl email that I chuck on my website gets a fair amount of spam, it always gets categorized correctly.
Just to add another data point - I recently switched as well and have seen no uptick in spam at all. Could just be my emailing habits though? Not sure.
I've considered this switch. You're saying that previously gmail was dropping the emails, or they were landing in spam?
I switched from Fastmail to Gmail/Workspace a year ago. I think but cannot conclusively prove that Gmail drops Apple transaction emails on occasion ( like receipts ). But I also think Fastmail dropped other emails too.
Presumably they were in spam. But I rarely ever checked my spam folder to know with certainty.
The first thing that comes to my mind is: how come viva.com is unable to send emails to google workspaces and nobody at viva.com noticed before ? For how long has this going on ?
The second thing is, what email software are they using ? If it was any relatively used software I would not expect this problem to arise (maybe it is some commond software but misconfigured).
Third, while the header is not mandatory, I usually read SHOULD as a "if you don't implement it prepare for possible problems". SHOULD is not MAY.
Fourth, they should be thankful that Google bounced the messages with some appropriate error explaining how to solve it. I have plenty of issues in the past with both Google and Microsoft where they accept the message for then sending to /dev/null
> Who's in the right
I don't think either are. The payment processor should be sending it, but, at least according to the RFC, it is incorrect to reject an email that doesn't have it. I suspect the reason it is SHOULD, and not MUST is for backwards compatibility with software that predates the RFC that adds the message-id header.
Maybe there is a correlation between missing that header and being spam, but then it should go to the spam folder, not be outright rejected.
----------------------------
The experience with support is also similar to experiences I've had with support at many companies. I provide enough details that an engineer could probably easily fix the problem, but the support representative just dismisses it, and it is doubtful an engineer even hears about it.
Exactly. This minutiae is all so weird. Email as a formal specification does not work, and the industry as a whole has accepted that for decades now. It's not possible to filter spam from valid traffic without applying a truckload of heuristics and leveraging an ever growing set of auxiliary signals (SPF, DKIM, yada yada).
To wit: basically everything in this world is a "SHOULD", at best. The rules are a conversation.
Then why does my email program reliably distinguish spam from ham without any server-side filtering involved?
If you have a client app that you think is reliably filtering spam, then to be blunt: you aren't receiving any spam, at least to first approximation. My stack of stuff on my three decade old personal account has a 95%+ hit rate (something I might naively be tempted to label as "reliably filtering spam"), and I see see more spam than signal in the inbox.
GMail, on the other hand, is damn close to 100%. And it does it through excruciating application of heuristics like "don't trust agents that don't set SHOULD headers".
But GMail is accepting the Mails from his payment processor? It's just Google Workspace that won't
They're variants of the same software stack. But that's sort of the thing: email filtering (the only thing that makes email useful) requires tuning and heuristics. Decisions that are effective for personal email may not be for the spam faced by corporate clients, and vice versa.
No one (at least no one serious) thinks you can just implement some RFCs and get a product that works and doesn't suck. Life just isn't like that.
I'm just speculating, but probably because you're on an email provider that isn't a big enough target to worry about the persistent threat-actor model.
Google is a big enough target to justify spending the resources on dedicated attacks against their infra. Other providers may simply get less spam because their email domain shows up less often in the sources attackers use to pick targets.
It's not necessarily the support person's fault. I worked in support way back when. There were some long-standing issues that needed only minor work to fix that engineering management didn't get, didn't care about or just wouldn't prioritize. Engineers weren't free to work on what wasn't prioritized. Acknowledging a problem and leaving the ticket open doomed to be unsolved forever negatively affected my performance rating. I had to respond with this sort of cheery everything's fine message and write it off as a customer issue, even though I didn't believe it.
The most damning thing about this is they didn't test their email infra w/ Google Workspaces. Imagine what else they didn't test.
They are testing it, every time someone signs up and it fails. We don't know that this wasn't something that changed on Google's side, so IMO it's a bigger indictment that no one is monitoring their live email deliverability
yeah, because the whole world uses Google workspaces, right /s
That and MS Office are pretty darn popular. Not the whole world, but a very decent percentage of your users.
Maybe the whole thing was intentional, right at the footer of viva "Cloud services by Microsoft Azure" ; #1 I've never heard of viva before #2 I've never seen an azure logo at the footer of a website.
If I were to test an email delivery system, I would test Gmail. I probably wouldn't test Google Workspaces, because I'd (wrongly) assume that they work the same.
No, just over 6 million paying business customers.
But hey, if you're in a business domain where categorically leaving 6 million potential clients-who-are-demonstrated-to-spend-on-things isn't an issue? One fewer thing to worry about, right? ;)
Certainly enough where this is embarrassing incompetence by them.
"Email is tough", software development is tough, IT is tough, walking and talking at the same time is tough, mailing a letter is tough.
When orgs frame problems like this, it erodes trust in the message they try to convey. Email isn't a tough problem, but its a problem nobody wants to really deal with. Email is simple - its a text based protocol, that started out open, but now you need to add security to ensure your email is delivered.
The Gmail requirement is actually slightly different: the header must be present and unique. Gmail only keeps one copy of a message per user and message ID. Combined with a mail source that uses predictable message IDs (such as Github), you can abuse this to suppress delivery of certain messages to Gmail users.
Interesting, but what do you gain to send an email which you know will not land?
They mean to send an email in advance, with a message ID that would later be used in the target email. First email gets ignored, moved to spam, or not read yet. Then the target email gets sent with the predicable message ID, and gets bounced.
Comments on issues use the format <[OrgName]/[RepoName]/issues/[IssueNumber]/[CommentID]@github.com>
A mitigation to this would be to take the combination of message ID and the sending domain and use that as the unique value, because message ID is not guaranteed to actually contain a domain label that's owned by the sender.
For example SendGrid's message IDs are <[RandomValue]@geopod-ismtpd-[Integer]>.
Minor correction: The message doesn't get bounced, it gets de-duplicated against the first message. Effectively, it's deleted.
If I send it first, the real message won't get delivered. The real message could be be a newly reported security issue.
> The reason Message-ID is SHOULD rather than MUST? Mail clients
> sometimes send messages without one to their submission server, which
> adds it on their behalf. As for why Google enforces it anyway:
> spam. Messages with minor RFC violations are far more likely to be
> spam, so rejecting them is a reasonable heuristic. In practice, Google
> and Microsoft have become the de-facto standards bodies for email —
> what the RFCs say matters less than what their servers accept.
Surely the problem is on Google's end? And a metaproblem is that we are allowing corporations to change or ignore standards for critical infrastructure?
The email landed in the spam folder. A bounced email means it didn't find the inbox. If it didn't find an inbox there would be no log for him to check. Technical knowledge of emails and what the terms mean out him instantly as a liar. The fact this is still up on the front page is an embarrassment for the tech community in my opinion.
[flagged]
Please don't cross into attacking another user, no matter how wrong they are or you feel they are. We're trying for something different here.
https://news.ycombinator.com/newsguidelines.html
Dude, you keep on just believing a blog post even though it makes no technical sense. That's why none of the hardcore nerds are even approaching this and why Viva are going to sue.
An email bouncing means it was rejected because it did not find an inbox or a recipient. You AI to question if that is true. So if the email service provider says we don't know who that belongs to, why would they provide logs.
And what email service provider is rejecting an email over a RFC thing when millions of emails a day because most devs don't care about the RFCs and only an idiot would have two different mail infrastructure and code versions running for gmail and workspaces. Especially on the sending and receiving part.
So are you going to apologies for being confidently wrong? Because I'm technically and confidently right. Again, which is why the hardcore nerds are leaving this the fuck alone because they saw me wreck the post in minutes of it being posted.
I think something is wrong with your blog
Typically I'm a DIY type who loves tinkering and building...
HOWEVER, I have learned the hard way to never apply that spirit to email.
In Europe you see this stuff all the time with old school "IT" (what old industrial companies call tech) people balking at the prices of commercial API-based senders and email marketing ESPs.
"Money to send emails in the cloud? HAH! Back at Siemens in 90s we were running millions of emails out of our servers just fine!"
Nobody understands that deliverability has gotten immensely harder these days, and trying to DIY it if its not your core business is just plain stupid. I would never in a million years try to roll my own email, it's nightmarish legacy cruft and footguns all the way down, in everything from IP/Domain Rep to something as simple as the HTML in the email templates themselves.
Microsoft Outlook and Gmail have the last word on everything in email, and their defacto duopoly (over B2B and consumer email respectively) means you play by the rules they set in 2008 and are too lazy to change or you don't get delivered. The protocol of email exists separately from the world of the actual inbox providers, which are locked down to insane degree given the security/spam concerns with email.
Google is at least less arbitrary than Microsoft. Microsoft will decide an email is spam today, and tomorrow the exact same email is perfectly fine. I think Google relies more and more on sending IP and domain reputation rather than content.
Google regularly sends legitimate email to my spam folder.
Microsoft regularly sends legitimate emails from Microsoft to my spam folder.
Deliverability to Microsoft famously took a dive a bit over a year ago due to random arbitrary failures within their infrastructure causing DMARC/DKIM problems which they clearly were having problems diagnosing.
Even with a six-figure email spend and weeks of troubleshooting the best response we could get from our mail provider was that they were having problems getting traction with Microsoft on the issue.
Worth mentioning is that there are several email umbrellas under Microsoft... including the newer office/365, the slightly older outlook.com hosting, the old corp hosting and hotmail and sub-properties... each with different rules and services to determine spam in inconsistent ways between them.
One of my main emails is still on a "free" outlook.com hosted with a personal domain that I never shifted to paid 365. I've also got an MTA server (mailu) of my own that I've been testing with... my own email under outlook.com is literally the only one of the MS systems I can't seem to deliver to, the rest work fine. Same for google.com for that matter... kinda wild.
This issue didn't seem to discriminate. I was seeing deliverability failures to Office 365 clients as well as consumer-facing brands like hotmail and msn
Okay... I was just adding my own experience as I was able to deliver to some without issue and not the one. It's entirely possible this has changed in the past couple years since I was actively testing.
I recently had the misfortune of looking into email delivery to help a small business. I came out tired.
Huh I've lived in Europe for most of my life and I've never heard of viva except as a poor name choice for Microsoft's corporate Facebook (yammer)
Most companies here use stripe on their website.
Yeah, they're defining 'major' fairly generously, here, I think. Difficult to find figures, but seems to be transaction volumes of tens of billions a year. So not actually particularly big.
it's a greek company, if you buy from greek websites i guess you'll deal with it
i'm not greek but a greek ecommerce i buy from uses viva
If that's how they handle email, I wouldn't want to see what they do with payment data.
"For simplicity, we support recipient account IDs in the range 1-10. Anything else gets silently ignored."
Maybe that's something to report to the "European System of Financial Supervision" or some other EU government agency.
They even have a Whistleblowing link at the bottom of their website: https://www.bankingsupervision.europa.eu/about/esfs/html/ind...
no, please read the article
they forgot to include a message-id
something the RFC standard recommend but doesn't require
but it being required is a de-facto industry standard for sending automated mails
and is clearly documented by support sides of large mail providers (like Google)
the mail standards only defines what parts you can put together, but widely fail to define how this parts can be interpreted, what are sensible combinations, etc.
and they don't cover spam/suspicious mail detection at all
so you can't just go by RFC, you need to read up on what all larger mail providers have as additional requirements (which mostly are the same, and Message-Id being the most common dominator) and then hope that another provider you didn't read up one doesn't have some other surprising rule (which doesn't tent to be the case if you don't do anything surprising, but it sucks anyway).
The specific bug is annoying, but that there's no way to report such a thing is an exact hallmark of our current corposphere.
Google's Postmaster Tools site has a "Report deliverabilty issue" link at the bottom left navigation column.
https://postmaster.google.com/v2/sender_compliance
it isn't a google bug
Message-Id being required for automated mails is a de-facto industry standard
while the consequences differ between mail provider, it missing will also make it much more likely for mail to be reject or put into the spam folder
It's also well known. Pretty much viva engineers fucked up doing proper research.
Now to be fair:
- it sucks that you can't just implement the RFC(s)
- the standards suck, docent of different RFCs overlapping and replacing each other and referencing often older versions of other RFCs, with docents of ways to do the same things of which only some can be used reliable in practice and a common gaps in the standards about edge cases or about the "higher level semantics" of constructs.
- so overall mail seems very simple at first but if you want to automated send mails reliable internationally it's a total pain and Message-Id is just the head of the iceberg.
Which is, of course, hidden behind a login wall
This feels jumping on the train of complain about Europe and fully bias.
If I apply my own bias I will call it:
I've often found when receiving a clueless support response like this, it can be effective to just follow up with a polite request to forward the ticket to an engineer or developer. Usually the front-line csr simply hasn't understood the issue. In this case I would say something like,
"Yes, I managed to work around the issue by switching to my personal email address, but this bug is still preventing me from using my work email domain. If you could please forward the error log I included to a developer, it should help them resolve the issue. Thank you."
But imagine that you do that, and they solve the problem. What would you write in your blog about?
10 percent of the effort in building software compatibility with open source file specifications is dealing with knowing the specifications. 90 percent of the effort is dealing with errors in generated files by less worthy software programs.
The RSS spec is one way. RSS readers do a fine job of interpreting files done the right way. Publishers don’t always do a good job with publishing error free RSS files. So RSS readers devs have to anticipate all sorts of errors and conduct error handling to ensure RSS items are properly handled.
This is why companies want to keep their file format proprietary. Other devs can really do damage to the ecosystem and ruin the experience
My personal fork of ttrss, from 2005, is a dodgy patchwork of fixes for badly formatted RSS. I can't imagine trying to host a service that deals with RSS feeds from random sites at scale.
One that always irks me to no end, is every time I see someone ham fisting csv handling by hand instead of using an established, well-sourced library. They almost always fail at commas or newlines in quoted text... It's one of the more annoying things.
Currently working on replacing a couple decades old system, and my csv output is using a library that isn't quoting all the strings that don't require quotes... so I'm forced to do it (for compatibility) with the other system this csv is going to. (sigh).
> This experience fits a pattern I keep running into with European business-facing APIs and services. Something is always a little bit broken. Documentation is incomplete, or packaged as a nasty PDF, edge cases are unhandled, error messages are misleading, and when you report issues, the support team doesn't have the technical depth to understand what you're telling them.
I can definitely confirm that this is a common thing. But I think this is a "small org"-problem more than a "European business"-problem. Apparently, the company has somewhere between 500 and 1000 employees (I couldn't find good data, sadly). With a size like this, the "support" is probably outsourced (meaning they don't know anything), there are maybe 100 engineers (probably less) and the mailing is either done via a third-party or set up by an Admin that left three years ago.
Without any basis, I will speculate that you will notice this more in Europe because there is simply no company at the size of Stripe or similar.
While there is some truth in while you saying, I have to say that from my European perspective all the big American companies feel enormously bloated. For example, I've recently learned thats Atlassian has 13000 employees, and I have to ask myself, what do all those people do?
A minor correction, but Atlassian was founded in Australia, and their global headquarters is still in Australia.
I bet Atlassian's email notifications never bounced.
They're busy hardcoding passwords into their source code
https://www.bleepingcomputer.com/news/security/atlassian-con...
> This experience fits a pattern I keep running into with European business-facing APIs and services
honestly, it's a pattern I have been running into with many start ups, fintech, banks and some other places _no matter where_.
It also often makes sense. For many large orgs (e.g. Banks) this APIs are often a side business, sometimes one they don't want but have to have for compliance (or market pressure). But for strip it's their live blood.
And many startups (like actual start ups, not 200 man companies running by investor money) often simply don't have the resources to prioritize a "very nice to use API".
Lastly API design which is both nice to use and stable for existing integrations is surprisingly hard to get right (if you don't have some senior engine prioritizing it very highly and forcing it being kept prioritized; Or a surprisingly "clean"/"clear" use case.).
> For viva.com's engineering team, in case this reaches you: add a Message-ID header to your outgoing transactional emails.
Don't know what they're using for sending emails, but that's something that should be handled by their email service provider, unless they're hosting their own email servers.
Interestingly the MX record of via.com points to Google, but their verification emails could come from anywhere of course. The IP address in the log is also a Google IP, although that could also be the receiving IP.
message-id is (or at least was ~5-10 years ago) only required for automated mails, i.e. if you send a mail which looks mostly the same to a lot of users
so if you (ab-)use protocols for mail clients to send automated mails they might very well not add Message-Id. In general many mail providers have both a way to send a mail where "they do some clever parts" (like adding Message-Id) and interfaces where they mostly just send what you give them. It's possible that they migrated from one solution which did automatically add Message-Id to one using interfaces where it doesn't happen without realizing that there is a mismatch in this solutions do implicitly add.
I went through this hell last year when trying to tell my customers I had to change payment processors.
In my case it turns out I was victim of Google having a beef with afraid.org years ago (the DNS authoritative record for my domain). 100% of my emails to gmail were going to spam, as soon as I switched NS record to fastmail (with same zone file) I hit the inbox.
I never would have caught it without using this site: https://www.gmass.co/inbox
I did not show up on any of the major blacklists with mxtoolbox, this took months to figure out and I felt like I was going insane with hardly anyone responding to my emails.
If a business like that doesn't get its emails delivered, it will slowly go out of business. Merchants will find another processor that is able to deliver emails to every inbox. That is, Google could be less picky, but the company with a problem at hand is Viva.
Might want to consider Adyen, which should support IRIS, the Greek instant payment system.
Thank you for the recommendation! That I couldn't sign up using a form and I had to "talk to their team" was a turn-off for my (extremely extroverted) self.
That usually means you can't afford it unless you have people working for you that do the 'talk to the team' thing.
> Why this matters
Hello AI (Claude?)
It used to be said that the reason the Internet evolved so well was because of the basic principle of "be strict when you send, but tolerant when you receive". Clearly Google has forgotten this.
That worked when the Internet participants were mostly benign. Nowadays you have to account for abusive participants (spammers, malware writers, etc., etc.).
That is unstable if defectors are rewarded - which is a common issue on the internet.
If you are too tolerant then bad or lazy actors will cost you too much.
You end up with the tragedy of the commons called "quirks mode".
The RFC MUST SHOULD discussion is funny ...
Considering that Google Workspace is a major player and a place where your customer are, do you want your customers to succesfully use your product?
If the answer is no then yes go ahead and debate the RFC.
If you care about your customers and your product working broadly with a diverse set of mail services then please spend 15 minutes to write a patch to add a Message-ID header?
> For viva.com's engineering team, in case this reaches you: [...]
That's too kind of you, but on the other hand it really doesn't solve the issue of bad priorities and lack of overall Quality. Some engineer might log a couple hours fixing a Level 3 severity bug, emails will start working better, but the poor (or at the least, dubious) backwards technical stewardship (or lack of it) will keep going on inside the company, unnoticed from outside (until something bad eventually happens to some client)
Message-ID is a requirement for Usenet where it came from.
It is a requirement for being able to reply to messages and in general for email threading.
Message-ID is a requirement to archive email.
Practically every email client has included Message-ID since dial-up internet was fast and fashionable.
Given all of the above I am amazed more places don't drop email without a Message-ID.
Not including a Message-ID seems to be saying you don't want replies and you don't want your message to be archived. That seems very shady to me.
Veeery interesting. I have a personal domain that forwards to a @gmail.com account. There are several companies I interact with, from banks to retail to just about anything, that send mail to my personal domain that never arrives in my Gmail account, but I can see on the hosted mailserver. For those companies I have to use my @gmail.com address instead for the mail to get through.
Maybe there are more companies than we think that send malformed emails?
It's amazing how long this has been on the front page. Are people upvoting as a "hit" against the EU wanting out of Visa/MasterCard? It's certainly interesting timing
edit: It seems Viva primarily is a neobank, not a payment processor. I'd never even heard of it until today. It's tiny compared to Revolut, Monzo etc. And tiny compared to an actual payment processor eg Worldline (French, 18,000 employees)
I vaguely remember hitting this message id issue in Google Workspace, and being able to work around it in mail routing configuration.
Saidly I don't remember the specifics, it was something along the lines of not all, but only specific routing features requiring it. Workspace settings are a moving target anyway, so the behavior probably changed more than once since.
I'm not saying it's a good idea to send emails without message id, but i'd also double-check that workspace configuration.
A quick search for their developer support revealed they accept submissions via GitHub issues for their API. Perhaps try there? https://developer.viva.com/get-support/
Sometimes you have to get creative to reach out to a company's engineering department...
Email seems like such a silly protocol but it's importance cannot be overstated.
Do you want to enable receiving email for viva.com? sign up for VibeCodedSAAS for E49.99/month
Just did! My mac mini got pwned though and I wish I didnt give it SMTP accesss… sigh
I just hope my OpenClaw skills registry doesnt have malware anymore. I sure trust my supply chain of vibecoded software!
Mamiride dot com email verifications are not delivered to Gmail from a self-hosted mail server and I wonder if this is the reason. We got around this by making email verification an optional step instead of mandatory.
Amusing because gmail doesn't even follow spec. I had to workaround gmail quirks when I worked at an email company.
Better than outlook though. What a nightmare.
I hope OP took more effort in making Viva understand the problem than the obvious zero effort it took writing this post, given it is completely AI generated.
To author:
The phrase:
“sends verification emails without a Message-ID header — a recommendation of RFC 5322 since 2008”
can be misread as though RFC 5322 recommends not including a Message-ID.
I've never heard of them. Looks to be a company from Greece. That would explain their reply. Not exactly known for their tech.
> Their support team's response to my detailed bug report
As you said, its not a bug.
A feature request might fare better.
The problem is always e-mail itself. It is terrible standardised and hard to get "right".
That will teach those pesky Europeans not to start their own payment processors.
> Their support team's response to my detailed bug report: "your account has a verified email, so there's no problem."
Sadly I doubt their system is xkcd806 compatible ether.
This isn't an engineering problem, it's an ITIL problem. To be fair 99% of these complaints will be dealt with by the flow chart. Sadly people on the front line are either not knowledgable enough or not empowered enough to bust out of that straightjacket.
I usually reply with snark dialed up about 50% asking if anyone had actually read the original message. And include detail about how it is not, in fact, "ok"
The other day, I literally had trouble signing into a website... then I tried filling the contact us form, about the bug... only to have that fail... call in, have the person on the other end schedule my appointment, then almost drop the call without actually logging my bug report/complaint about the whole issue that had me calling in the first place.
SHIBBOLEET. I was seriously thinking of this when I contacted them :)
Sudden realization that one of my American banks must be having email problems with this too because I use a Google custom email and recently got an in-app notification from my bank saying "we're unable to email you" (and a letter) yet my email works perfectly fine... switching to consumer gmail worked.
Do you actually not receive their emails? Fidelity uses tracker dots to check email receipt, which drives me nuts, because like any sane person I don't allow emails to load images without damn good reason. My brokerage should not be sending me cute dog photos, thus I have no need of their images.
So they send me an email, send me another email saying they can't reach me by email, then mail me a letter with the same content as the original letter, and mail me an additional letter saying they can't reach me by email.
For some reason the announcement they sent me didn't show up in my email, which was odd. I've had other bank notifications come through unopened before, even after that announcement.
> This experience fits a pattern I keep running into with European business-facing APIs and services. Something is always a little bit broken.
I feel like this isn't just business services though.
American engineers are used to working for either big tech or "Silicon Valley inc." European engineers are used to working for Volkswagen, Ikea or Ryanair. Very different kinds of businesses who treat tech very differently.
Over here, competing on user experience and attracting users with a slick interface that people love to use isn't really something most companies think about (and so they get their lunch eaten by the Americans).
Nowhere is the European mentality more evident than in cybersecurity, where outdated beliefs still dominate. In this mentality, everybody is out to get you (and that notably incudes your vendors, your business partners and your customers), so all infrastructure has to be on prem, open source is free and hence suspicious by definition, obscurity is the best kind of security, encryption doesn't work so data should go over custom fiber, and if you have to expose an API on the public internet, an Authorization header isn't enough, it should also require MTLS behind a layer of IpSec.
And we're still getting passwords changed periodically or requiring a number, upper case letter, symbol...
I'm an European engineer and I can confirm that our tech is often broken and customer-reachable people are usually obtuse and hostile about it. We don't even seem to properly implement our own legal requirements. Sometimes, Americans implement the RGPD better than we do.
I offered to host a friends business email on my DO instance. Works 99% of the time but every now and then emails just disappear only to find out that MS and Apple block DO IP addresses, sometimes. Silently. There is a war on small email providers it seems.
The biggest war on small providers is waged by other small providers. They can be ancient and outdated or simply extremely picky. Which makes everything Google or others require a piece of cake, which it actually kinda is.
Shouldn't this be "doesn't want to send..."?
Gripe only related to email in general: what annoys me to no end is that if my boss forwards me an email and asks me to reply to it (to everybody in the original email) then I have to type in or copy+paste all the addresses from the Fwd attachment (using Fastmail, but this problem exists everywhere). Instead, there should be a button to make that easy.
There's actually nothing that prevents that, if you craft the right headers you can reply to a thread you were not included in, and have it show up as a reply in the thread of common clients (tested Gmail and Outlook).
We added this feature at my $dayjob and I was quite surprised there is no authentication. But thinking about it, this is how mailing lists work (you aren't explicitly specified in "To:") so it makes sense you can do this.
I’m sorry but in the context of a 50 year old technology like email, 2008 was yesterday. Gmail is in the wrong, you don’t get to just update the standard for email like it’s TikTok content or a Roblox update or whatever.
Email was here long before Gmail and will be here long after Google abandons it.
This is why I don’t use Gmail.
Also, get off my lawn.
[dead]
>"We can see your account now has a verified email address, so there doesn't appear to be an issue."
There are still too many edge cases like this one that can't get fixed because of ignorant support not doing it's job. In my life, every company that escalates to an engineer instead of punting the ticket with some asinine 'but it works right now, goodbye' message gets rewarded via keeping my business. The ones that don't are immediately cancelled. Sometimes I even do a chargeback as extra punishment. Maybe I'm just old, but I have near zero tolerance for immature support playing games with my time.
literally everything is tough when comes to emails
Email deliverability is the reason I gave up on email entirely for my side project and built on Telegram instead. Setting up SPF, DKIM, DMARC, warming up a domain, monitoring reputation, dealing with bounces and complaints... all of that just to maybe land in someone's inbox.
With Telegram you send a message via the Bot API and it arrives. 100% deliverability. No spam filters. No authentication chain. The message just shows up with a notification on their phone.
Obviously Telegram has its own limitations (smaller user base in the US, less formal). But for anything where you need reliable message delivery to people who opted in, messaging platforms have a massive advantage over email in 2026.
Unless you are running a more complex setup SPF, DKIM and DMARC really aren't that complicated. They are annoying and additional checkboxes you have to go trough that are hard to fully automate because they require access to DNS, but they are more busywork than difficult.
Domain and IP reputation and all the other quirks of deliverability are much more of a headache. DMARC is setup, test and done. But deliverability in praxis is something you cannot just test and can break at any time. The second worst are email providers that do whitelisting for email and require you to go through their process to even be allowed to send emails to their customers. The worst are providers that randomly decide to drop your emails without informing you or giving you a proper way to appeal as a small sender. If you're not a large email provider they have no problem just dropping you and the fault is on you because your service is the only one that is not working.
And then there are so many more intricacies of the ancient email protocol. Compatibility with horrendously outdated and misconfigured mail infrastructure is particular frustrating to me. I'm running a modern, properly configured, state of the art email server, but get ghosted by large email providers, yet have to deal with the broken mess, much bigger providers than myself are, which of course have no trouble delivering their broken spoofable email just because they are large enough.
I found it maddening that I couldn't whitelist a sender (MS Outlook web, Gmail); well I could, but they still blocked the messages.
In my case, it was reportedly (for MS) an IP associated with mine (same hosting provider) had previously been used to send spam.
My domain is decades old, never sent any spam, and I whitelisted it .. but nope, my host wasn't perfect.
This was some time ago now, but it looks like they've still not adopted proper whitelisting.
Most of that can be mitigated, or at least centrally managed, using an ESP like Mailgun or Sendgrid.
It is a pain in the ass though, coming from someone that had to dig their domain out of "low" reputation with Google Postmaster.
Until the platform owner bans for whatever reason and if communication by way of the platform was your only means of communication with your customer base, that's the platform owner having the power to destroy your business. No different that businesses that rely on the neverending goodwill of the mobile app store owners. One misstep and your business is gone with no recourse whatsoever. Protocols > Platforms. Always.
Some of us selfhost email, like me for 30+ years, and have 100% deliverability.
Unless your email daily volume is close to zero, this is unlikely.
I have had random emails from Google Workspaces to Microsoft 365 be rejected because of a rule that blocked whole IP blocks.
[dead]
[flagged]
Interesting, your take away is that Google is the one with the bug here?
If Gmail rejects emails from your domain it is up to you to fix it. Google is not going to change, and enough of your users will be interacting with people on Gmail that you have to fix it. It doesn't help that Google has been pushing people away from running their own email and into Google's services by ever tightening what it accepts over the years. More than one person has given up on their email server because it was a constant battle with Google, Microsoft, and company to not have important emails disappear into the void.
Their @gmail.com servers accept the messages (as said in the post) so it's not a problem for 99% of Google users either.
If you choose to host your email with Google, it's up to you to fix your email delivery settings (or find a better provider) for your domain.
An attempt no doubt to extenguish a standard google doesn't control
My takeaway is there is no bug. My takeaway is that his test email bounced because he didn't have the reputation Viva does. Emails are handled on a reputation basis, this is why we use email service providers like Sendgrid, Mailgun, Postmark, etc.
It always amazes me how people can read a blog post like this one that has a clear description of the problem with a log excerpts demonstrating the problem, and then people will confidently make up a completely different scenario that was not mentioned at all and blame the problem on that.
User is clearly mentally disturbed. Read his other comment: https://news.ycombinator.com/item?id=46992022
Social network is not good for the poor guy. I already regret replying to him in the first place but I cannot delete.
WTF you talking about? Rene, this is defamation and I'm probably going to take action because honestly, enough is enough. I'm fed up of folk like you who lack basic technical knowledge or any knowledge making up bullshit. Your hourly rate makes me like you have money to take.
Iain, I say this with the best of intentions. Please put down the keyboard. It is serving you poorly.
It amazes me people read that in this community and don't know for an email to bounce it means it didn't find an inbox. If it didn't find an inbox how did he check the logs?
[flagged]
Pretty certain that you're wrong.
TFA shows an excerpt from the email log for his google workspace account, showing the bounce of email sent from viva.com.
Then, TFA states that he switched "the account" (his viva.com account) from using his GWorkspace address to a personal @gmail.com address, and asked viva to send another verification email. That one arrived.
At no point does TFA describe the author themselves sending a test email.
I've added a screenshot at the end of the blog post just to clarify that.
> I decided to dig into Google Workspace's Email Log Search to see what was happening on the receiving end.
It amazes me that you can read an article and draw the exact wrong conclusions
Would you mind scrolling to the end of the post? Or, if you're in a hurry: https://atha.io/_next/image?url=%2Fstatic%2Fblog%2F2026%2Fvi...
https://support.google.com/a/answer/2618874?hl=en
[flagged]
I wish I had your confidence in life
Please read the blog post you are making such strong claims about.
What that is liable? That is a very small claim.
I think that's a misunderstanding of the tale. Viva sent a "click here to verify your email" to OP. That email never arrived because Google rejected it for missing a header. OP tried to tell viva, but they don't wanna hear it because OP worked around it.
> My takeaway is that his test email bounced
What test email? I see no mention of a test email in the blog post. The mail that bounced was the one with the verification link from Viva.
So you think he had access to Viva's email servers to see the response? No, he clearly tested it himself and used his credentials to send it.
The log line is from Google Workspace which exposes it to its customers for incoming mail
Thank you! I added a screenshot of the Google Workspace Admin log screen... just becuase.
Yeah. I think email receiving is a game of exceptions… the email receivers (In the business world it’s essentially just MSFT and GOOG of course) answer to the addressees because they are the customer, and those customers will start to shriek if their inbox doesn’t receive “Important Messages.” But GOOG or MS have no leverage over the senders in this case so they just add an exception: “if IP range is just right and message fault ___ is present, fix message” (or otherwise allow)
Of course, they do have leverage over “marketing email” senders since they can block it and no one will complain, so those senders always have impeccable compliance with every year’s new “anti-spam standard.”
Apple is another major player in the email receiving game for consumers. And they are awful, by far the worst of all the big providers. They do not send dmarc reports and they make it very difficult to tell why they accept some email and not others.
What test email?
https://atha.io/_next/image?url=%2Fstatic%2Fblog%2F2026%2Fvi...
If you read two paragraphs further than the Tl;Dr:
> To unblock myself, I switched to a personal @gmail.com address for the account. Gmail's own receiving infrastructure is apparently more lenient with messages, or perhaps routes them differently. The verification email came through.
1. Email didn’t arrive in his inbox of his Google workspace
2. He checked workspace email logs (with admin you can do this on gsuite)
3. It showed the intentional non-accept
4. Comprehending the problem, he switched to personal Gmail
5. The email arrived
6. He informed the sender of the original problem which he worked around
7. Sender is tech-illiterate and did not realize what the problem is. This is common with first line customer support so that happens.
The question to ask is whether you are literate in English or you skimmed too fast. Because I did a 30 s read of the article and got that.
> 1. Email didn’t arrive in his inbox of his Google workspace
> 2. He checked workspace email logs (with admin you can do this on gsuite)
But it didn't arrive, so how was it in his email logs!?!?!?! Are you tech literate? Isn't this the second time you've been asked or was I rate limited at that point?
It does seem unlikely that there are no customers on google workspace who have tried to use viva. I don't do payment processing, and my email is via zoho, so I've no idea how large either of those groups are.
I wonder what google workspace support said.
I suspect that Google going out of their way to make this required had a very reasonable and thought-out process, while the sender's omission was on oversight, so I haven't contacted Google Workspace support.
What's truly iffy is that GMail doesn't have the same strict requirements, and there's no way (at least that I found) to turn it off for my Google Workspace domain.
Wikipedia says Viva.com is a multi-billion dollar startup
It seems unlikely you're the first company using viva.com and using google workspace.
Clearly the problem here is that viva.com emails aren't arriving on your google workspace, despite what their support process says.
viva.com emails do arrive on other email providers, so seems unlikely to be problem with your viva.com account
It seems unlikely workplace blocks all viva.com emails otherwise more than you would have complained.
Whether that's viva's problem or google's problem is a separate problem.
[flagged]
EU doesn't require edits, what the hell?
And nobody "decided to stop you" if you just hit the end of the 2 hour edit window.
But go on, what's your tagline.
[flagged]
[flagged]
Please don't cross into attacking another user, no matter how wrong they are or you feel they are. We're trying for something different here.
(I know you know this and are not in the habit of doing it, but it bears repeating for others.)
https://news.ycombinator.com/newsguidelines.html
Sorry.
> You don't have the right to complain to random websites without punishment. And nobody punished you. And "ability to redress" is something you still have. And there are no damages.
Well, you see Dang aka Daniel edited the settings on my account to stop me replying, or that's what I've heard, and pointing out the absolute lie that the blog post is. And that did hamper my ability to redress and people have been lying about me so there are damages. But even if it's just rate limiting that hampers my ability especially since it's been optimised heavily for high traffic so there is no technical reason for it since it's a file-based datastore.
> You brought it up. Though it's pretty obvious you're a liar.
I said someone there should know. I didn't say everyone should know. I clearly made it quite clear it was insider knowledge. This is what is known as a flex. I want to tell you my tagline because it's awesome but you can just call me a keyboard warrior.
I thought the problem was inability to edit, now it's inability to reply? Also you're wrong about the post and you don't know what damages are and your flex failed real bad. Good luck in life.
And nobody ever has to let you post on their private website. Right to redress is unrelated. If you take away anything from this conversation, please let it be that.
The bigger issue here is that Europe depends way too much on the USA in so many areas. This is not good - you can be constantly blackmailed when you have people such as Trump in charge. I don't think the EU can be fixed, but at the same time I also think the less Europeans depend on outside factors (in particular the USA) the better. Canada kind of showed how to do it. Granted, Canada is also dependent on the USA in numerous ways and most of this is hard to fix (most Canadians live in the south aka close to the USA and trade is primarily done via the USA; security has also been largely outsourced onto the USA and so forth). The sooner people in Canada and Europe get moving away towards more independence from the USA, the better. And more cooperation would not harm either.
As a European (who spent 15 years in the US), I coudln't agree more. And while I agree, at the end of the day, I just want the better product for me.
Google NOT following the spec is not surprising. SHOULD does not mean MUST and they are completely in the wrong here.
The definition of SHOULD is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course."
I suspect viva.com didn't consider the full implications, and I suspect Google did some hard math on hours saved for their customers
This bug will not be fixed before the Environmental Impact Study is concluded on it.
Postel’s Law would put the onus on Google to be forgiving in what it receives. Unsure how you could safely use a sender-created Message-Id for anything anyway.
Following Postel's law results in the normalisation and proliferation of defective implementations. The actual standard becomes irrelevant, and new implementations have to be coded against the defective ones.
My opinion is that Postel's law should be approached in the same way that Linus Torvalds did CVS when designing Git. If in doubt about an implementation decision, consider what Postel's law would recommend, and then do the exact opposite.
That “law” if from a different time, before protocols like SMTP became adversarial. It assumed everyone was acting in good faith.
Yep. And even a world of perfect good faith, "forgiving in what you receive" has both costs and scaling problems - from researching what "spec" you'll need to design to, to customer service when the added complexity and permissiveness cause interesting stuff to happen.
Hilarious - German users lecturing Google on how to interpret the English RFC?
I say this lovingly, having significant German ancestry:)
But taking a step back :
did viva previously send message ids and pushed a change to prod to strip it? Was it on purpose or an accident?
And other email providers like proton or Hotmail - do they accept messages without message ids?
Have other clients of Google workspace complained about this issue?