Email threads are great for improving the user experience of your app. In this post we will learn how the RFC 5322 specification expects us to thread emails. We will also learn that emails don’t always work as we expect them to. At the end of this post you will have email threading as another tool in your Rails belt.

• Why do we need email threads?
• Email Headers
  • Viewing Email Headers
• Threading Emails: Our first Approach
  • Step 1: The message_id method
  • Step 2: The in_reply_to and reviews_in_thread methods
  • Step 3: The references method
  • Step 4: The headers method
  • Step 5: The review_notification method
  • Step 6: Send emails
  • Gmail is not threading
• Threading Emails: The Github Approach
  • Step 1: Add thread_id
  • Step 2: Update in_reply_to and references
  • Step 3: Do we need the Message-ID?
  • Does the subject line matter?
• Final thoughts
• Further Reading

TLDR;

Feel free to skip along to Threading Emails: The Github Approach if you are in an email thread emergency.

Why do we need email threads?

Email threads help us keep email conversations coherent. Think about the apps we use on a daily basis. We have conversations on Github that result in email notifications that show in a single thread to help us retain context for each notification. Threaded emails also make it much easier to view all the emails related to a single discussion.

Today we will work on an imaginary application called Great Books. Great Books allow users to follow books and receive notification whenever someone reviews a book. To make it more convenient for our users we would like to make sure that the notification emails thread. Users can then look through a single thread to view all the reviews related to a specific book they follow.

For this post we will focus on the mailer part of this feature.

We can create a NotificationMailer like this:

# app/mailers/notification_mailer.rb  

class NotificationMailer < ApplicationMailer
end

We will get back to Great Books, let us first dig a bit deeper into email headers.

Email Headers

The RFC 5322 - Internet Message Format opens a new window protocol defines the syntax required by modern email messages. For this post we are interested in the 3.6.4.Identification Fields opens a new window section. We are specifically interested in the three header fields: Message-ID:, In-Reply-To: and References: described in that section.

The Message-ID: field has to be unique as it is used to distinguish one message from another. The In-Reply-To: and References: headers are used when creating a reply to a message. The message being replied to is also known as the parent message. The latter headers hold the message identifier of the message being replied to and the message identifiers of other messages in the thread.

The Message-ID: header contains a single unique message identifier. The References: and In-Reply-To: fields each contain one or more unique message identifiers, optionally separated by Comments, Folding White Space (CFWS) opens a new window . In this post we will use a single space as our separator.

Viewing Email Headers

Before we go any further it might be fun to first take a look at some email headers. In the early days of email, the email headers would have been shown along with the body of the email. Modern email applications hide the headers to make reading the body of the email easier. It is fairly easy to do a quick google search to find out where you can view the email headers for your specific email client. We will take a moment to look for the email headers in Gmail.

We can open Gmail and select any email. Then we click on the vertical three dot menu button. Next we will select the Show original option. Gmail will open a new tab with all the headers right there for us to inspect.

Please take a moment to see if you can find the Message-ID, In-Reply-To and References fields in the headers from your inbox.

Threading Emails: Our first Approach

Back to Great Books, we would like to send a notification whenever a book is reviewed. The email should thread along with any other review notification sent for the same book in the past.

Step 1: The `message_id` method

The first thing we need to do is to generate a message id. We need to make sure that the id we generate is unique. We have a few options to make sure the generated message id is unique.

1. Generate and store a UUID we can easily retrieve whenever we send a new notification:

 <cb80c10f-181a-4746-9fa3-57294e9b4edd@our-domain.com>

1. Generate a unique id by combining the created at time (in seconds) and perhaps the actual id of a model that is represented in the mail:

 <[id]-[time]@our-domain.com> => <123-1634056600@our-domain.com>

I am sure we can come up with many more unique ways to generate a message id. These two examples should help us get a good understanding of what a message id might look like. If we go with option 1 then we would need to store the message id because we won’t be able to recompute the UUID. So to keep it simpler for now, we will go with option 2.

Let us add some more code to Great Books. We will take the simpler approach and compute the message id on the fly. We can add the following code snippet to our NotificationMailer class.

# app/mailers/notification_mailer.rb   

private

def message_id(review)
  return "" if review.nil?

  "<notification-#{review.id}-#{review.created_at.to_i}@#{ENV["domain"]}>"
end

Firstly we return an empty string if the passed in review is nil. Otherwise, notice that it computes the email’s message id based on some review id and created_at time. It prepens the word notification and appends the domain name. By adding the prefix and the suffix we ensure the uniqueness of the message id within the scope of our application.

Interestingly, the Message-ID is the text between the two angle brackets. The angle brackets are required, but they are not technically part of the Message-ID.

It might look a bit cryptic at the moment but follow along and soon the rest of this mailer will come together nicely.

Step 2: The `in_reply_to` and `reviews_in_thread` methods

According to the RFC:

The “In-Reply-To:” field may be used to identify the message (or messages) to which the new message is a reply. It contains the contents of the “Message-ID:” of the “parent message”. If the reply is part of a thread with multiple parent messages, then the “In-Reply-To:” field will contain the contents of all of the parents’ “Message-ID:” fields.  The “In-Reply-To:” header can be omitted if no parent message in the thread has a “Message-ID:”.

When we read the RFC it looks like the In-Reply-To field should at least refer to its direct parent message. We don‘t really have emails going back and forth between the application and our users. Instead, emails are only sent out from Great Books to users. This simplifies things because we can use the Message-ID of the last notification to make it seem like a reply email.

The trickiest part might be to find the Message-ID of the previously sent notification. So we will add two methods. One to retrieve the previous reviews and another to generate the In-Reply-To field. Remember that we only care about the reviews for a specific book, as all the reviews for that book will belong to a single email thread.

First we add the below method that retrieves the other reviews that belong to the thread.

# app/mailers/notification_mailer.rb 

def reviews_in_thread(review)
  Review.where.not(id: review.id)
        .where(book_id: review.book_id)
        .order(:created_at)
end

We will have to assume that each Great Books review belongs to a book so a review will have a book_id field. We retrieve all the previous reviews in the thread, not the current review we are notifying users about, and we order them by their created_at time. If there are no other reviews it will return an empty relation. If we do have other reviews the last review in the query result will be the most recent review.

Now we need to generate content for the In-Reply-To header field.

# app/mailers/notification_mailer.rb  

def in_reply_to(review)
  previous_review = reviews_in_thread(review).last
  message_id(previous_review)
end

We take only the last review as it will be associated with the last notification we sent for the currently reviewed book. The last notification is therfore the parent email of the notification we are currently sending out. We will soon combine the entire set of headers, but this concludes the In-Reply-To: field section.

Step 3: The `references` method

The last header we need to take care of is the References header. The RFC defines the References: field as follows:

“References:” field may be used to identify a “thread” of conversation.
…
The “References:” field will contain the contents of the parent's “References:” (if any) followed by the contents of the parent's “Message-ID:” (if any).  If the parent message does not contain a “References:” field but does have an “In-Reply-To:” field containing a single  message identifier, then the “References:” field will contain the contents of the parent's  “In-Reply-To:” field followed by the contents of the parent's “Message-ID:” field (if any).  If the parent has none of the “References:”, “In-Reply-To:”, or “Message-ID:” fields, then the new message will have no “References:” field.

It helps to sometimes draw a scenario out. So let us consider what this header will look like with each additional notification.

The first notification will not have a References: field because there won’t be a parent message. So we will send a notification where the email headers include only the Message-ID:. Let us imaging an email with this:

Message-ID: <notification-1-1634556591@example.com>

The next notification in the thread should have a References: field that contains the Message-ID of the previous message in this thread.

And finally, a third notification will contain the references of the second message, its parent message, followed by the message id of the second message.

The RFC documentation mentions that some implementations can make use of the References: field to display the “thread of the discussion”. From the examples above it is becoming clearer that it is possible to walk backwards through the “References:” field to find the parent of each message listed.

Adding a method that gives us the contents for the References: header is fairly easy. We already have a method that gives us all the other reviews in the thread. Remember that the reviews_in_thread method also sorts the reviews from earliest to latest.

# app/mailers/notification_mailer.rb  

def references(review)
  reviews_in_thread(review).map { |x| message_id(x) }.join(" ")
end

Notice that we separate the message ids with spaces.

Step 4: The `headers` method

Now we have a method for each of the three headers that we are interested in. To bring them all together we will add a very appropriately named method called headers.

def headers(review)
  {
    in_reply_to: in_reply_to(review),
    message_id: message_id(review),
    references: references(review)
  }.reject { |_, v| v.blank? }
end

This method can be added to the private section of our NotificationMailer class. Notice that it accepts and passes along the current review. It also removes any nil or empty headers with the use of the built-in Ruby reject method.

We are very close and the final thing left to do is to add the method that gets called when we want to send a notification.

Step 5: The `review_notification` method

Each ActionMailer class should contain at least one method that will send an email. You can read more about ActionMailer in the ruby on rails guides opens a new window . We are going to add a new method and call it review_notification. This method will accept a recipient and a review. With a recipient and a review we will be able to send an email with the headers required for threading.

def review_notification(recipient, review)
  @review = review
  @recipient = recipient
  mail_settings = {
    to: "#{recipient.name} <#{recipient.email}>",
    subject: "Review: #{review.book.title}"
  }.merge(headers(review))
  mail(mail_settings)
end

We create the instance variables to pass data along to our mailer views. Then we set the recipient name and email followed by the email subject line. The subject line is not important right now, but we will talk a bit more about it later on. Finally we merge the headers we already created.

Step 6: Send emails

To send an email we need to call the mailer opens a new window . We can make use of deliver_now or deliver_later but the purpose and implementation of async queues are out of scope for this post. So we will make use of deliver_now

We typically want to send emails from a controller or services object and often with the use of some queuing service. Sending an email will look something like this:

NotificationMailer.review_notification(recipient, review).deliver_now

Gmail is not threading

When we send emails using this mailer we notice that Apple Mail app threads perfectly. But the celebration is only short lived because emails are not threaded in Gmail.

From the official google feed/blog opens a new window we learn that if you receive two emails with the same subject from the same sender, these emails will not be threaded together unless one explicitly references the other (using the References: header).

It seems like we are adhering to this rule, so where do we go from here?

Threading Emails: The Github Approach

Going back to the drawing board we might decide to see how other service providers manage to send emails that thread correctly. One such a company is Github. If you have any notification in your inbox from Github then you will notice that they have done a very good job of threading emails.

If you look at the headers for a few messages in a Github thread, you will notice that The very first email in a thread has no In-Reply-To: or References: header fields. It contains only the Message-ID:.

Message-ID: <orgname/project/pull/id@github.com>

The next email in the thread will have In-Reply-To: and References: fields that refer to the first Message-ID:. And all following emails in the thread will have the exact same In-Reply-To: and References: fields:

This implementation is clearly different from how the RFC instructs us to set our headers for threading to work. But if this approach is good enough for Github then surely it is good enough for Great Books.

Now it is refactor time.

Step 1: Add `thread_id`

To make sure we are all on the same page, let us examine the headers from the Github emails. From the second mail onwards all emails in the thread have the same In-Reply-To: and References: fields. Both of these header fields refer to the same single Message-ID. There is no header called the Thread-Id but we will call this message id the thread id for ease of reference.

The thread id refers to the first review notification so we will use the first review in a thread when we construct the thread id.

# app/mailers/notification_mailer.rb  

def thread_id(review)
  first_review = reviews_in_thread(review).first
  message_id(first_review)
end

Our thread id is a message id that refers to the first review in a thread. No matter which review in the thread we look at, the thread id will always be the same. Except for the very first review. The first review notification will not have a In-Reply-To: or References: field

Remember that reviews_in_thread will return an empty relation if there are no other reviews in a thread. In that case first_review will be nil and we already made sure that message_id returns an empty string when a nil review is passed in.

Step 2: Update `in_reply_to` and `references`

# app/mailers/notification_mailer.rb  

def in_reply_to(review)
  thread_id(review)
end

def references(review)
  thread_id(review)
end

Both methods now return the same value. Which in most people’s books is a code smell. When we look at the headers method we realise that these two methods are “man in the middle” methods. So we can actually update the headers method like so:

def headers(review)
  {
    in_reply_to: thead_id(review),
    message_id: message_id(review),
    references: thead_id(review)
  }.reject { |_, v| v.blank? }
end

We no longer use the in_reply_to or references method so we can remove them 🎉.

Step 3: Do we need the `Message-ID`?

You might find that your email service provider overwrites or ignores the message_id that you provide. If that is the case for you then there is no point in providing the message_id, as it will be overwritten. Does that mean we have wasted our time up to this point?

The good news is that not all is lost! We don’t need to pass the message id for threading to work. As long as our In-Reply-To: and References: fields are consistent our emails will thread.

We can now simplify our NotificationMailer class. We can remove the message_id header. And now the headers method becomes.

def headers(review)
  {
    in_reply_to: thead_id(review),
    references: thead_id(review)
  }.reject { |_, v| v.blank? }
end

To illustrate what our emails will look like now, let us consider the following thread_id.

  thread_id = "<notification-3-1634824271@example.com>"

The first email in a thread will not have References: or In-Reply-To: fields. But from the second email onwards the References: and In-Reply-To: fields will both use the thread_id as their value.

We have a very simple mailer class and we have emails that thread, what a great day. I am sure we can optimise this class even further but I will leave it as it is for now.

Does the subject line matter?

The subject line does matter. Sending emails with consistent In-Reply-To: and References: fields will not thread correctly if we don’t use the same subject line. Documentation on the internet makes it seem like all we need is the email headers for threading to work, but trial and error has proved differently.

So Great Books will use the same subject line for all emails that belong in the same thread.

Final thoughts

We took a detour but landed on a very elegant looking mailer class. We learned that we cannot always follow the specifications that an RFC provides. The RFC was a good starting point, but we could have saved ourselves a bit of effort if we learned from those (like Github) that came before us.

Email threading is a great tool for improving the user experience of our applications.

Contact OmbuLabs to hear how our expertise can help you skip right to the elegant solutions. We save our clients time and deliver products that provide great user experiences.

Further Reading

• First Approach Gist opens a new window
• Final Approach Gist opens a new window
• RFC 5322 opens a new window
• Understanding email headers blog post I opens a new window
• Understanding email headers blog post II opens a new window
• Cyclescape repo that demonstrates threading opens a new window