Lab 6 - Email lab¶

Welcome to the mail lab. Here is a quick action list of what we will be doing in the lab:

1. Validation of previous week's tasks
2. Setting an MX record in your DNS server
3. Email
   1. MTA and SMTP
   2. IMAP and POP3
4. Postfix
   1. Setting up an SMTP and MTA server
   2. Send local email
   3. Open ports for remote sending
   4. Setup Postfix to allow for remote email
   5. Test sending to domains
5. Dovecot
   1. Setting up an IMAP server
   2. Testing IMAP login
6. Setting up a web service for mailbox access

1. Validation of previous week's tasks¶

Please make sure the following is true before starting this lab:

• Personal domains are configured. Machine is accessible over .sa.cs.ut.ee inside the University network.
• All the tests on scoring.sa.cs.ut.ee are green.
• The DNS of your personal computer is set up correctly and can resolve sa.cs.ut.ee domains.
• Apache web server is up and running.

2. Setting up an MX record in your DNS server¶

Sending email between servers utilizes an MX or mail exchange record. This record defines which server on which IP address is responsible for accepting a domain's email. If there is no MX record, it is impossible to send emails to the domain.

Our goal is to set up the MX records so that the emails for the <vm_name>.sa.cs.ut.ee domain will be directed to mail.<vm_name>.sa.cs.ut.ee, which in turn points to the IP of your virtual machine - this means that you will need to create an A record pointing to your mail server.

In the case of MX records, we also have something called priority. Priority is an extra parameter in the DNS record which allows you to set, in the case of multiple MX servers, which servers are more responsible than others. The format is the following:

<vm_name>.sa.cs.ut.ee. IN MX 0 mail.<vm_name>.sa.cs.ut.ee.

Where 0 stands for the highest priority possible. If you specify multiple servers here, each with different priority values, then the lowest one will be receiving the most email.

Here are some extra materials for MX records:

• MX Record
• Wikipedia: MX Record

3. Email in general¶

One would think explaining email is unnecessary because it's being used everywhere. Sadly, you would be surprised how basic and insecure email is, mainly due to its protocol for the sending/receiving part - SMTP.

The other part of the service is the protocol that allows email clients (webmail, outlook, thunderbird, etc.) to access mail inside the servers - IMAP. IMAP is a protocol that synchronizes email between the client and the server by occasionally polling and checking the inbox in the server.

3.1 SMTP and MTA¶

Within the Internet email system, a message transfer agent or mail transfer agent MTA or mail relay is software that transfers electronic mail messages from one computer to another using Simple Mail Transfer Protocol or SMTP.

SMTP is a text-based protocol - the messages sent are represented in UTF8 text nowadays. Even files, images or encrypted messages are first converted into a binary message, and then that binary message is transformed into a textual representation before being added to an email.

There are three parts to an SMTP message:

1. The first is the headers, like "Subject", "From", and "To" in the picture. These define the message's metadata, including the sender and receiver address.
2. Then there's the "Body" or the actual message. This contains the message you usually see when opening the email.
3. In the end, there's an optional part for attachments. These are usually massive blocks of seemingly random text, which is the textual representation of the binary we talked about.

All email security works additively - methods are applied on top of this core methodology. For example, encrypted emails do what is being done with images - encrypt the message, convert to binary representation, convert to textual representation and finally add it to the body. If you read that encrypted email with an email client that does not support encryption, it will look like a scrambled mess of letters.

Network design-wise, the SMTP server uses several ports for different purposes:

• 25 - Primary SMTP relay port - used for submission as well, but not recommended nowadays.
• 465 - Used to be a valid secure SMTP (SMTPS) port. It should not be used nowadays.
• 587 - This is the default mail submission port. This uses a proper authenticated approach and can be encrypted.

We will be using port 25 until the TLS lab.

3.2 IMAP and POP3¶

IMAP stands for Internet Message Access Protocol, and it's the open standard that describes how to access messages in a remote email mailbox.

Due to SMTP being text-based, IMAP has to be text-based as well. The conversation between a client and a server looks something like this:

C: <open connection>
S:   * OK IMAP4rev1 Service Ready
C:   a001 login mrc secret
S:   a001 OK LOGIN completed
C:   a002 select inbox
S:   * 18 EXISTS
S:   * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
S:   * 2 RECENT
S:   * OK [UNSEEN 17] Message 17 is the first unseen message
S:   * OK [UIDVALIDITY 3857529045] UIDs valid
S:   a002 OK [READ-WRITE] SELECT completed
C:   a003 fetch 12 full
S:   * 12 FETCH (FLAGS (\Seen) INTERNALDATE "17-Jul-1996 02:44:25 -0700"
      RFC822.SIZE 4286 ENVELOPE ("Wed, 17 Jul 1996 02:23:25 -0700 (PDT)"
      "IMAP4rev1 WG mtg summary and minutes"
      (("Terry Gray" NIL "gray" "cac.washington.edu"))
      (("Terry Gray" NIL "gray" "cac.washington.edu"))
      (("Terry Gray" NIL "gray" "cac.washington.edu"))
      ((NIL NIL "imap" "cac.washington.edu"))
      ((NIL NIL "minutes" "CNRI.Reston.VA.US")
      ("John Klensin" NIL "KLENSIN" "MIT.EDU")) NIL NIL
      "<B27397-0100000@cac.washington.edu>")
      BODY ("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 3028
      92))
S:   a003 OK FETCH completed
C:   a004 fetch 12 body[header]
S:   * 12 FETCH (BODY[HEADER] {342}
S:   Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT)
S:   From: Terry Gray <gray@cac.washington.edu>
S:   Subject: IMAP4rev1 WG mtg summary and minutes
S:   To: imap@cac.washington.edu
S:   cc: minutes@CNRI.Reston.VA.US, John Klensin <KLENSIN@MIT.EDU>
S:   Message-Id: <B27397-0100000@cac.washington.edu>
S:   MIME-Version: 1.0
S:   Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
S:
S:   )
S:   a004 OK FETCH completed
C    a005 store 12 +flags \deleted
S:   * 12 FETCH (FLAGS (\Seen \Deleted))
S:   a005 OK +FLAGS completed
C:   a006 logout
S:   * BYE IMAP4rev1 server terminating connection
S:   a006 OK LOGOUT completed

There also is another protocol with the same purpose - POP3, but this one is much-much older and incapable of several essential features. It also wants to pull all the emails from the remote server to your device and will not leave a copy in the remote server. It is recommended never to use this protocol anymore.

IMAP network design by port usage:

• 143 - Standard, unsecured IMAP port.
• 993 - Secure IMAP (IMAPS) port.

4. Postfix¶

The first task will be to set up a service capable of sending and receiving emails. We use an open-source service called Postfix for that. It's the default MTA for many Linux distributions.

Postfix's documentation is available http://www.postfix.org/.

4.1 Setting up an SMTP and MTA server¶

Using the package manager, install a package named postfix.

The locations of the Postfix configuration and log files:

• Configuration file: /etc/postfix/main.cf
• Log file for e-mail related log messages: /var/log/maillog

4.2 Send local email¶

As a test, we can see if emails work, by sending an email from a user inside the VM (the user you run as) to a user inside the VM (the user specified in the command), by doing:

echo "text" | sendmail "root@localhost"

There will be no output from this command. But what will happen is that an email will be put into the root users' inbox.

You can check how the email was transferred from the mail log, /var/log/maillog:

Mar 14 08:35:38 b123123 postfix/cleanup[2440729]: EA7172402742: message-id=<20220314063538.EA7172402742@b123123.sa.cs.ut.ee>
Mar 14 08:35:38 b123123 postfix/bounce[2440732]: D41A62402740: sender delivery status notification: EA7172402742
Mar 14 08:35:38 b123123 postfix/qmgr[2432807]: EA7172402742: from=<>, size=1966, nrcpt=1 (queue active)
Mar 14 08:35:38 b123123 postfix/local[2440731]: EA7172402742: to=<root@b123123.sa.cs.ut.ee>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Mar 14 08:35:38 b123123 postfix/qmgr[2432807]: EA7172402742: removed

Should look something like that.

You can also check the root user's inbox to see if the email was received. User's inboxes are, if not configured otherwise, in /var/spool/mail/<username>. If you cat the file for root user's inbox, you should see the email you sent together with the headers and other information.

4.3 Open ports for remote sending¶

Even though we can send emails between the users inside the machine, sending between different devices is more complicated. The first thing to do is open up the mail submission port 25, but you can also open up the other SMTP ports ahead of time.

4.4 Setup Postfix to allow for remote email¶

We also need to configure Postfix to listen on all interfaces and be able to accept email for our domain before we can send email between machines.

After making sure all the ports are open, you can restart your postfix server.

Now we can test sending emails between the different users in your VM and also other machines.

4.5 Try sending to domains¶

5. Dovecot¶

We used a "local mailbox" model for testing the email service, but as you can see, it's not very convenient. Nowadays, reading email is mainly done over the network protocols such as IMAP and POP3. Most popular, by far, are web-based solutions like Gmail, but these also usually use IMAP behind the scenes.

5.1 Setting up an IMAP server¶

We will now set up an IMAP server called Dovecot. Install it with the package manager.

Main configuration file for dovecot is /etc/dovecot/dovecot.conf and because of the modular design each module has separate configuration file under /etc/dovecot/conf.d directory.

Details about Dovecot configuration files are available here: Dovecot documentation - Dovecot Config File Syntax

PS! Similar to postconf, there is a doveconf command for checking the Dovecot configuration. The manual man doveconf suggests that most of the switches are similar to the ones of the postconf command (-n, -d, -P).

5.2 Configuring dovecot authentication module¶

The most straightforward authentication mechanism is PLAIN. The client simply sends the password unencrypted to Dovecot. All clients support the PLAIN mechanism, but obviously, there's the problem that anyone listening on the network can steal the password. For that reason (and some others), other mechanisms were implemented. But this week, for learning, we will set it up without any security mechanisms.

One step of setting up Dovecot involves moving the incoming email from the /var/spool/mail location to now using something called maildir format. Instead of putting incoming emails to where they were before, they are now put into the user's home directory.

Also, let's enable automatic folder creation to prevent problems.

mailbox Trash {
  auto = create
  special_use = \Trash
}

Also, before we can start testing the IMAP login, open up the necessary ports from firewalls. You can check the netstat command or /etc/dovecot/conf.d/10-master.conf to see which ports to open.

5.2 Testing IMAP login¶

$ telnet 127.0.0.1 143
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

A1 login mailuser mailuser
A1 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY LITERAL+ NOTIFY SPECIAL-USE] Logged in

A1 logout
* BYE Logging out
A1 OK Logout completed (0.001 + 0.000 secs).

Now try connecting with telnet but instead of 127.0.0.1 connect to mail.<vm_name>.sa.cs.ut.ee As you can see, you get an error logging in:

* BAD [ALERT] Plaintext authentication not allowed without SSL/TLS, but your client did it anyway. If anyone was listening, the password was exposed.
a1 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.

The error is because, by default, dovecot requires encrypted connections from anyone trying to log in from outside. Of course, this is quite a sane thing to do to protect the users' private info from hackers who might be listening to the network traffic. We, however, want to disable encrypted connections for now and tackle traffic encryption in a later lab, as mentioned before. To allow login from unencrypted connections:

5.3 Give mailbox handling from postfix to dovecot¶

SMTP communication between mail servers uses smtp port 25. On the other hand, Mail clients often submit outgoing emails to a mail server on submission port 587.

That is because port 25 usually doesn't require authentication, which is fine if you want to receive mail to the mail server from other mail servers since it is practically impossible to have every mail server authenticate themselves to yours. However, relaying mail with no authentication is an absolutely horrible idea - it would allow anyone to send emails via your mail server. So let's get to configuring now.

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth

# Explanations:
#  -o syslog_name=postfix/submission: The name with which submission related events are logged
#  -o smtpd_sasl_auth_enable=yes:     Enable sasl authentication (`plain` in our case)
#  -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject: Permit recepients outside of $mynetworks only to authenticated users
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject: Allow mail relaying only for authenticated users
#  -o smtpd_sasl_type=dovecot:      Set dovecot as the SASL provider
#  -o smtpd_sasl_path=private/auth: Set authentication socket location

service auth {
    unix_listener /var/spool/postfix/private/auth {
      mode = 0600
      user = postfix
      group = postfix
    }
}

5.4 Recap and testing¶

We should be in a state where we have three main ports in use.

• Port 25 is just for relaying mail without authentication.
• Port 143 is used to view users' emails with the Dovecot protocol.
• Port 587 is used by email clients to submit email; this requires authentication.

This kind of setup is close to what is used in the real world. We are only missing TLS from our configuration, and we could become a fully-fledged email provider.

Before going forward, let's test whether the new setup works. Send an email to your mailuser user, and let's see if we can see it from the dovecot login.

$ telnet 127.0.0.1 143 # (1)
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

A1 login mailuser mailuser # (2) 
A1 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY LITERAL+ NOTIFY SPECIAL-USE] Logged in

b select inbox # (3)
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1647324753] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
b OK [READ-WRITE] Select completed (0.005 + 0.000 + 0.004 secs).

2 FETCH 1:* (FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (SUBJECT)]) # (5)
* 1 FETCH (FLAGS (\Seen) INTERNALDATE "15-Mar-2022 08:32:28 +0200" BODY[HEADER.FIELDS (SUBJECT)] {2}

)

3 FETCH 1 BODY[TEXT] # (4) 
* 1 FETCH (BODY[TEXT] {6}
my email body
)
3 OK Fetch completed (0.004 + 0.000 + 0.003 secs).

As you can see, our Dovecot works but is not that pleasant to actually use. Usually, this is solved by utilizing a desktop mail client, but nowadays, mail clients are particular about which servers they connect to and might fail when there's no TLS, so instead, let's set up a webmail service.

6 Setting up a web service for mailbox access¶

Roundcube is an open-source webmail client that you can use to access your email account. It works by connecting to the Dovecot service with your credentials and then mediates the connection between the client and Dovecot, giving it a friendly UI.

pecl install <package>

After that, set up a database for Roundcube to use.

MariaDB [(none)]> CREATE DATABASE roundcubemail /*!40101 CHARACTER SET utf8 COLLATE utf8_general_ci */;
MariaDB [(none)]> CREATE USER 'roundcube'@'localhost' IDENTIFIED BY 'the_random_password';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON roundcubemail.* TO 'roundcube'@'localhost';
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> exit;

Roundcube can be downloaded from its github repository. Currently the latest release is 1.6, so lets set it up in /var/www/html/roundcubemail.

wget https://github.com/roundcube/roundcubemail/releases/download/1.6.1/roundcubemail-1.6.1-complete.tar.gz -O /var/www/html/roundcube.tar.gz

Now that we have set up the necessary files and a database, we can create a web server configuration for Roundcube.

<VirtualHost *:80>
  ServerAdmin root@<vm_name>.sa.cs.ut.ee
  ServerName mail.<vm_name>.sa.cs.ut.ee
  DocumentRoot /var/www/html/roundcubemail 
  <Directory /var/www/html/roundcubemail>
      Allowoverride all
  </Directory>

  ErrorLog /var/log/httpd/mail.<vm_name>.sa.cs.ut.ee-error.log
  CustomLog /var/log/httpd/mail.<vm_name>.sa.cs.ut.ee-access.log combined
  ForensicLog /var/log/httpd/mail.<vm_name>.sa.cs.ut.ee-forensic.log
</VirtualHost>

After you have done all that, Roundcube setup page should be available from mail.<vm_name>.sa.cs.ut.ee/installer. Make sure that all the PHP extensions are OK. Once you have verified that Roundcube can find its extensions, click "Next" and set up Roundcube with the following information:

After this is all done, you should be able to access the mailboxes of the mail accounts you set up before by using the username and password of these users by going to the website you set up. If you cannot access the mailboxes, ensure that:

• The user exists
• Dovecot is running and configured correctly

If you could log in, try sending mail to another student's mailuser account from your mailuser. If you get an error, make sure that:

• The other student has their postfix up and running
• That you have your postfix service up and running and configured correctly.

7. Ansible¶

The most difficult part of this lab to automate is the management of the MySQL users and databases. Thankfully, Ansible has necessary utilities for this. Ansible has following modules

• mysql_db
• mysql_user

Which can manage MySQL easily. Examples:

- name: Create a new database with name '<database_name>'
  community.mysql.mysql_db:
    name: <database_name>
    state: present

- name: Create database user using password with all database privileges
  community.mysql.mysql_user:
    name: <database_user>
    password: '<database_password>'
    priv: '<database_name>.*:ALL'
    state: present

[client]

password = <root_pass>