Installing and setting up a correct email system is not easy. There are lots of open source systems to be integrated to work together.

Here are the documents I used to setup my mail system. Some were written by me, some are by others.

First, we need the basic functioning system which has MTA, LDA, SASL, IMAP modules, and a webmail included. Please reference my article below.

Install postfix dovecot squirrelmail letsencrypt on ubuntu

The next, you may want to setup virtual mailboxes and virtual aliases, read my following article.

How to setup virtual alias domains in postfix

The next, you would take time to setup sender reputation, including SPF, DKIM and DMARC. There are third party systems and modules for implementation. For SPF checks, you could read my following article.

Two policyd services I used for postfix

I run two policyd services to postfix, one is for SPF checks, another is for sending rate.

For DKIM you could use OpenDKIM, for which the article is made by Linuxbabe.

How to Set up SPF and DKIM with Postfix on Ubuntu Server

This article has also included the stuff for SPF.

And this is the article by me for how to setup DMARC with postfix,

How to setup OpenDMARC for postfix

The next, you want to setup the basic antispam systems for your mailserver. Without antispam you can't have a reliable mailserver.

The following article is made by me for how to enable RBL checks in postfix.

How to enable RBL checks for Postfix

In this article I use postscreen for DNSBL checks, and I only used spamhaus.org as RBL source.

Also read Linuxbabe's following article.

Blocking Email Spam with Postfix SMTP Server

If you need deep antispam, a milter system should be integrated for email content analysis, such as Spamassassin or Rspamd. For Rspamd installaton and integration please read this article,

Own mail server based on Dovecot, Postfix, MySQL, Rspamd and Debian 9 Stretch

If you prefer Spamassassin, then reference this one by Linuxbabe.

Block Email Spam with Postfix and SpamAssassin Content Filter

If you want to use port 465 as submission, please reference this article.

Enable SMTPS Port 465 in Postfix SMTP Server

Linuxbabe does write a serial of good articles for how to install and setup an email system, see the URL below.

Build Your Own Email Server on Ubuntu

The last, I wish you can setup a correct mailserver for yourself. That's full of challanges, but fun enough.