Installing and setting up a correct email system is not easy. There are lots of open source systems to be integrated to work together.
Here are the documents I used to setup my mail system. Some were written by me, some are by others.
First, we need the basic functioning system which has MTA, LDA, SASL, IMAP modules, and a webmail included. Please reference my article below.
The next, you may want to setup virtual mailboxes and virtual aliases, read my following article.
The next, you would take time to setup sender reputation, including SPF, DKIM and DMARC. There are third party systems and modules for implementation. For SPF checks, you could read my following article.
I run two policyd services to postfix, one is for SPF checks, another is for sending rate.
For DKIM you could use OpenDKIM, for which the article is made by Linuxbabe.
This article has also included the stuff for SPF.
And this is the article by me for how to setup DMARC with postfix,
The next, you want to setup the basic antispam systems for your mailserver. Without antispam you can't have a reliable mailserver.
The following article is made by me for how to enable RBL checks in postfix.
In this article I use postscreen for DNSBL checks, and I only used spamhaus.org as RBL source.
Also read Linuxbabe's following article.
If you need deep antispam, a milter system should be integrated for email content analysis, such as Spamassassin or Rspamd. For Rspamd installaton and integration please read this article,
If you prefer Spamassassin, then reference this one by Linuxbabe.
Linuxbabe does write a serial of good articles for how to install and setup an email system, see the URL below.
The last, I wish you can setup a correct mailserver for yourself. That's full of challanges, but fun enough.