Why hostname is so important in mail system
When setting up an email system, the first step is to make sure you have a correct hostname for mail server.
For example, your mail server has this IP: 1.2.3.4, and you have this domain: example.com. You should assign a DNS record which specify a domain name to the IP. Generally it's A record with the hostname mail.example.com.
mail.example.com. 300 IN A 1.2.3.4
Then you should request your hosting provider to assign a PTR record for this IP, pointing to the hostname mail.example.com as well. This calls reverse DNS.
4.3.2.1.in-addr.arpa. 3600 IN PTR mail.example.com.
These A and PTR records are so important. Without them you most probably can't send messages to other mail servers.
For most MTA servers (say it's Postfix) they have these settings:
smtpd_sender_restrictions = reject_unknown_reverse_client_hostname, reject_unknown_sender_domain
The statement "reject_unknown_reverse_client_hostname" requires sender IP has correct PTR record. The statement "reject_unknown_sender_domain" requires sender domain (i.e, example.com) has either MX or A record.
If the peer MTA has more strict setting like this:
smtpd_sender_restrictions = reject_unknown_client_hostname
It will reject messages when 1) the client IP address->name mapping fails, or 2) the name->address mapping fails, or 3) the name->address mapping does not match the client IP address.
So in general speaking your A and PTR records should keep consistent. PTR points to the hostname which has just the same IP in its A record.
Besides DNS records you should also setup the system hostname. In linux OS it's the configuration file /etc/hostname. After putting correct hostname into this file you could run the following command manually to make hostname available immediately.
$ sudo hostname mail.example.com
In Postfix's configuration file /etc/postfix/main.cf, you will also have this setting for hostname:
myhostname = mail.example.com
The system hostname is important because when your mail server talks to other mail servers, it will use hostname as HELO host.
If you are using a fake hostname, the HELO session will most probably get rejected by peer MTAs.
More further, if you want to setup SSL for mail server, hostname is specially useful. When you issue certificates to mail server, they are differed by hostnames.
Given the case you would like to setup SSL certificates for mail.example.com. This can be done by Letsencrypt and Certbot.
After deploying certificates to mail.example.com, the clients such as thunderbird can connect to mail server via secure way.
You must issue certificates by providing the correct hostname, otherwise cliet connections will break due to secure risk. Though SMTP session between MTAs may not validate hostname for certificates, but MUA clients as thunderbird do verify hostname.
To be additional, besides hostname your IP should keep clean for successful delivery. It should be listed in no RBLs. Most MTAs will check RBL against sending IP. For example, considering Postfix configuration below.
smtpd_recipient_restrictions =
check_policy_service unix:private/policyd-spf,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client bl.spamcop.net
It will check either SPF or RBL for sending IP. So you should make a correct SPF record for your sending IP. And check if your IP is listed in any RBLs. You could check RBLs manually from this URL,
Please also reference the following data.