How to Set Up an Email Server on a VPS?

Running your own email server usually comes from a need for more control, privacy, and flexibility. Many users turn to a VPS after finding hosted services too limiting or expensive over time. When compared to regular hosting, a VPS gives you dedicated space to manage everything on your terms. 

Whether you’re aiming for customization or full control, an email server on a VPS is a worthwhile option. We understand its compatibility with enterprise requirements and created a guide explaining how to build and manage an email server on a VPS step by step. Besides, you will get hands on to the key details that help ensure your emails are delivered reliably instead of ending up in spam. 

Key Takeaways from the Article

• A VPS email server gives complete control of the email infrastructure and data.

• Dovecot and Postfix are a good system when it comes to receiving and sending emails.

• Email deliverability depends on the use of DNS authentication (SPF, DKIM, DMARC).

• The dedicated IP enhances trust and minimises spam status.

• Email hosting needs to be monitored and maintained.

What is an Email Server?

An email server is a system that handles the sending, receiving, and storage of email messages. It uses protocols like SMTP for delivery and IMAP or POP3 for retrieval, working in the background every time an email is sent or opened.

When you host an email server on a VPS, you run this entire system on your own virtual private server instead of relying on providers like Gmail or Outlook. This gives you full control over how emails are stored, secured, and delivered, while keeping all data within your own infrastructure.

Why Self-Host Email on a VPS?

Full Control Over Infrastructure

With a VPS, you manage every aspect of the email system. You decide how emails are routed, stored, and archived. This level of control is not available with most hosted solutions.

Privacy and Data Ownership

Your emails remain on your server. No third-party provider scans or analyses your data. This is especially important for sensitive communication.

Long-Term Cost Efficiency

Subscription-based email services charge per user. As your team grows, costs increase, and a VPS allows you to host multiple accounts under a single predictable cost.

Customization and Flexibility

You can create custom rules, filters, and workflows, and it is useful for businesses with specific operational requirements.

Compliance and Regulatory Needs

Certain industries require strict control over data storage and processing. A self-hosted email server helps meet those requirements.

Prerequisites for Setting Up an Email Server on a VPS

Minimum Requirements

• VPS with at least 2 GB RAM (4 GB recommended for better performance, especially if you plan to host multiple mailboxes or handle higher email volume)

• NVMe or SSD storage for faster data access and improved mailbox performance

• Dedicated IPv4 address with a clean reputation to avoid deliverability issues

• An Ubuntu or Debian-based operating system for better compatibility with mail server tools

A higher-spec VPS ensures smoother operation, faster email processing, and better stability over time. It also reduces the chances of performance bottlenecks as your email usage grows.

Domain and DNS Access 

A domain name is essential for email hosting. It acts as your identity in the email ecosystem and plays a major role in how receiving servers evaluate your emails.

You must also have:

• Full control over DNS settings so you can create and update records when needed

• Ability to create A, MX, TXT, and PTR records for proper email routing and authentication

In addition, you should be able to verify DNS changes and monitor propagation. DNS configuration plays a critical role in email authentication and directly affects whether your emails reach the inbox or the spam folder.

Skills and Knowledge

One should have some fundamental knowledge of the Linux command line. You must feel at ease with command usage, package installation, and configuration editing through such tools as nano or other editors. The process will be very easy with the knowledge of networking concepts like ports, IP addresses, and DNS. It can also be used to learn the interaction of email protocols such as SMTP, IMAP and POP3. The slightest degree of troubleshooting skills is also handy, as email servers frequently need to be tweaked and tracked once in place.

Step-by-Step Process to Set Up an Email Server on a VPS

Step 1: Update Your Server and Set the Hostname

Start by connecting to your VPS via SSH and updating all packages:

sudo apt update && sudo apt upgrade -y

Next, set the server's fully qualified domain name (FQDN). It must match the hostname you will use for your mail server:

sudo hostnamectl set-hostname mail.yourdomain.com

Edit /etc/hosts and add a line mapping your server's IP to the FQDN:

127.0.0.1   localhost

YOUR_SERVER_IP   mail.yourdomain.com   mail

Verify the hostname with hostname -f. It should return mail.yourdomain.com.

Step 2: Install Postfix (SMTP Server)

Postfix handles the sending and receiving of email via SMTP. Install it with:

sudo apt install postfix -y

During installation, select Internet Site when prompted for the mail configuration type. Enter your domain name (e.g., yourdomain.com) as the system mail name.

Postfix is the Mail Transfer Agent (MTA), which routes email between servers. It listens on port 25 for incoming mail and uses port 587 for authenticated outgoing mail submissions.

Step 3: Install Dovecot (IMAP/POP3 Server)

Dovecot provides mailbox access for email clients via IMAP and POP3. Install the core package and the IMAP module:

sudo apt install dovecot-core dovecot-imapd -y

While Postfix handles mail transport between servers, Dovecot handles mail retrieval, which allows you (or your users) to read emails through a client like Thunderbird, Outlook, or a mobile app. Postfix delivers mail to the server; Dovecot serves it to the user.

Step 4: Configure Postfix

Open the Postfix main configuration file at /etc/postfix/main.cf and set the following directives:

myhostname = mail.yourdomain.com

mydomain = yourdomain.com

myorigin = $mydomain

mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain

inet_interfaces = all

inet_protocols = ipv4

home_mailbox = Maildir/

smtpd_sasl_type = dovecot

smtpd_sasl_path = private/auth

smtpd_sasl_auth_enable = yes

The home_mailbox = Maildir/ directive tells Postfix to store each user's email in their home directory using the Maildir format, which stores each message as a separate file. It is more reliable than the older mbox format, especially for IMAP access.

After editing, restart Postfix: sudo systemctl restart postfix

Step 5: Configure Dovecot

Edit /etc/dovecot/dovecot.conf to enable the IMAP protocol:

protocols = imap

Set the mail storage location in /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:~/Maildir

Configure authentication in /etc/dovecot/conf.d/10-auth.conf:

auth_mechanisms = plain login

Set up the authentication socket for Postfix in /etc/dovecot/conf.d/10-master.conf. Inside the service auth block, add:

unix_listener /var/spool/postfix/private/auth {

  mode = 0660

  user = postfix

  group = postfix

}

Restart Dovecot: sudo systemctl restart dovecot

Step 6: Create Mail User Accounts

Each email account maps to a system user. Create a new user and set their password:

sudo adduser john

sudo passwd john

It creates the user [email protected]. The Maildir directory will be created automatically when the first email arrives. For production environments with many users, consider setting up virtual mailbox users with a database backend (MySQL or PostgreSQL) instead of system accounts.

Step 7: Secure with SSL/TLS Using Let’s Encrypt

Unencrypted email connections are a security risk and a deliverability problem; many receiving servers will reject or flag mail sent without TLS. Install Certbot and obtain a free SSL certificate:

sudo apt install certbot -y

sudo certbot certonly --standalone -d mail.yourdomain.com

Once the certificate is issued, update Postfix to use it. Add these lines to /etc/postfix/main.cf:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem

smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

smtpd_use_tls = yes

smtpd_tls_auth_only = yes

Update Dovecot's SSL configuration in /etc/dovecot/conf.d/10-ssl.conf:

ssl = required

ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem

ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

Restart both services:

sudo systemctl restart postfix dovecot

Enable the submission port (587) for outgoing mail by uncommenting the submission block in /etc/postfix/master.cf. The port is used by email clients for authenticated sending. For IMAP over TLS, clients connect on port 993.

Set up a cron job to auto-renew the certificate:

sudo certbot renew --dry-run

5 DNS Records You Must Configure

A Record and MX Record

• A Record: Links your mail server hostname (such as mail.yourdomain.com) to your VPS IP address. It ensures that your domain points to the correct server.

• MX Record: Directs incoming email traffic to your mail server. It tells other servers where to send emails for your domain.

Both records must be correctly configured and aligned. For example, your MX record should point to a hostname that has a valid A record. Any mismatch can lead to failed delivery or delays.

PTR Record

PTR record or reverse DNS (rDNS) is used to reverse map the IP address of your server to your domain name. It acts as a trust signal for receiving mail servers, confirming that your server is legitimate. The PTR checks are done by most major email providers prior to accepting messages. In case it is missing or not the same as your hostname, there is a chance that your emails will get rejected or spammed. PTR configuration will have to be requested of your VPS provider in most cases, since it is configured to operate at the IP level and not within your domain DNS configuration.

SPF 

SPF (Sender Policy Framework) is used to specify the servers that have permission to issue emails on your domain. It is recorded in your DNS under TXT. Upon receiving an email, a receiving server will check the SPF record to see whether the IP address that is sending the email is authorized. The email can be considered suspicious or discarded if the IP is not listed. An SPF record is well configured and helps minimize the danger of spoofing and enhances the overall credibility of your domain in the long run.

DKIM

DKIM (Domain Keys Identified Mail) is a protocol that is added to all the outbound emails, and it forms a digital signature for the email. This signature is created on your server with the private key, and the matching public key is kept in your DNS. When an email is received, the server will compare the signature to the public key. When they are identical, it is a testament to the fact that the email content has not been distorted in transit. DKIM is a significant component that maintains the integrity of emails and is often required to overcome existing spam filters.

DMARC 

DMARC (Domain-Based Message Authentication, Reporting and Conformance) builds on SPF and DKIM by defining how receiving servers should handle emails that fail authentication checks. It allows you to set policies such as:

• None (monitor only)

• Quarantine (send to spam)

• Reject (block completely)

Moreover, DMARC also offers reporting capabilities, and thus, you can keep track of how your domain is currently being utilised and find possible abuse. An adequately set DMARC record enhances your email security, and the overall deliverability is greatly improved.

Test Your Email Server

Send a Test Email

Start with a simple test from your VPS. It helps confirm that your server can send outgoing emails successfully.

echo "Test Email" | mail -s "Test" [email protected]

Send the email to an external address such as Gmail or Outlook. After sending, check whether the message arrives and how long it takes.

If the email does not arrive, it usually points to a configuration issue. It could be related to Postfix settings, blocked ports, or missing DNS records. Even if the email arrives in the spam folder, that still gives you useful information about what needs improvement. 

Verify with MXToolbox

After sending a test email, the next step is to check your domain and server using external tools. MXToolbox is one of the most widely used platforms for this purpose. It allows you to:

• Check if your MX records are correctly configured

• Verify whether your mail server is reachable

• Identify if your IP address is listed on any blacklists

• Test basic SMTP connectivity

Running these checks gives you a clear picture of how other servers see your setup. Even if everything looks correct locally, MXToolbox may highlight issues that are not immediately visible.

Check Email Headers

When your test email arrives, do not stop at simply reading it. Open the email headers and review the authentication results. Most email providers include a section that shows:

• SPF status (PASS or FAIL)

• DKIM status

• DMARC result

If all three show PASS, your configuration is on the right track. If any of them fail, it indicates a problem with your DNS or server setup. Headers may look technical at first, but they provide valuable insight. 

IP Reputation and Blacklist Monitoring

Email delivery does not depend only on configuration. Your server’s IP reputation plays a major role as well. Even a perfectly configured server can struggle if the IP address has a poor history. Start by checking whether your IP is listed on any blacklists. Tools like MXToolbox can help with that. If your IP appears on a blacklist, you need to:

• Identify the reason for listing

• Fix the underlying issue (such as misconfiguration or suspicious activity)

• Submit a removal request

In addition, avoid sending large volumes of emails right after setup. A new IP needs time to build trust.

Common Pitfalls and How to Avoid Them?

Port 25 Blocked by Provider

Blocked SNMP traffic in port 25 is one of the most frequent problems new users encounter. This is a default port blocking of many VPS providers to block spam and abuse on their network. When port 25 is unavailable, your server will not be capable of sending emails to other mail servers.

There are two alternatives to this. To begin with, you may call your VPS provider and ask him to open port 25. Others permit it upon approval of your use case. Second, you can make your server authenticate with port 587, which is generally acceptable for email sending.

rDNS not Configured

Reverse DNS, or PTR record, is often overlooked during setup, yet it plays a major role in email trust. When your server sends an email, the receiving servers will look to see whether your IP address is connected back to a legitimate domain name. If rDNS is not present and/or not equal to your hostname, most email servers will consider your mail as spam.

To configure rDNS, it is often necessary to request your VPS service provider to do it because it is done at the IP level and not in your domain DNS configuration. After configuring, make sure that your hostname and PTR record are identical.

Missing DMARC

Users often configure SPF and DKIM and leave DMARC alone, with the assumption that it is optional. As a matter of fact, DMARC is a significant component that makes all the difference. Lacking DMARC, receiving servers lack explicit guidelines on how to respond to authentication failures. 

A proper DMARC policy not only improves security but also gives you visibility into how your domain is being used. You can receive reports that show authentication results and potential misuse.

No SSL/TLS

Without SSL/TLS encryption, your email traffic is exposed during transmission, which creates both security and trust issues. Modern mail servers expect encrypted connections. If your server does not support TLS, some providers may refuse connections or downgrade your trust level.

Using tools like Let’s Encrypt makes it easy to generate and renew certificates. Once installed, ensure that both Postfix and Dovecot are configured to use them properly.

Sending Bulk Email from a Fresh IP

A newly set-up email server starts with no reputation. From the perspective of other mail providers, your IP address is unknown. Sending a large number of emails immediately can raise red flags. This often leads to emails being flagged as spam or your IP getting listed on blacklists. Recovering from such situations can take time and effort.

Instead, start slowly. Send a small number of emails and increase the volume gradually over several days or weeks. This process, often referred to as warming up the IP, helps build trust with receiving servers.

When Self-Hosting Email Makes Sense (and When it Doesn’t)?

Self-hosting email is a worthwhile option if you know how to manage a server and have the discipline. It works well for teams with Linux skills and professionals who can keep up with ongoing maintenance. From privacy to regulatory requirements, a self-hosted email addresses all the concerns by offering full control over the email data.

However, it is not for everyone. If no one on your team can troubleshoot DNS issues, monitor IP reputation, or apply security patches regularly, a managed service will save you from headaches that compound over time. Email downtime affects everything from client communication to internal workflows, so the decision should be practical, not ideological.

 

 

The honest answer is that self-hosting gives you more control but asks more of you in return. If you are comfortable with that trade-off and have the skills to back it up, it is a strong option. If not, there is no shame in choosing a managed service and focusing your energy elsewhere.

The Bottom Line 

Running an email server on a VPS offers unmatched flexibility and control. However, it requires careful setup and continuous management. Every component, from DNS to security, plays a role in overall success. For reliable email hosting, choose a KVM VPS with NVMe storage and a dedicated IP by Hostsailor. It ensures better performance, stronger isolation, and consistent deliverability, essential for running a stable email server. 

Frequently Asked Questions About Self-Hosting Email on VPS

What is the difference between Postfix and Dovecot?

Postfix is responsible for sending and routing emails using SMTP. It ensures messages move between servers correctly. Dovecot handles email access, allowing users to read messages via IMAP or POP3. Together, Postfix delivers emails, while Dovecot manages how users receive and interact with them.

What is the minimum VPS spec for an email server?

A VPS with 2 GB RAM and 20 GB of SSD or NVMe storage can handle a basic Postfix and Dovecot setup. However, if you plan to run spam filtering tools like SpamAssassin alongside it, 4 GB RAM and 40 GB storage give you more breathing room.

What are SPF, DKIM, and DMARC?

SPF defines which servers are allowed to send emails from your domain. DKIM adds a digital signature to verify message integrity. DMARC builds on both, setting rules for failed authentication. These three work together to improve trust, prevent spoofing, and increase email deliverability.

How do I stop my VPS emails from going to spam?

Ensure SPF, DKIM, and DMARC are properly configured. Use a dedicated IP with a clean history. Set up reverse DNS correctly. Start with low email volume and increase gradually. Monitor blacklists regularly and avoid spam-like content to maintain a strong sending reputation.

Can my VPS provider block port 25?

Yes, many VPS providers block port 25 by default to prevent spam abuse. You can request unblocking through support, depending on their policy. Alternatively, you can use port 587 with authentication, which is commonly accepted for sending emails securely.

Do I need a dedicated IP for a VPS email server?

Yes, a dedicated IP is strongly recommended. Shared IPs carry the reputation of all users on that address. If another user sends spam, your deliverability suffers. A clean dedicated IP helps build and maintain sender trust.