What is a mail server installation on a VPS

Installing a mail server on a VPS gives full control over email and allows it to be configured to meet the needs of a specific project. This solution is suitable for those who want to use mailboxes on their domain, manage internal correspondence, and ensure maximum data protection.

A mail server on a VPS eliminates dependence on external services. You can create any number of mailboxes, set individual parameters for processing emails, enable encryption, filtering, auto-replies, and forwarding. This provides flexibility and simplifies integration with other systems on the server.

With proper configuration using SPF, DKIM, and DMARC records, emails are delivered directly to the inbox and are not blocked by filters. This is important for notifications, newsletters, order processing, and customer support. Having your mail server allows you to work autonomously, improve delivery reliability, and control all processes at the system level.

Preparing the VPS and choosing mail software

Before installing a mail server, it is important to properly prepare the VPS. First, you need to choose a suitable plan based on the volume of mail traffic, the number of mailboxes, and storage capacity. QHost offers VPS plans that allow you to deploy a mail system with the necessary parameters. For stable mail operation, a clean operating system, a stable network connection, a static IP address, and the ability to configure DNS records are required.

After installing the OS, you need to update the packages and configure basic security: change the SSH port, create a user with limited rights, enable the firewall, and install antivirus software. Then make sure that ports 25, 465, and 587 are open and not blocked by the provider.

The next step is choosing the mail software. Among popular solutions:

1. Postfix – a simple and reliable mail agent with extensive configuration options;
2. Exim – a flexible system with advanced email processing logic;
3. Mail Server from ISPmanager – a solution with a graphical interface that simplifies setup.

For full functionality, you also need to install an IMAP/POP3 server (e.g. Dovecot), a webmail system (Roundcube, RainLoop), and protection components (SpamAssassin, ClamAV).

Installing and configuring a mail server on a VPS

After preparing the VPS, proceed with installing the mail software. First, update the system and install the necessary components: the mail agent (e.g. Postfix), IMAP/POP3 server (usually Dovecot), as well as modules for security and email filtering. Installation can be done manually through the console or via a control panel if provided by the hosting service.

Then configure the basic parameters for sending and receiving emails: domain, mailboxes, limits, log paths, and storage folders. Specify the server address, allowed ports, and authentication parameters. For reliable message delivery, enable encryption protocols (STARTTLS, SSL), activate SMTP authentication, and enable virus and spam checks.

At this stage, it is also important to configure DNS records:

• MX – points to the domain’s mail server;
• A and PTR – must match and point to the VPS IP;
• SPF – defines which servers are allowed to send mail;
• DKIM – signs messages, confirming their authenticity;
• DMARC – sets the policy for handling unauthorized messages.

After applying all settings, it is recommended to test sending and receiving emails, check operation via a mail client, and ensure that messages are not going to spam.

Security configuration: SPF, DKIM, DMARC

For reliable mail delivery and protection against sender spoofing, it is necessary to configure three mechanisms: SPF, DKIM, and DMARC. They form a system for verifying the authenticity of outgoing messages and help prevent emails from being marked as spam. These records work together.

SPF (Sender Policy Framework) – this is a TXT record in DNS that specifies the servers allowed to send mail on behalf of your domain. If a message comes from another address, it may be rejected or marked as suspicious. Example record:

DKIM (DomainKeys Identified Mail) – adds a digital signature to each outgoing email. The recipient verifies the signature against the public key published in DNS. This confirms that the message has not been altered in transit. Example DNS record:

DMARC (Domain-based Message Authentication, Reporting and Conformance) – defines a policy for handling messages that fail SPF or DKIM checks and allows receiving reports about their rejection. Example record:

Testing and maintaining the mail server

After completing the installation and configuration of the mail server, it is necessary to ensure it operates correctly. This involves checking sending and receiving emails via a mail client, web interface, and contact form on the website. Special attention should be paid to the correctness of DNS records: MX, SPF, DKIM, and DMARC must be correctly configured and recognized by external services.

To verify mail delivery, you can use third-party tools such as Mail-Tester or MXToolbox. These tools help assess whether emails are being delivered to the inbox, pass authentication, and identify which technical parameters need improvement.

It is also important to monitor mail server operation over the long term. This includes regular log checking, monitoring the mail queue, system load, and disk space. If necessary, set up alerts for failures, restrictions, or limit exceedances.

It is recommended to periodically update server software components, check certificate validity, and monitor DMARC reports.

FAQ

Why install your mail server on a VPS when ready-made services are available?

Installing your mail server on a VPS provides full control over data, high confidentiality, and flexible configuration. It helps avoid the limitations of ready-made services in terms of email volume, number of mailboxes, and functionality, and can reduce long-term costs with high email volumes.

What are the popular software stacks used to build a mail server?

Popular stacks for building a mail server often include an MTA (Mail Transfer Agent) such as Postfix, an MDA (Mail Delivery Agent) like Dovecot for IMAP/POP3 access, and anti-spam/antivirus solutions (e.g. SpamAssassin, ClamAV). Roundcube or RainLoop are typically used for the webmail interface.

What are the minimum VPS requirements to install a mail server?

Minimum VPS requirements for installing a mail server typically include at least 1 GB of RAM, 20–30 GB of disk space, and 1 CPU core. It is also important to have a dedicated IP address and the ability to configure DNS records (MX, SPF, DKIM, DMARC) for proper mail operation.