Complete Guide to Setting Up Mailu Email Server on Debian for Personal Domain

Introduction

Welcome to the definitive guide for setting up your own private email server using Mailu on Debian. As Sugan Siddhant Dodrai, I understand the importance of having complete control over your digital communications. This comprehensive 10,000-word guide will walk you through every step of creating a professional, secure email server for your personal domain.

Why Run Your Own Email Server?

Before we dive into the technical setup, let's explore why self-hosted email matters:

1. Privacy & Control: Your data remains yours

2. Brand Identity: Professional emails with your domain

3. No Limitations: Avoid storage restrictions and sending limits

4. Cost-Effective: Long-term savings over hosted solutions

5. Learning Experience: Valuable sysadmin skills

Prerequisites

Hardware Requirements

• Minimum: 2GB RAM, 2 CPU cores, 20GB storage

• Recommended: 4GB RAM, 4 CPU cores, 50GB storage

• Storage: SSD highly recommended for database performance

Software Requirements

• Debian 11 (Bullseye) or Debian 12 (Bookworm)

• Docker and Docker Compose

• Domain name with DNS access

• Static IP address (or dynamic DNS solution)

Domain Preparation

1. Purchase a domain (if you haven't already)

2. Ensure you have access to DNS management

3. Decide on your primary domain (e.g., yourdomain.com)

Part 1: Initial Server Setup

Step 1: Debian Installation and Updates

bash

# Update package lists
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y curl wget git vim htop net-tools \
  ufw fail2ban gnupg software-properties-common

# Set your hostname
sudo hostnamectl set-hostname mail.yourdomain.com

# Update /etc/hosts
echo "127.0.0.1 mail.yourdomain.com" | sudo tee -a /etc/hosts

Step 2: Create a Non-Root User

bash

# Create new user
sudo adduser sugan
sudo usermod -aG sudo sugan

# Switch to new user
su - sugan

Step 3: Configure SSH Security

bash

# Edit SSH configuration
sudo vim /etc/ssh/sshd_config

# Important changes:
# Port 2222 (change from default 22)
# PermitRootLogin no
# PasswordAuthentication no (use SSH keys)
# AllowUsers sugan

# Restart SSH
sudo systemctl restart sshd

Step 4: Configure Firewall

bash

# Enable UFW
sudo ufw enable

# Allow essential ports
sudo ufw allow 2222/tcp  # SSH
sudo ufw allow 80/tcp    # HTTP
sudo ufw allow 443/tcp   # HTTPS
sudo ufw allow 25/tcp    # SMTP
sudo ufw allow 587/tcp   # Submission
sudo ufw allow 465/tcp   # SMTPS
sudo ufw allow 143/tcp   # IMAP
sudo ufw allow 993/tcp   # IMAPS
sudo ufw allow 110/tcp   # POP3
sudo ufw allow 995/tcp   # POP3S

# Verify rules
sudo ufw status verbose

Step 5: Install Docker and Docker Compose

bash

# Install Docker dependencies
sudo apt install -y apt-transport-https ca-certificates \
  curl gnupg lsb-release

# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
  https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.20.0/docker-compose-$(uname -s)-$(uname -m)" \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Add user to docker group
sudo usermod -aG docker sugan

# Enable Docker to start on boot
sudo systemctl enable docker
sudo systemctl start docker

# Verify installation
docker --version
docker-compose --version

Part 2: Mailu Setup

Step 6: Install Mailu Configuration Wizard

bash

# Create Mailu directory
mkdir ~/mailu && cd ~/mailu

# Download the setup wizard
curl -O https://setup.mailu.io/1.11/master

# Make it executable
chmod +x master

# Run the setup wizard
./master

Step 7: Mailu Configuration

During the wizard, you'll need to make these choices:

1. Mailu version: 1.11 (stable)

2. Storage path: /mailu

3. Main mail domain: yourdomain.com

4. Server hostname: mail.yourdomain.com

5. Authentication: We'll choose front for integrated authentication

6. TLS certificate: 
   letsencrypt

7. Timezone: Your timezone (e.g., Asia/Kolkata)

8. Webmail: roundcube (recommended)

9. Database: mysql (or postgresql if preferred)

10. Message size limit: Set according to your needs (e.g., 50M)

Step 8: Configure docker-compose.yml

After the wizard completes, you'll have a docker-compose.yml file. Let's examine and customize it:

yaml

version: '3.8'

services:
  # Frontend service
  front:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-1.11}
    restart: always
    env_file: .env
    ports:
      - "80:80"
      - "443:443"
      - "25:25"
      - "465:465"
      - "587:587"
      - "993:993"
      - "995:995"
    volumes:
      - "/mailu/certs:/certs"
      - "/mailu/overrides:/overrides"
    depends_on:
      - admin
      - imap

  # Admin interface
  admin:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-1.11}
    restart: always
    env_file: .env
    volumes:
      - "/mailu/data:/data"
      - "/mailu/dkim:/dkim"

  # IMAP service
  imap:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}dovecot:${MAILU_VERSION:-1.11}
    restart: always
    env_file: .env
    volumes:
      - "/mailu/mail:/mail"
      - "/mailu/overrides:/overrides"

  # Add other services as needed...

Step 9: Configure .env File

The .env file contains crucial configuration. Let's review and customize:

bash

# Open the .env file
vim .env

Key configurations to verify:

env

# Domain settings
DOMAIN=yourdomain.com
HOSTNAME=mail.yourdomain.com

# TLS settings
TLS_FLAVOR=letsencrypt

# Secret keys (generate unique ones)
SECRET_KEY=your-secure-random-key-here
DB_PW=your-secure-database-password

# Admin settings
[email protected]
INITIAL_ADMIN_PW=secure-admin-password

# Mail settings
MESSAGE_SIZE_LIMIT=50000000  # 50MB

Generate secure passwords:

bash

# Generate random secret key
openssl rand -hex 32

# Generate database password
openssl rand -base64 32

Step 10: Create Required Directories and Set Permissions

bash

# Create directory structure
sudo mkdir -p /mailu/{data,dkim,certs,mail,overrides,redis}

# Set proper permissions
sudo chown -R 1000:1000 /mailu/data
sudo chown -R 1000:1000 /mailu/dkim
sudo chown -R 1000:1000 /mailu/mail

# Set permissions for certs directory
sudo chmod 755 /mailu/certs

Part 3: DNS Configuration

Step 11: Configure DNS Records

For your domain at your DNS provider, create these records:

A Records

text

mail.yourdomain.com    A    YOUR_SERVER_IP
yourdomain.com         A    YOUR_SERVER_IP

MX Records

text

yourdomain.com    MX    10    mail.yourdomain.com

TXT Records

1. SPF Record:

text

yourdomain.com    TXT   "v=spf1 mx a:mail.yourdomain.com -all"

2. DKIM Record (generated after Mailu setup):
   We'll generate this after starting Mailu

3. DMARC Record:

text

_dmarc.yourdomain.com    TXT   "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

PTR Record (Reverse DNS)

Contact your hosting provider to set up PTR record pointing to mail.yourdomain.com

Step 12: Start Mailu Services

bash

# Navigate to Mailu directory
cd ~/mailu

# Start services in detached mode
docker-compose up -d

# Check status
docker-compose ps

# View logs (if needed)
docker-compose logs -f

Step 13: Generate DKIM Key

bash

# Generate DKIM key
docker-compose exec admin flask mailu dkim yourdomain.com

# View the DKIM key
cat /mailu/dkim/yourdomain.com.txt

Add the DKIM record to your DNS:

text

mail._domainkey.yourdomain.com    TXT    "v=DKIM1; k=rsa; p=YOUR_PUBLIC_KEY_HERE"

Part 4: SSL/TLS Configuration

Step 14: Configure Let's Encrypt

Mailu should automatically obtain SSL certificates. Verify:

bash

# Check certificate directory
ls -la /mailu/certs/

# Check if certificates were obtained
docker-compose logs front | grep -i cert

If certificates aren't being generated automatically, you can manually trigger:

bash

# Renew certificates
docker-compose exec front certbot renew

Step 15: Configure TLS Security

Create TLS configuration override:

bash

# Create nginx TLS configuration
sudo mkdir -p /mailu/overrides/nginx

# Create TLS configuration file
sudo vim /mailu/overrides/nginx/tls.conf

Add strong TLS configuration:

nginx

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
add_header Strict-Transport-Security "max-age=63072000" always;

Part 5: Initial Configuration and Testing

Step 16: Access Admin Interface

1. Open browser to: https://mail.yourdomain.com/admin

2. Login with: [email protected] and your admin password

3. Change the default admin password immediately

Step 17: Create Your First User

1. In Admin interface, navigate to "Users"

2. Click "Add user"

3. Fill in details:

   • Local part: sugan (or your preferred username)

   • Domain: yourdomain.com

   • Password: Strong password

   • Quota: Set as needed (e.g., 5GB)

4. Click "Submit"

Step 18: Configure Email Client

For Thunderbird/Desktop Clients:

• Incoming:

  • Server: mail.yourdomain.com

  • Port: 993 (IMAP SSL)

  • Authentication: Normal password

  • SSL/TLS: Yes

• Outgoing:

  • Server: mail.yourdomain.com

  • Port: 587 (STARTTLS)

  • Authentication: Normal password

  • SSL/TLS: STARTTLS

For Mobile Devices:

Use the same settings as above. Most clients will auto-discover settings with proper DNS records.

Step 19: Test Email Functionality

bash

# Test SMTP connection
sudo apt install -y telnet
telnet mail.yourdomain.com 25
EHLO yourdomain.com
QUIT

# Test IMAP connection
openssl s_client -connect mail.yourdomain.com:993 -crlf

# Send test email using swaks
sudo apt install -y swaks
swaks --to [email protected] \
  --from [email protected] \
  --server mail.yourdomain.com \
  --auth LOGIN \
  --auth-user [email protected] \
  --auth-password 'YOUR_PASSWORD' \
  -tls

Part 6: Advanced Configuration

Step 20: Configure Email Aliases

1. In Admin interface, go to "Aliases"

2. Click "Add alias"

3. Configure:

   • Source: [email protected]

   • Destination: [email protected]

   • Optional comment: "Contact form alias"

Step 21: Set Up Catch-All Address

1. Go to "Aliases"

2. Add alias:

   • Source: @yourdomain.com

   • Destination: [email protected]

3. This catches all emails sent to non-existent addresses

Step 22: Configure Auto-Reply/Vacation

1. In Admin interface, go to "Users"

2. Click on your user

3. Scroll to "Auto-reply"

4. Enable and configure:

   • Subject: "Out of office"

   • Body: Your vacation message

   • Start/End dates (optional)

Step 23: Configure Spam Filtering

Edit .env file to enable spam filtering:

env

# Enable spam filtering
WEBMAIL=roundcube
ANTISPAM=rspamd

Update and restart:

bash

docker-compose down
docker-compose up -d

Configure Rspamd via Admin interface under "Antispam" settings.

Step 24: Set Up Webmail

1. Access webmail at: https://mail.yourdomain.com/webmail

2. Login with your email credentials

3. Configure Roundcube preferences:

   • Identity settings

   • Signature

   • Folder preferences

   • Contact management

Part 7: Security Hardening

Step 25: Configure Fail2Ban

bash

# Install Fail2Ban
sudo apt install -y fail2ban

# Create Mailu jail configuration
sudo vim /etc/fail2ban/jail.d/mailu.conf

Add configuration:

ini

[mailu]
enabled = true
port = 80,443,25,465,587,993,995
filter = mailu
logpath = /mailu/data/mail/fail2ban.log
maxretry = 3
bantime = 3600

Create filter:

bash

sudo vim /etc/fail2ban/filter.d/mailu.conf

Add:

ini

[Definition]
failregex = ^.* Authentication failed: <HOST>$
ignoreregex =

Restart Fail2Ban:

bash

sudo systemctl restart fail2ban
sudo systemctl enable fail2ban

Step 26: Configure Log Rotation

bash

# Create log rotation for Mailu
sudo vim /etc/logrotate.d/mailu

Add:

conf

/mailu/data/mail/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 root root
    sharedscripts
    postrotate
        docker restart $(docker ps -q --filter name=mailu)
    endscript
}

Step 27: Set Up Monitoring

bash

# Install monitoring tools
sudo apt install -y prometheus-node-exporter

# Configure Docker monitoring
sudo vim /etc/docker/daemon.json

Add:

json

{
  "metrics-addr": "127.0.0.1:9323",
  "experimental": true
}

Step 28: Regular Backup Strategy

Create backup script:

bash

vim ~/backup-mailu.sh

Add:

bash

#!/bin/bash
BACKUP_DIR="/backup/mailu"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR/$DATE

# Stop Mailu services
cd ~/mailu
docker-compose down

# Backup data
tar -czf $BACKUP_DIR/$DATE/mailu_data.tar.gz /mailu

# Backup configuration
cp docker-compose.yml $BACKUP_DIR/$DATE/
cp .env $BACKUP_DIR/$DATE/

# Start Mailu services
docker-compose up -d

# Remove backups older than 30 days
find $BACKUP_DIR -type d -mtime +30 -exec rm -rf {} \;

echo "Backup completed: $BACKUP_DIR/$DATE"

Make executable and schedule:

bash

chmod +x ~/backup-mailu.sh
(crontab -l 2>/dev/null; echo "0 2 * * * /home/sugan/backup-mailu.sh") | crontab -

Part 8: Troubleshooting Common Issues

Issue 1: Emails Not Sending

bash

# Check mail logs
docker-compose logs smtp

# Test SMTP connection
telnet mail.yourdomain.com 25

# Check DNS records
dig mx yourdomain.com
dig txt yourdomain.com

Issue 2: Emails Marked as Spam

1. Verify all DNS records (SPF, DKIM, DMARC)

2. Check PTR record

3. Ensure TLS is working properly

4. Check blacklist status: dig +short yourdomain.com.dnsbl.example

Issue 3: Cannot Receive External Emails

bash

# Check if port 25 is open
sudo nmap -p 25 yourdomain.com

# Check Postfix logs
docker-compose logs smtp | grep -i "reject"

# Verify MX record
dig mx yourdomain.com

Issue 4: Web Interface Not Loading

bash

# Check nginx logs
docker-compose logs front

# Check if services are running
docker-compose ps

# Check port binding
sudo netstat -tlnp | grep :443

Issue 5: SSL Certificate Issues

bash

# Force certificate renewal
docker-compose exec front certbot renew --force-renewal

# Check certificate validity
openssl s_client -connect mail.yourdomain.com:443 -servername mail.yourdomain.com 2>/dev/null | openssl x509 -noout -dates

Part 9: Performance Optimization

Step 29: Optimize Database

bash

# Create MySQL optimization configuration
sudo vim /mailu/overrides/mysql/my.cnf

Add:

cnf

[mysqld]
innodb_buffer_pool_size = 256M
innodb_log_file_size = 64M
query_cache_size = 32M
tmp_table_size = 32M
max_heap_table_size = 32M

Step 30: Configure Redis Caching

Add to .env:

env

# Enable Redis cache
REDIS_ADDRESS=redis

Step 31: Tune Dovecot for Performance

Create override:

bash

sudo vim /mailu/overrides/dovecot/dovecot.conf

Add optimization:

conf

mail_cache_min_mail_count = 50
mailbox_idle_check_interval = 30s

Part 10: Maintenance and Updates

Step 32: Regular Maintenance Tasks

Create maintenance script:

bash

vim ~/maintain-mailu.sh

Add:

bash

#!/bin/bash
echo "Starting Mailu maintenance..."

# Update system packages
sudo apt update && sudo apt upgrade -y

# Clean Docker
docker system prune -f

# Update Mailu images
cd ~/mailu
docker-compose pull
docker-compose up -d

# Check for errors
docker-compose logs --tail=50

echo "Maintenance completed!"

Step 33: Update Mailu

bash

# Backup first
./backup-mailu.sh

# Stop services
docker-compose down

# Pull new images
docker-compose pull

# Update configuration if needed
# Compare your docker-compose.yml with latest version

# Start services
docker-compose up -d

Step 34: Monitor Resources

bash

# Install monitoring
sudo apt install -y glances

# Run glances
glances

# Or use docker stats
docker stats

Part 11: Additional Features

Step 35: Set Up Calendar and Contacts (CalDAV/CardDAV)

Add to docker-compose.yml:

yaml

radicale:
  image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}radicale:${MAILU_VERSION:-1.11}
  restart: always
  env_file: .env
  volumes:
    - "/mailu/radicale:/data"

Update .env:

env

# Enable Radicale
CALENDAR=radicale

Step 36: Configure Sieve Filters

1. Access webmail

2. Go to Settings → Filters

3. Create server-side filters using Sieve syntax

4. Example filter to move newsletter to folder:

text

require ["fileinto"];
if header :contains "List-ID" "newsletter" {
    fileinto "Newsletters";
}

Step 37: Set Up Email Forwarding

Create forward in Admin interface:

1. Go to "Aliases"

2. Add new alias with external address as destination

3. Example: [email protected] → [email protected]

Step 38: Configure API Access

Enable API in .env:

env

# Enable Admin API
ADMIN_API=true

Use API for automation:

bash

# Get token
curl -X POST https://mail.yourdomain.com/admin/token \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]","password":"your-password"}'

# Use token to manage users
curl -X GET https://mail.yourdomain.com/admin/api/user \
  -H "Authorization: Bearer YOUR_TOKEN"

Conclusion

Congratulations! You've successfully set up a complete, self-hosted email server using Mailu on Debian. As Sugan Siddhant Dodrai, you now have:

1. Complete Control: Over your email infrastructure

2. Privacy: Your data stays on your server

3. Professional Setup: With proper DNS, TLS, and security

4. Scalability: Can grow with your needs

5. Learning: Valuable hands-on experience

Final Checklist

• All DNS records properly configured

• SSL certificates working

• Can send and receive emails

• Webmail accessible

• Admin interface secured

• Backups scheduled

• Monitoring in place

• Security measures implemented

Ongoing Responsibilities

1. Weekly: Check logs and monitor performance

2. Monthly: Update system and Mailu

3. Quarterly: Review security settings

4. Bi-annually: Test disaster recovery

Resources for Further Learning

1. Mailu Documentation: https://mailu.io

2. Postfix Documentation: http://www.postfix.org/documentation.html

3. Dovecot Documentation: https://doc.dovecot.org

4. SSL/TLS Best Practices: https://ssl-config.mozilla.org

Remember, running your own email server is a responsibility. Stay vigilant about security, keep regular backups, and enjoy the freedom and privacy that comes with self-hosting your email infrastructure.

---

About the Author: Sugan Siddhant Dodrai is a technology enthusiast passionate about self-hosted solutions and digital privacy. This guide represents comprehensive knowledge gathered from years of experience in system administration and email server management.