Dovecot with FTS Lucene

Discussion in 'Tips/Tricks/Mods' started by Loveless, Jul 6, 2017.

  1. Loveless

    Loveless Member

    If you're running dovecot for IMAP, I can highly recommend installing dovecot lucene instead of solr or just dovecot's plain Full Text Search. CLucene yields extremely fast IMAP search results on huge mailfolders, using webmail packages like Squirrelmail or Rainloop.
    Here's how I did it on a Debian 9 server:

    Be sure your apt repositories support a recent dovecot version and the Lucene plugin, these in your /etc/apt/sources.list for debian 9 will make sure of that:
    # stretch-backports
    deb http://httpredir.debian.org/debian/ stretch-backports main contrib non-free
    deb-src http://httpredir.debian.org/debian/ stretch-backports main contrib non-free
    Then, sudo, or as root, do:
    ~# apt-get update && apt-get install dovecot-lucene -y​
    which installs an up to date C++ CLucene library and the fts_lucene plugin module.

    I then changed my /etc/dovecot/dovecot.conf to activate and make proper use of lucene;
    mail_plugins = $mail_plugins fts fts_lucene
    plugin {
    fts = lucene
    fts_lucene = whitespace_chars=@. normalize no_snowball
    fts_autoindex=yes
    fts_autoindex_max_recent_msgs=80
    fts_index_timeout=90
    }​

    Below you'll see my full non-default dovecot configs. Note some performance improvements for dovecot that I added, and I removed the quota plugin (I don't bother my users with that, I think it's an outdated subject for mail. You could auto purge trash and junk folders by putting this under /etc/cron.monthly :

    #!/bin/sh
    doveadm expunge -A mailbox Junk savedbefore 60d
    doveadm expunge -A mailbox Trash savedbefore 660d
    doveadm force-resync -A '*'
    exit 0

    Code:
    ~# doveconf -n -P
    # 2.2.34 (874deae): /etc/dovecot/dovecot.conf
    # Pigeonhole version 0.4.22 (22940fb7)
    # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.3
    auth_cache_size = 24 M
    auth_cache_ttl = 18 hours
    auth_mechanisms = plain login
    disable_plaintext_auth = no
    listen = *,[::]
    log_path = /var/log/dovecot.log
    log_timestamp = "%Y-%m-%d %H:%M:%S "
    login_greeting = mydomainname ready.
    mail_fsync = never
    mail_max_userip_connections = 500
    mail_plugins = " fts fts_lucene"
    mail_privileged_group = vmail
    mail_temp_dir = /ramdsk/dovecot
    mailbox_idle_check_interval = 59 secs
    mailbox_list_index = yes
    maildir_very_dirty_syncs = yes
    passdb {
      args = /etc/dovecot/dovecot-sql.conf
      driver = sql
    }
    plugin {
      fts = lucene
      fts_autoindex = yes
      fts_autoindex_max_recent_msgs = 60
      fts_index_timeout = 60
      fts_lucene = whitespace_chars=@. normalize no_snowball
      sieve = /var/vmail/%d/%n/.sieve
      sieve_max_redirects = 25
    }
    protocols = imap pop3
    service auth {
      unix_listener /var/spool/postfix/private/auth {
      group = postfix
      mode = 0660
      user = postfix
      }
      unix_listener auth-userdb {
      group = vmail
      mode = 0600
      user = vmail
      }
      user = root
    }
    service imap-login {
      client_limit = 1000
      process_limit = 512
      process_min_avail = 6
      service_count = 64
      vsz_limit = 256 M
    }
    service lmtp {
      unix_listener /var/spool/postfix/private/dovecot-lmtp {
      group = postfix
      mode = 0600
      user = postfix
      }
    }
    ssl_cert = </etc/postfix/smtpd.cert
    ssl_key = </etc/postfix/smtpd.key
    userdb {
      driver = prefetch
    }
    userdb {
      args = /etc/dovecot/dovecot-sql.conf
      driver = sql
    }
    protocol pop3 {
      pop3_uidl_format = %08Xu%08Xv
    }
    protocol lda {
      mail_fsync = optimized
      mail_plugins = sieve
      postmaster_address = [email protected]
    }
    protocol lmtp {
      mail_fsync = optimized
      mail_plugins = sieve
      postmaster_address = [email protected]
    }
    
    I use multitail in another terminal to witness if all the changes go as planned:
    ~# multitail /var/log/dovecot.log /var/log/syslog /var/log/mail.log​

    Make sure ISPconfig's database has the right entries in them (you'll see errors when Searching IMAP in your dovecot log if you don't do this). Database `dbispconfig` needs to have two new columns:

    alter table `mail_user` add `disableindexer-worker` enum('n','y') NOT NULL DEFAULT 'n';
    alter table `mail_user` add `disablequota-status` enum('y','n') NOT NULL DEFAULT 'y';​

    If you disable quota status, like I do, you can also remove the CONCAT storage queries from /etc/dovecot/dovecot-sql.conf, so they change into this:

    # password-query with prefetch
    password_query = SELECT email as user, password, maildir as userdb_home, CONCAT( maildir_format, ':', maildir, '/', IF(maildir_format='maildir','Maildir',maildir_format)) as userdb_mail, uid as userdb_uid, gid as userdb_gid, CONCAT(maildir, '/.sieve') as userdb_sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1'

    user_query = SELECT email as user, maildir as home, CONCAT( maildir_format, ':', maildir, '/', IF(maildir_format='maildir','Maildir',maildir_format)) as mail, uid, gid, CONCAT(maildir, '/.sieve') as sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1'
    Restart dovecot, now with Lucene
    ~# service dovecot restart​
    check if it runs OK:
    ~# service dovecot status​

    Then, last but most importantly, force dovecot to reindex all mailfolders, as root do;

    # doveadm -v index -u '*' '*'​
    (the -v is verbose, so you see it actually notices the changed indexing config)
    For assurance, also sync the caches:

    # doveadm force-resync -A '*'​

    That should do it. Now try your superfast search on some string on [ All folders ] in squirrelmail or rainloop or roundcube or whatever your IMAP server client is.
     
    Last edited: Mar 6, 2018
    cwispy, Jesse Norell and till like this.

Share This Page