How to have a quote for development of a function already present in the Features Request list?

Discussion in 'General' started by stefano.b, Sep 8, 2020.

  1. stefano.b

    stefano.b New Member

    hi,
    I'm interested in push up the feature request here:
    https://git.ispconfig.org/ispconfig/ispconfig3/-/issues/3875
    "Limit number of mails sent by user per hour/day"
    posted by m4recek.

    How can I proceed?

    Stefano

    Sorry if there is a best place for this request, i serched the forum but I can't find a better place where to post it.
     
  2. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Hello Stefano,
    it is important to know if you are using rspamd or amavis as filter.
    In rspamd you can (relatively) easily configure that kind of "group" basis. Let's say "basic" 200/h, premium "5000/h", limited "10/h" etc.
    Of course without ISPConfig integration you'd have to edit the list files for each group by hand (simple text-file with user/domain in it)
     
    stefano.b likes this.
  3. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Hello @stefano.b, I've been planning to implement this with postfwd eventually, so it probably will get done at some point. It certainly won't be in 3.2. If you want to sponsor getting to it sooner than later I'd be glad to discuss it, as well as your requirements (in particular the OS version).
     
    Gwyneth Llewelyn and stefano.b like this.
  4. stefano.b

    stefano.b New Member

    Hi @Croydon,
    thank you for your reply and suggestion, the filter was Amavis, but now I set up Rspamd, I readed the documentation about the ratelimit of Rspamd.
    In ISPConfig words, it's possible to set up a server in which every Client and/or Web Domain has his limit?

    So if we have 2 clients with total of 3 web domains every web site can send its newsletter (monitored and controlled by Rspamd) without generate trouble for other domains?

    before i finish to read the documentation :) have you any hint about where I found this files?

    have a nice day

    S.
     
  5. stefano.b

    stefano.b New Member

    hi @Jesse Norell,
    yes I'd like to get it sooner :) than later, so let's me know.

    For my requirements I'm testing on a Test VPS, the OS version is:
    CentOS 7.4 x64
    Linux web.xxx.com 3.10.0-1062.7.1.el7.x86_64 #1 SMP Mon Dec 2 17:33:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
    ISPConfig Version: 3.1dev

    have a nice day

    S.
     
  6. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Sadly not. It is a custom setup that I use. I set it up like that:

    1. Config in /etc/rspamd/local.d/ratelimit.conf
    Code:
    custom_keywords = "/etc/rspamd/my_ratelimit.lua";
    2. In that mentioned file:
    Code:
    function table_print (tt, indent, done)
      done = done or {}
      indent = indent or 0
      if type(tt) == "table" then
        local sb = {}
        for key, value in pairs (tt) do
          table.insert(sb, string.rep (" ", indent)) -- indent it
          if type (value) == "table" and not done [value] then
            done [value] = true
            table.insert(sb, key .. " = {\n");
            table.insert(sb, table_print (value, indent + 2, done))
            table.insert(sb, string.rep (" ", indent)) -- indent it
            table.insert(sb, "}\n");
          elseif "number" == type(key) then
            table.insert(sb, string.format("\"%s\"\n", tostring(value)))
          else
            table.insert(sb, string.format(
                "%s = \"%s\"\n", tostring (key), tostring(value)))
           end
        end
        return table.concat(sb)
      else
        return tt .. "\n"
      end
    end
    
    function to_string( tbl )
        if  "nil"       == type( tbl ) then
            return tostring(nil)
        elseif  "table" == type( tbl ) then
            return table_print(tbl)
        elseif  "string" == type( tbl ) then
            return tbl
        else
            return tostring(tbl)
        end
    end
    
    
    
    
    local custom_keywords = {}
    local d = {}
    
    d['limit5k'] = rspamd_config:add_map({
       ['url'] = '/etc/rspamd/ratelimit_5k.map',
       ['type'] = 'set',
       ['description'] = 'Incoming 5k limit'
    })
    
    d['limit5k_out'] = rspamd_config:add_map({
            ['url'] = '/etc/rspamd/ratelimit_5k_out.map',
            ['type'] = 'set',
            ['description'] = 'Outgoing 5k limit'
    })
    
    d['limit1k'] = rspamd_config:add_map({
            ['url'] = '/etc/rspamd/ratelimit_1k.map',
            ['type'] = 'set',
            ['description'] = 'Incoming 1k limit'
    })
    
    d['limit1k_out'] = rspamd_config:add_map({
            ['url'] = '/etc/rspamd/ratelimit_1k_out.map',
            ['type'] = 'set',
            ['description'] = 'Outgoing 1k limit'
    })
    
    custom_keywords.customrl = function(task)
       local rspamd_logger = require "rspamd_logger"
    
       local mail_to = task:get_principal_recipient()
       local mail_from_tbl = task:get_from("mime")
       local mail_from = nil
       local from_domain = nil
      
       if mail_from_tbl and mail_from_tbl[1] and mail_from_tbl[1].addr then
           mail_from = mail_from_tbl[1].addr
           if mail_from_tbl[1].domain then
               from_domain = "@" .. mail_from_tbl[1].domain
           end
       else
           rspamd_logger.infox(rspamd_config, "Mail from is %s", to_string(mail_from_tbl))
       end
      
       local mail_auth = task:get_user()
       local to_domain = string.match(mail_to, ".*(@.*)")
      
       local crl = "200 / 1h"
    
       rspamd_logger.infox(rspamd_config, "Mail (auth %s) from %s (%s) to %s (%s) limit check coming.", mail_auth, mail_from, from_domain, mail_to, to_domain)
    
       if mail_auth then
           if d['limit5k_out']:get_key(mail_auth) then
               crl = "5000 / 1h"
               rspamd_logger.warnx(rspamd_config, "User %s is in limit5k_out map. Limit is %s", mail_auth, crl)
               return "my_rlo_" .. mail_auth, crl
           elseif d['limit1k_out']:get_key(mail_auth) then
               crl = "1000 / 1h"
               rspamd_logger.warnx(rspamd_config, "User %s is in limit1k_out map. Limit is %s", mail_auth, crl)
               return "my_rlo_" .. mail_auth, crl
           end
       end
    
       if mail_from then
           if d['limit5k_out']:get_key(mail_from) then
               crl = "5000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit5k_out map. Limit is %s", mail_from, crl)
               return "my_rlo_" .. mail_from, crl
           elseif d['limit1k_out']:get_key(mail_from) then
               crl = "1000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit1k_out map. Limit is %s", mail_from, crl)
               return "my_rlo_" .. mail_from, crl
           end
       end
    
       if mail_to then
           if d['limit5k']:get_key(mail_to) then
               crl = "5000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit5k map. Limit is %s", mail_to, crl)
               return "my_rli_" .. mail_to, crl
           elseif d['limit1k']:get_key(mail_to) then
               crl = "1000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit1k map. Limit is %s", mail_to, crl)
               return "my_rli_" .. mail_to, crl
           end
       end
    
       if from_domain then
           if d['limit5k_out']:get_key(from_domain) then
               crl = "5000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit5k_out map. Limit is %s", from_domain, crl)
               return "my_rlo_" .. from_domain, crl
           elseif d['limit1k_out']:get_key(from_domain) then
               crl = "1000 / 1h"
               rspamd_logger.warnx(rspamd_config, "%s is in limit1k_out map. Limit is %s", from_domain, crl)
               return "my_rlo_" .. from_domain, crl
           end
       end
    
       if to_domain then
           if d['limit5k']:get_key(to_domain) then
               crl = "5000 / 1h"
               rspamd_logger.warnx(rspamd_config, "User %s is in limit5k map. Limit is %s", to_domain, crl)
               return "my_rli_" .. to_domain, crl
           elseif d['limit1k']:get_key(to_domain) then
               crl = "1000 / 1h"
               rspamd_logger.warnx(rspamd_config, "User %s is in limit1k map. Limit is %s", to_domain, crl)
               return "my_rli_" .. to_domain, crl
           end
       end
      
       rspamd_logger.warnx(rspamd_config, "Mail from %s to %s not found in maps. Limit is %s", mail_from, mail_to, crl)
      
       return crl
    end
    
    return custom_keywords
    
    I am no expert in LUA programming language, so there might be much better solutions.

    3. As you can see in LUA file there are mentioned different "map" files, e. g. /etc/rspamd/ratelimit_5k_out.map
    Each of that files is a simple text file with content like:
    Code:
    @domain1.com
    @domain2.com
    [email protected]
    
    That's all. You could extend that to make it even more dynamic or "fine-grained".
     
  7. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    A more dynamic approach would be to use key-value-maps instead of simple sets. So you would only need 2 maps (incoming/outgoing) and could insert domain or user to limit string in there. Have not tried that, yet.
     
  8. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    For outbound (from your clients) mail I had in mind server-specific default limits for mail from a domain and from a mailbox, with ability to override that in domain/mailbox settings, does that meet your needs? Ie. I did not plan to implement per-client limits.

    Inbound would have some server-wide limits, not customizable (via gui) for individual sender ip addrs/domains. There would need to be some hook for custom postfwd config (either local file that is included, or maybe post raw postfwd conf into server config section; probably the former.
     
  9. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    I think this wouldn't be a good idea anyway. Way too complicated, I guess.
     
    Th0m likes this.
  10. stefano.b

    stefano.b New Member

    Hi @Jesse Norell,
    Yes, I think so.
    In any case this is the scenario: some of my clients use their CMS/software for sending their newsletters. I set up both the client software (for lower throughput) and the Control Panel Server for limit the number of mails they can send.

    I need a per domain (or mail) control/limit so that I can set up the ISPConfig for control (and maybe manage?) the sending process nomatter what the client software try to do.

    Please tell me if it Is this clear anought.

    Personally I'm not interessed in control the incoming mail, but it's ok this process too.

    have a nice day

    S.

    PS.
    I never asked for a customization to a GPL community software, so I don't know how procede, please tell me some extra data here o in private so I can understand if I can do it alone or I need help from some other users.
     
    Last edited: Sep 10, 2020
  11. stefano.b

    stefano.b New Member

    Hi @Croydon,
    If I understood the logic of this code, here there is not control, but "only" a writing log. Is it correct?
    have a nice day

    S.
     
  12. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    No, it does as well block messages exceeding the limits.
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    ISPConfig is not GPL Software, it is BSD Licensed. The BSD license is a liberal OpenSource License, so you can extend it and either keep your changes for yourself or provide them to the community, that's up to you.
     
  14. stefano.b

    stefano.b New Member

    hi @till,
    thank you for your reply and the correction, what I meant was that "it is GPL compliant" as we can see here.

    In any case, we are talking about the possibility that the personalization is made by a ISConfig Develooper, and I (and some other users) can sponsor this features. For this procedure I asked some more info.

    have a nice day
    S.
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    The issue is that: BSD license is GPL compliant, but GPL is not BSD compliant, which means, if someone would choose a GPL compliant license that is not the BSD license or MIT license, then the code may not be combined with ISPConfigs existing code.
     
  16. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Hello @stefano.b,

    I've wrapped up my work for 3.2 now, if you want to discuss implementing postfwd I can send you a PM. I think making a contribution to the ISPConfig project is what you have in mind, not just local customization, but to be clear I would *much* prefer that route (it would be less work, and subsequently supported by the project, and I wouldn't have to re-implement it for the project later ;).
     
  17. stefano.b

    stefano.b New Member

    Hi @Jesse Norell,
    yes absolutely, I'm interested in implement this function for the ISPconfig's community :)
    Please send me a PM.
    have a nice day
    S.
     
  18. florian030

    florian030 Well-Known Member HowtoForge Supporter

    I made some changes to the script from @Croydon.
    Code:
    mkdir -p /etc/rspamd/ratelimit
    touch /etc/rspamd/ratelimit/{limits_auth,limits_in,limits_out}
    And set the global values crl_out and crl_in (default is nil)
    Format of the auth-files:
    Code:
    SOURCE/DESTINATION:LIMIT_IN:LIMIT:OUT
    For a ratelimit different to the defaults:
    Limit outbound to 5 / minute to example.com:
    Code:
    @example.com::5/1m
    Limit inbound to 1 / minute from [email protected]:
    Code:
    [email protected]:1/1m:
    Limit in- and out for the domain example.com:
    Code:
    @example.com:1/1m:5/1m
    Add domain / addresses to the map-files

    limits_auth limit_auth_out.map and is used for authenticated senders only.
    Code:
    function table_print (tt, indent, done)
            done = done or {}
            indent = indent or 0
            if type(tt) == "table" then
                    local sb = {}
                    for key, value in pairs (tt) do
                            table.insert(sb, string.rep (" ", indent)) -- indent it
                            if type (value) == "table" and not done [value] then
                                    done [value] = true
                                    table.insert(sb, key .. " = {\n");
                                    table.insert(sb, table_print (value, indent + 2, done))
                                    table.insert(sb, string.rep (" ", indent)) -- indent it
                                    table.insert(sb, "}\n");
                            elseif "number" == type(key) then
                                    table.insert(sb, string.format("\"%s\"\n", tostring(value)))
                            else
                                    table.insert(sb, string.format("%s = \"%s\"\n", tostring (key), tostring(value)))
                            end
                    end
                    return table.concat(sb)
            else
                    return tt .. "\n"
            end
    end
    
    function to_string( tbl )
            if  "nil"       == type( tbl ) then
                    return tostring(nil)
            elseif  "table" == type( tbl ) then
                    return table_print(tbl)
            elseif  "string" == type( tbl ) then
                    return tbl
            else
                    return tostring(tbl)
            end
    end
    
    function Split(s, delimiter)
            result = {};
            for match in (s..delimiter):gmatch("(.-)"..delimiter) do
                    table.insert(result, match);
            end
            return result;
    end
    
    local custom_keywords = {}
    
    local rl_auth = {}
    local rl_auth_limits = {}
    local rl_auth_out_limits = {}
    local rl_in = {}
    local rl_in_limits = {}
    
    -- auth user
    local cfg_file = io.open("/etc/rspamd/ratelimit/limits_auth", "r")
    for line in cfg_file:lines() do
            split_string = Split(line, ':')
            local limit_in = ''
            local limit_out = ''
            local temp_in = split_string[2]:gsub('%s+', '') -- trim string
            local temp_out = split_string[3]:gsub('%s+', '') -- trim string
            if temp_in ~= '' then limit_in = split_string[2] else limit_in = nil end
            if temp_out ~= '' then limit_out = split_string[3] else limit_out = nil end
    
            rl_auth_limits[split_string[1]] = {
                    ['limit_in'] = limit_in,
                    ['limit_out'] = limit_out
            }
            -- maps
            rl_auth[split_string[1]] = rspamd_config:add_map({
                    ['url'] = '/etc/rspamd/ratelimit/maps/limit_auth.map';
                    ['type'] = 'set',
                    ['description'] = 'rl_auth for ' .. split_string[1]
            })
    
    end
    io.close(cfg_file)
    
    -- auth user outbound
    local cfg_file = io.open("/etc/rspamd/ratelimit/limits_out", "r")
    for line in cfg_file:lines() do
            split_string = Split(line, ':')
            local limit_in = ''
            local limit_out = ''
            local temp_in = split_string[2]:gsub('%s+', '') -- trim string
            local temp_out = split_string[3]:gsub('%s+', '') -- trim string
            if temp_in ~= '' then limit_in = split_string[2] else limit_in = nil end
            if temp_out ~= '' then limit_out = split_string[3] else limit_out = nil end
    
            rl_auth_out_limits[split_string[1]] = {
                    ['limit_in'] = limit_in,
                    ['limit_out'] = limit_out
            }
            -- maps
            rl_auth[split_string[1]] = rspamd_config:add_map({
                    ['url'] = '/etc/rspamd/ratelimit/maps/limit_auth_out.map';
                    ['type'] = 'set',
                    ['description'] = 'rl_auth for ' .. split_string[1]
            })
    
    end
    io.close(cfg_file)
    
    -- inbound
    local cfg_file = io.open("/etc/rspamd/ratelimit/limits_in", "r")
    for line in cfg_file:lines() do
            split_string = Split(line, ':')
            local limit_in = ''
            local limit_out = ''
            local temp_in = split_string[2]:gsub('%s+', '') -- trim string
            local temp_out = split_string[3]:gsub('%s+', '') -- trim string
            if temp_in ~= '' then limit_in = split_string[2] else limit_in = nil end
            if temp_out ~= '' then limit_out = split_string[3] else limit_out = nil end
    
            rl_in_limits[split_string[1]] = {
                    ['limit_in'] = limit_in,
                    ['limit_out'] = limit_out
            }
            -- maps
            rl_in[split_string[1]] = rspamd_config:add_map({
                    ['url'] = '/etc/rspamd/ratelimit/maps/limit_in.map';
                    ['type'] = 'set',
                    ['description'] = 'rl_in for ' .. split_string[1]
            })
    end
    io.close(cfg_file)
    
    custom_keywords.customrl = function(task)
            local rspamd_logger = require "rspamd_logger"
            local mail_to = task:get_principal_recipient()
            local mail_from_tbl = task:get_from("mime")
            local mail_from = nil
            local from_domain = nil
    
            if mail_from_tbl and mail_from_tbl[1] and mail_from_tbl[1].addr then
                    mail_from = mail_from_tbl[1].addr
                    if mail_from_tbl[1].domain then
                            from_domain = "@" .. mail_from_tbl[1].domain
                    end
            else
                    rspamd_logger.infox(rspamd_config, "Mail from is %s", to_string(mail_from_tbl))
            end
    
            local mail_auth = task:get_user()
            local to_domain = string.match(mail_to, ".*(@.*)")
    
            -- default values
            crl_out = nil
            crl_in = nil
    
            if mail_auth then -- limit auth user
                    local check = nil
                    rspamd_logger.debugx(rspamd_config, "Mail (auth %s) from %s (%s) to %s (%s) limit check for out coming.", mail_auth, mail_from, from_domain, mail_to, to_domain)
                    if rl_auth[mail_auth] then
                            if rl_auth[mail_auth]:get_key(mail_auth) then
                                    check = rl_auth_limits[mail_auth].limit_out
                                    rspamd_logger.warnx(rspamd_config, "User %s is limited to %s", mail_auth, check)
                                    if check ~= nil then return "my_rlo_" .. mail_auth, check end
                            end
                            if mail_to then -- limit destination email
                                    if rl_auth[mail_to] then
                                            if rl_auth[mail_to]:get_key(mail_to) then
                                                    check = rl_auth_out_limits[mail_to].limit_out
                                                    rspamd_logger.warnx(rspamd_config, "Mail to (mail_to %s) is limited to %s", mail_to, check)
                                                    if check ~= nil then return "my_rlo_" .. mail_to, check end
                                            end
                                    end
                            end
                            if to_domain then -- limit destination domain
                                    if rl_auth[to_domain] then
                                            if rl_auth[to_domain]:get_key(to_domain) then
                                                    check = rl_auth_out_limits[to_domain].limit_out
                                                    rspamd_logger.warnx(rspamd_config, "Mail to (to_domain %s) is limited to %s", to_domain, check)
                                                    if check ~= nil then return "my_rlo_" .. to_domain, check end
                                            end
                                    end
                            end
                            rspamd_logger.debugx(rspamd_config, "User %s has no custom limit. Limited to default %s", mail_auth, crl_out)
                            return crl_out
                    end
            else -- limit incoming mails
                    local check = nil
                    rspamd_logger.debugx(rspamd_config, "Mail from %s (%s) to %s (%s) limit check for in coming.", mail_from, from_domain, mail_to, to_domain)
                    if mail_from then
                            if rl_in[mail_from] then
                                    if rl_in[mail_from]:get_key(mail_from) then
                                            check =  rl_in_limits[mail_from].limit_in
                                            rspamd_logger.warnx(rspamd_config, "Mail from (mail_from %s) is limited to %s", mail_from, check)
                                            if check ~= nil then return "my_rlo_" .. to_domain, check end
                                    end
                            end
                    end
                    if from_domain then
                            if rl_in[from_domain] then
                                    if rl_in[from_domain]:get_key(from_domain) then
                                            check = rl_in_limits[from_domain].limit_in
                                            rspamd_logger.warnx(rspamd_config, "Mail from (from_domain %s) is limited to %s", from_domain, check)
                                            if check ~= nil then return "my_rlo_" .. to_domain, check end
                                    end
                            end
                    end
                    rspamd_logger.debugx(rspamd_config, "Mail from %s (%s) to %s (%s) has no custom limit. Limited to default %s", mail_from, from_domain, mail_to, to_domain, crl_in)
                    return crl_in
            end
    end
    
    return custom_keywords
    
    
     
    atle, Jesse Norell and ahrasis like this.

Share This Page