Modifying ISPConfig, chances of getting it integrated upstream,…

Discussion in 'Developers' Forum' started by kwisarts, Jul 25, 2024.

  1. kwisarts

    kwisarts New Member

    I am currently working on integrating traefik as an SSL terminating reverse proxy and load-balancer in front of an ISPConfig master-master set-up.

    I guess I'm trying to figure out how interesting this is going to be for others. In other words, am I going to have to maintain my changes as a patchset for all eternity or is there an ever so slight chance we'll get this integrated upstream. I can see how this is a pretty specific modification, but for us (and our client) it makes sense.
    Also, it can be debated if this really needs consul or if I couldn't just have added some endpoint in ISPConfig fir traefik to query as a provider… in any case, none of the consul set-up is currently part of my code and it also won't be unless there is a chance this code will be integrated.
    This would be different if ISPConfig was a little (a lot ;)) more customization-friendly, but the fact that almost any extensions are changes to core files make it all the more interesting to get something integrated upstream.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    In general, it sounds like an interesting addition. What's important is that you structure your code correctly in plugins, and we require that the contributor is at least willing to maintain his contribution. We often get very special code or features offloaded, and then the contributor is nowhere to be seen again, and then we have to take the work to maintain it or remove it again.

    ISPConfig can be customized without overwriting the core. It is fully event-based and has modules, plugins, template overrides in the GUI and server parts, etc., which you can use to customize it. If you don't know how to customize a certain part, you might want to ask how to do it instead of claiming it can't be customized. ISPConfig itself is just a set of plugins and modules, you could change or replace nearly any function in ISPConfig without touching the core, you could e.g. replace the web server with a completely different server, or postfix with exim, or dovecot with another imap server, or the spam filter system, or the DNS server with an API call to an external provider or a completely different server. That's so easy because what you call the core are just customizations provided by us consisting of plugins and modules which can easily be extended, replaced etc. which completely different functionalities, applications etc.
     
    Last edited: Jul 25, 2024
    ahrasis likes this.
  3. kwisarts

    kwisarts New Member

    In that case, I apologize for the hasty, unfounded claim. I have not found any documentation on this, apart from some other threads here asking about it.

    And I'll ask: where can I find information on template overrides and interface action extensions?
    I have found (a link I cannot post without my reputation), but that's very little information and nothing about extending existing templates.

    I do think I understand the server plugin-ins part, but even that still feels quite tightly coupled to me, so my guess is, I'm also missing some information on this and would appreciate guidance on this too.

    Here's what I have done or will need to do for a first mvp:

    • extended the client, client limits and web vhost domain templates, forms and action files to handle making the feature available and enabling it. (I did manually modify the DB at this point, so I'll also need a place to add the logic or SQL to run ALTER statements I suppose)
    • create a new plugin to run on web_domain_* events and a class that contains the logic to create consul services and register them with the catalog.
     
    Last edited: Jul 25, 2024
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  5. till

    till Super Moderator Staff Member ISPConfig Developer

    There is currently no documentation besides the code itself. We would love to write one, but as no company wanted to sponsor the work of doing this, it could not be written yet. But we are constantly working on adding more documentation step-by-step.

    1) Your plugin must provide an SQL statement to install the missing database fields in its install routine. ISPConfig is built not to override or mess with any custom fields.
    2) You create a plugin in /interface/lib/plugins which e.g. binds e.g. to the onShow of the client form to add the custom fields and saves them by binding to onBeforeInsert, onAfterInsert, onbeforeUpdate or onAfterIpdate. See interface/lib/classes/tform_actions.inc.php to get an overview of the event handling of a form.

    One thing to note, plugin events are parsed once and then stored at login for a user session, so when you create a new plugin, you must re-login once.

    In addition to what I wrote above, you can also have custom UI elements to achieve that, as an example see e.g. interface/lib/classes/plugin_listview.inc.php which implements the DNS record list from ISPConfig DNS manager.

    That's even easier than the interface, simply create a new plugin file in server/plugins-available/, take care your plugin name comes after the apache and Nginx plugin so its not called before the website is created. For the actual event binding, check the first part of apache and Nginx plugin, your plugin simply binds to the same events. You can have as many plugins bound to a event as you like.


    That#s not correct too. The server part is a loosely coupled event-based system where everything can be replaced and you can have as many event listeners as you like to override actions. Everything the server part is doing is a customization like a plugin or module) on its own and can be overridden or replaced easily. So you can replace any software ISPConfig uses and ISPConfig is configuring with completely different software without touching the core.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    And of course, there might still be very few things that can not be overridden that easily and in this case, it makes sense that we look at it if it makes sense to extend the core. But as all form handling and all server actions can be overridden already, it should be possible to do most things already.
     
  7. kwisarts

    kwisarts New Member

    Thanks for the explanations. I can write documentation/a tutorial as I go along with this. What format does it need to be? Is it part of a git repository? Personally, I'd prefer it was part of the docs at the URL that Taleman posted (and I couldn't), not a thread in a forum.

    BTW, part of my confusion stems from the fact that, inside server, the plugins directories are on the root level, while inside interface, they're inside lib/ and don't follow the same -available/enabled approach. I was actually looking for a similar structure to the server area in the interface part but didn't look into the lib/ dir.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    The difference is that the UI is module-based and uses a form handler in the modules for most things. So, plugins in the Ui are just events attached to the form handler or custom form UI elements. While the server part is a event-based plugin system where events are changes in database tables and plugins subscribe to them. To organize things, we have also split it into modules so one can easily say this is just a web server or a DNS server or a mail system by enabling/disabling a module.

    I will have a look into that to see what the best format is. I will also try to write a small sample on how to add a custom field to a form as a starting point.
     
  9. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I guess you can write it in a form of a thread in this forum for the time being, and later convert that into proper tutorial / documentation?
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    That's one option. Or write it in markup and store it as .md files in the docs folder of the ISPConfig source tree, so GitLab will show it as formatted pages then.
     
    ahrasis likes this.
  11. kwisarts

    kwisarts New Member

    I must say I'm having a hard time understanding the structure of all this.

    So far, I was able to
    1. register to an event that allows me to extend the template for the web domains edit form (took me a while to find this).
    2. load my own template and language file and return the results of combining both (but only interface texts so far, no form fields).
    What I don't understand and haven't been able to figure out on my own so far is how the form/*tform.php files tie in with the templating engine? I can see that the tform types define a formtype for each field, but how exactly, in a plugin file, would I go from having a template and a tform definition to having e.g. a checkbox rendered?

    I understand I can instantiate a new tform class, then call loadFormDef() to load my form definition.
    Or should I not create a new tform, but instead extend the existing $app->tform with the field definitions that I need?
    In both cases, how can I tie this in with my extra template?

    I might switch to the server part for a while, it seems more straightforward than extending the UI.

    P.S.: I found there is yet another notion of plugins which can be found within interface/lib/classes and which seem to get registered inside the tform.php files as part of the form structure, then loaded not in plugins.inc.php, but tform_actions.inc.php which raises events handled by the plugins in lib/plugins, but also loads the plugins registered through tform files.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    As I mentioned, its likely better if I prepare a short example on how to extend a form as its not that easy to figure out when you don't know the code base.

    You do not need a new tform class for that. A new tform class would be used only if you create your complete own new form without extending a existing one.

    Server part is definitely easier.

    I've explained this in post #5, see:

     
  13. kwisarts

    kwisarts New Member

    I'll keep adding instead of modifying posts. Sort of as a journal of my journey of discovery through ISPConfig.

    Using the events raised through template hooks to extend the views is one way to do it, but I have found that a more elegant way is to hook into
    Code:
    $module:$form:on_after_formdef
    Not only does it allow me to directly modify the form definition, e.g. adding a new tab (which can have its own template) to the form, it will also allow to to register the type of plugins that I have mentioned in the P.S. above. I'm not sure yet if there is an advantage to registering a plug-in from a plug-in though and it will probably depend on whether extending the formDef will be enough or whether I'll need to run my own code inside another onShow action.

    There is one issue I am running into with this method, which is that after switching from the new tab to another tab, I get validation errors for fields that are not part of the tab and the form breaks entirely. E.g. Column 'allow_override' cannot be null.

    Note: One job to do, to add to the documentation that is to be written is a complete list of events that are raised, including what parameters are passed. Such an index might have saved me a lot of time.
     
  14. kwisarts

    kwisarts New Member

    Looking forward to your example.
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Template hooks are for modifying the template file only, while you can modify the form definition using on_after_formdef event. And alternatively you can build also a custom GUI element which you then insert into the form. Buts custom GUI elements are mostly useful when you want to insert more complex things like a table with data inside of a form and not just some standard input elements.
     
  16. kwisarts

    kwisarts New Member

    Another question that related to both interface and server plugins: Directory structure
    Let's say my plug-in brings some extra files, e.g. templates, language files… so far, plug-ins, in both the interface and the server part are just individual PHP files. The code in plugin.inc.php that indexes plug-ins does not recurse into subdirectories (which may be a potential extension of that code?).

    So, currently, if I wanted my plug-in code to be as self-contained as possible, the structure I'd most likely pursue would be:

    Code:
    /usr/local/ispconfig/interface/lib/plugins/
      my_plugin_file.inc.php
      my_plugin/
        templates/my_plugin_foo.html
        lang/en_my_plugin.lng
    /usr/local/ispconfig/server/plugins-available/
      my_plugin_file.inc.php
      my_plugin/
        templates/my_plugin_consul.hcl
        …
    
    What is your take on this till or what would be your recommendation?
     
    Last edited: Jul 31, 2024
  17. CubAfull

    CubAfull Member

    After reading the comments I realize that there is practically everything I need to make my own extensions. A small tutorial or at least a brief description of the existing hooks/events and the correct structure to place the extensions would be of great help.

    I think that for the structure the correct thing would be to use:
    /usr/local/ispconfig/interface/web/ for a complete new module.
    /usr/local/ispconfig/interface/lib/plugins/ to extend existing module.
    /usr/local/ispconfig/server/plugins-available/ for server functions like exec, but no templates or lang here.
     
  18. CubAfull

    CubAfull Member

    Hello @till
    I create the plugin and I can add a new tab to "Email Mailbox" and "Domain" with needed options. In the "Email Mailbox" everything work OK when I swish between tabs, all the record are saved, but with the "Domain" tab I'm having a really bad time.
    The first error I get when I go back to the "Domain" tab from my new tab "Cloud" is "The Server can not be changed.". I can fix that by setting the server_id in a hidden input and now I can go back without any error, but the BIG problem is that the "spamfilter_users" in the db linked to this domain is deleted, also any email mailbox with this domain is modified and ends only with "theemailuser@".

    It is possible that the domains part is not designed to work with tabs? What can I do in this case?

    Captura desde 2024-08-05 03-31-32.png Captura desde 2024-08-05 03-31-15.png

    PHP:
    <?php

    /*
    Copyright (c) 2010, Till Brehm, projektfarm Gmbh
    All rights reserved.
    ......
    */

    class mail_cloud_plugin
    {
        var 
    $plugin_name 'mail_cloud_plugin';
        var 
    $class_name 'mail_cloud_plugin';

        var 
    $plugin_dir;

        private 
    $mail_cloud_tables = array(
            
    'mail_domain' => array(
                
    'cloud_enabled',
                
    'cloud_group'
            
    ),
            
    'mail_user' => array(
                
    'cloud_user_enabled',
                
    'cloud_user_quota',
                
    'cloud_user_group'
            
    )
        );

        public function 
    __construct()
        {
            
    $this->plugin_dir ISPC_ROOT_PATH '/lib/plugins/' $this->plugin_name;
        }

        
    /**
         * This function is called when the plugin is loaded
         *
         * @return void
         */
        
    function onLoad()
        {
            global 
    $app;

            
    // Check if needed columns exist
            
    $this->checkColumns();

            
    // Register for the events

            // ONLY for TEST: mail_domain
            
    $app->plugin->registerEvent('mail:mail_domain:on_before_insert''mail_cloud_plugin''mail_domain_edit');
            
    $app->plugin->registerEvent('mail:mail_domain:on_before_update''mail_cloud_plugin''mail_domain_edit');
            
    $app->plugin->registerEvent('mail:mail_domain:on_after_insert''mail_cloud_plugin''mail_domain_edit');
            
    $app->plugin->registerEvent('mail:mail_domain:on_after_update''mail_cloud_plugin''mail_domain_edit');

            
    // ONLY for TEST: mail_user
            
    $app->plugin->registerEvent('mail:mail_user:on_before_insert''mail_cloud_plugin''mail_user_edit');
            
    $app->plugin->registerEvent('mail:mail_user:on_before_update''mail_cloud_plugin''mail_user_edit');
            
    $app->plugin->registerEvent('mail:mail_user:on_after_insert''mail_cloud_plugin''mail_user_edit');
            
    $app->plugin->registerEvent('mail:mail_user:on_after_update''mail_cloud_plugin''mail_user_edit');

            
    // Insert Tab
            
    $app->plugin->registerEvent('mail:mail_domain:on_after_formdef''mail_cloud_plugin''mail_domain_form');
            
    $app->plugin->registerEvent('mail:mail_user:on_after_formdef''mail_cloud_plugin''mail_user_form');
        }

        function 
    checkColumns()
        {
            global 
    $app;

            foreach (
    $this->mail_cloud_tables as $table => $columns) {
                
    $list "'" implode("','"$columns) . "'";
                
    $sql "SHOW COLUMNS FROM $table WHERE Field IN($list)";
                
    $result $app->db->queryAllArray($sql);

                if (!empty(
    $result)) {
                    
    $diff array_diff($columns$result);
                    if (
    $diff) {
                        
    $this->createColumns($diff);
                    }
                } else {
                    
    $this->createColumns($columns);
                }
            }
        }

        function 
    createColumns($columns)
        {
            global 
    $app;

            foreach (
    $columns as $column) {
                
    $file $this->plugin_dir "/sql/$column.sql";
                if (
    is_file($file)) {
                    
    $sql file_get_contents($file);
                    
    $app->db->query($sql);
                }
            }
        }

        function 
    mail_domain_edit($event_name$page_form)
        {
            global 
    $app;

            return;
        }

        function 
    mail_user_edit($event_name$page_form)
        {
            global 
    $app;

            return;
        }

        function 
    mail_domain_form($event_name$page_form)
        {
            
    $this->loadLang($page_form);

            
    $page_form->formDef['tabs'] += array(
                
    'cloud' => array(
                    
    'title' => 'Cloud',
                    
    'width' => 100,
                    
    'template' => $this->plugin_dir '/templates/mail_domain_edit.htm',
                    
    'fields' => array(
                        
    'cloud_enabled' => array(
                            
    'datatype' => 'VARCHAR',
                            
    'formtype' => 'CHECKBOX',
                            
    'default' => 'n',
                            
    'value' => array(=> 'y'=> 'n')
                        ),
                        
    'cloud_group' => array(
                            
    'datatype' => 'VARCHAR',
                            
    'formtype' => 'TEXT',
                            
    'default' => '',
                            
    'validators' => array(
                                
    => array(
                                    
    'type' => 'REGEX',
                                    
    'regex' => '/^(?:[\w][\w\s,]+[\w\s])*$/u',
                                    
    'errmsg' => 'cloud_group_error_regex'),
                            ),
                            
    'value' => '',
                            
    'width' => '40',
                            
    'maxlength' => '255'
                        
    ),
                    )
                )
            );
        }

        function 
    mail_user_form($event_name$page_form)
        {
            
    $this->loadLang($page_form);

            
    $page_form->formDef['tabs'] += array(
                
    'cloud' => array(
                    
    'title' => 'Cloud',
                    
    'width' => 100,
                    
    'template' => $this->plugin_dir '/templates/mail_user_edit.htm',
                    
    'fields' => array(
                        
    'cloud_user_enabled' => array(
                            
    'datatype' => 'VARCHAR',
                            
    'formtype' => 'CHECKBOX',
                            
    'default' => 'n',
                            
    'value' => array(=> 'y'=> 'n')
                        ),
                        
    'cloud_user_quota' => array(
                            
    'datatype' => 'VARCHAR',
                            
    'formtype' => 'TEXT',
                            
    'validators' => array(
                                
    => array(
                                    
    'type' => 'ISINT',
                                    
    'errmsg' => 'cloud_quota_error_isint'),
                                
    => array(
                                    
    'type' => 'REGEX',
                                    
    'regex' => '/^([0-9]{1,})$/',
                                    
    'errmsg' => 'cloud_quota_error_value'),
                            ),
                            
    'default' => '0',
                            
    'value' => '',
                            
    'width' => '30',
                            
    'maxlength' => '255'
                        
    ),
                        
    'cloud_user_group' => array(
                            
    'datatype' => 'VARCHAR',
                            
    'formtype' => 'CHECKBOX',
                            
    'default' => 'y',
                            
    'value' => array(=> 'y'=> 'n')
                        ),
                    )
                )
            );
        }

        function 
    loadLang($page_form)
        {
            global 
    $app$conf;

            
    $language $app->functions->check_language(
                isset(
    $_SESSION['s']['user']['language']) ? $_SESSION['s']['user']['language'] : $conf['language']
            );

            
    $file $this->plugin_dir "/lib/lang/$language.lng";

            if (!
    is_file($file)) {
                
    $file $this->plugin_dir "/lib/lang/en.lng";
            }

            @include 
    $file;

            if (isset(
    $page_form->wordbook) && isset($wb) && is_array($wb)) {

                if (
    is_array($page_form->wordbook)) {
                    
    $page_form->wordbook array_merge($page_form->wordbook$wb);
                } else {
                    
    $page_form->wordbook $wb;
                }
            }
        }
    }
     
    ahrasis and till like this.
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, it's possible that the code does not take that into account if the form has just one tab.

    The best thing would be to open an issue in the issue tracker and if possible, provide a fix for the main code. Basically, look into the code and add a proper if statement where needed that either checks what the current tab is (there must be a function for that in form, if I remember correctly, or alternatively check if the needed value like server_id is set in $this->dataRecord['server_id'] and if not, skip the code.
     
    CubAfull likes this.
  20. CubAfull

    CubAfull Member

    Fixed the problem with the "Domain" tab by restoring some missing vars.
    Everything is working ok now.
    @till you see any security flaw in this restored vars?. I think not because I'm using the on_before_insert/on_before_update and the custom plugin is running before the ISPConfig check/validations. What do you think?
    PHP:
    function onLoad():
            
    $app->plugin->registerEvent('mail:mail_domain:on_before_insert''mail_cloud_plugin''mail_domain_edit');
            
    $app->plugin->registerEvent('mail:mail_domain:on_before_update''mail_cloud_plugin''mail_domain_edit');
    --------------------------------
        function 
    mail_domain_edit($event_name$page_form)
        {
            global 
    $app;

            
    // Restore this vars in the form
            
    if (isset($page_form->dataRecord)) {
                
    $app->tpl->setVar(
                    
    'server_id_value',
                    (isset(
    $page_form->dataRecord['server_id']) ? $page_form->dataRecord['server_id'] : ''),
                    
    true
                
    );
                
    $app->tpl->setVar(
                    
    'domain_id_value',
                    (isset(
    $page_form->dataRecord['domain']) ? $page_form->dataRecord['domain'] : ''),
                    
    true
                
    );
                
    $app->tpl->setVar(
                    
    'policy_id_value',
                    (isset(
    $page_form->dataRecord['policy']) ? $page_form->dataRecord['policy'] : ''),
                    
    true
                
    );
            }
        }
    --------------------------------
    template:
    <
    tmpl_if name='server_id_value' op="!=" value="">
    <
    input type="hidden" name="server_id" value="{tmpl_var name='server_id_value'}" />
    </
    tmpl_if>
    <
    tmpl_if name='domain_id_value' op="!=" value="">
    <
    input type="hidden" name="domain" value="{tmpl_var name='domain_id_value'}" />
    </
    tmpl_if>
    <
    tmpl_if name='policy_id_value' op="!=" value="">
    <
    input type="hidden" name="policy" value="{tmpl_var name='policy_id_value'}" />
    </
    tmpl_if>
     
    ahrasis likes this.

Share This Page