Modifying ISPConfig, chances of getting it integrated upstream,…

I am currently working on integrating traefik as an SSL terminating reverse proxy and load-balancer in front of an ISPConfig master-master set-up.

I guess I'm trying to figure out how interesting this is going to be for others. In other words, am I going to have to maintain my changes as a patchset for all eternity or is there an ever so slight chance we'll get this integrated upstream. I can see how this is a pretty specific modification, but for us (and our client) it makes sense.
Also, it can be debated if this really needs consul or if I couldn't just have added some endpoint in ISPConfig fir traefik to query as a provider… in any case, none of the consul set-up is currently part of my code and it also won't be unless there is a chance this code will be integrated.
This would be different if ISPConfig was a little (a lot ) more customization-friendly, but the fact that almost any extensions are changes to core files make it all the more interesting to get something integrated upstream.

 

In that case, I apologize for the hasty, unfounded claim. I have not found any documentation on this, apart from some other threads here asking about it.

And I'll ask: where can I find information on template overrides and interface action extensions?
I have found (a link I cannot post without my reputation), but that's very little information and nothing about extending existing templates.

I do think I understand the server plugin-ins part, but even that still feels quite tightly coupled to me, so my guess is, I'm also missing some information on this and would appreciate guidance on this too.

Here's what I have done or will need to do for a first mvp:

• extended the client, client limits and web vhost domain templates, forms and action files to handle making the feature available and enabling it. (I did manually modify the DB at this point, so I'll also need a place to add the logic or SQL to run ALTER statements I suppose)
• create a new plugin to run on web_domain_* events and a class that contains the logic to create consul services and register them with the catalog.

 

Have you already examined these materials?
https://www.ispconfig.org/development/

 

Thanks for the explanations. I can write documentation/a tutorial as I go along with this. What format does it need to be? Is it part of a git repository? Personally, I'd prefer it was part of the docs at the URL that Taleman posted (and I couldn't), not a thread in a forum.

BTW, part of my confusion stems from the fact that, inside server, the plugins directories are on the root level, while inside interface, they're inside lib/ and don't follow the same -available/enabled approach. I was actually looking for a similar structure to the server area in the interface part but didn't look into the lib/ dir.

 

I guess you can write it in a form of a thread in this forum for the time being, and later convert that into proper tutorial / documentation?

 

I must say I'm having a hard time understanding the structure of all this.

So far, I was able to

1. register to an event that allows me to extend the template for the web domains edit form (took me a while to find this).
2. load my own template and language file and return the results of combining both (but only interface texts so far, no form fields).

What I don't understand and haven't been able to figure out on my own so far is how the form/*tform.php files tie in with the templating engine? I can see that the tform types define a formtype for each field, but how exactly, in a plugin file, would I go from having a template and a tform definition to having e.g. a checkbox rendered?

I understand I can instantiate a new tform class, then call loadFormDef() to load my form definition.
Or should I not create a new tform, but instead extend the existing $app->tform with the field definitions that I need?
In both cases, how can I tie this in with my extra template?

I might switch to the server part for a while, it seems more straightforward than extending the UI.

P.S.: I found there is yet another notion of plugins which can be found within interface/lib/classes and which seem to get registered inside the tform.php files as part of the form structure, then loaded not in plugins.inc.php, but tform_actions.inc.php which raises events handled by the plugins in lib/plugins, but also loads the plugins registered through tform files.

 

I'll keep adding instead of modifying posts. Sort of as a journal of my journey of discovery through ISPConfig.

Using the events raised through template hooks to extend the views is one way to do it, but I have found that a more elegant way is to hook into

Code:

$module:$form:on_after_formdef

Not only does it allow me to directly modify the form definition, e.g. adding a new tab (which can have its own template) to the form, it will also allow to to register the type of plugins that I have mentioned in the P.S. above. I'm not sure yet if there is an advantage to registering a plug-in from a plug-in though and it will probably depend on whether extending the formDef will be enough or whether I'll need to run my own code inside another onShow action.

There is one issue I am running into with this method, which is that after switching from the new tab to another tab, I get validation errors for fields that are not part of the tab and the form breaks entirely. E.g. Column 'allow_override' cannot be null.

Note: One job to do, to add to the documentation that is to be written is a complete list of events that are raised, including what parameters are passed. Such an index might have saved me a lot of time.

 

Looking forward to your example.

 

Another question that related to both interface and server plugins: Directory structure
Let's say my plug-in brings some extra files, e.g. templates, language files… so far, plug-ins, in both the interface and the server part are just individual PHP files. The code in plugin.inc.php that indexes plug-ins does not recurse into subdirectories (which may be a potential extension of that code?).

So, currently, if I wanted my plug-in code to be as self-contained as possible, the structure I'd most likely pursue would be:

Code:

/usr/local/ispconfig/interface/lib/plugins/
  my_plugin_file.inc.php
  my_plugin/
    templates/my_plugin_foo.html
    lang/en_my_plugin.lng
/usr/local/ispconfig/server/plugins-available/
  my_plugin_file.inc.php
  my_plugin/
    templates/my_plugin_consul.hcl
    …

What is your take on this till or what would be your recommendation?

 

After reading the comments I realize that there is practically everything I need to make my own extensions. A small tutorial or at least a brief description of the existing hooks/events and the correct structure to place the extensions would be of great help.

I think that for the structure the correct thing would be to use:
/usr/local/ispconfig/interface/web/ for a complete new module.
/usr/local/ispconfig/interface/lib/plugins/ to extend existing module.
/usr/local/ispconfig/server/plugins-available/ for server functions like exec, but no templates or lang here.

 

Hello @till
I create the plugin and I can add a new tab to "Email Mailbox" and "Domain" with needed options. In the "Email Mailbox" everything work OK when I swish between tabs, all the record are saved, but with the "Domain" tab I'm having a really bad time.
The first error I get when I go back to the "Domain" tab from my new tab "Cloud" is "The Server can not be changed.". I can fix that by setting the server_id in a hidden input and now I can go back without any error, but the BIG problem is that the "spamfilter_users" in the db linked to this domain is deleted, also any email mailbox with this domain is modified and ends only with "theemailuser@".

It is possible that the domains part is not designed to work with tabs? What can I do in this case?

Expand: Plugin

PHP:

<?php

/*
Copyright (c) 2010, Till Brehm, projektfarm Gmbh
All rights reserved.
......
*/

class mail_cloud_plugin
{
    var $plugin_name = 'mail_cloud_plugin';
    var $class_name = 'mail_cloud_plugin';

    var $plugin_dir;

    private $mail_cloud_tables = array(
        'mail_domain' => array(
            'cloud_enabled',
            'cloud_group'
        ),
        'mail_user' => array(
            'cloud_user_enabled',
            'cloud_user_quota',
            'cloud_user_group'
        )
    );

    public function __construct()
    {
        $this->plugin_dir = ISPC_ROOT_PATH . '/lib/plugins/' . $this->plugin_name;
    }

    /**
     * This function is called when the plugin is loaded
     *
     * @return void
     */
    function onLoad()
    {
        global $app;

        // Check if needed columns exist
        $this->checkColumns();

        // Register for the events

        // ONLY for TEST: mail_domain
        $app->plugin->registerEvent('mail:mail_domain:on_before_insert', 'mail_cloud_plugin', 'mail_domain_edit');
        $app->plugin->registerEvent('mail:mail_domain:on_before_update', 'mail_cloud_plugin', 'mail_domain_edit');
        $app->plugin->registerEvent('mail:mail_domain:on_after_insert', 'mail_cloud_plugin', 'mail_domain_edit');
        $app->plugin->registerEvent('mail:mail_domain:on_after_update', 'mail_cloud_plugin', 'mail_domain_edit');

        // ONLY for TEST: mail_user
        $app->plugin->registerEvent('mail:mail_user:on_before_insert', 'mail_cloud_plugin', 'mail_user_edit');
        $app->plugin->registerEvent('mail:mail_user:on_before_update', 'mail_cloud_plugin', 'mail_user_edit');
        $app->plugin->registerEvent('mail:mail_user:on_after_insert', 'mail_cloud_plugin', 'mail_user_edit');
        $app->plugin->registerEvent('mail:mail_user:on_after_update', 'mail_cloud_plugin', 'mail_user_edit');

        // Insert Tab
        $app->plugin->registerEvent('mail:mail_domain:on_after_formdef', 'mail_cloud_plugin', 'mail_domain_form');
        $app->plugin->registerEvent('mail:mail_user:on_after_formdef', 'mail_cloud_plugin', 'mail_user_form');
    }

    function checkColumns()
    {
        global $app;

        foreach ($this->mail_cloud_tables as $table => $columns) {
            $list = "'" . implode("','", $columns) . "'";
            $sql = "SHOW COLUMNS FROM $table WHERE Field IN($list)";
            $result = $app->db->queryAllArray($sql);

            if (!empty($result)) {
                $diff = array_diff($columns, $result);
                if ($diff) {
                    $this->createColumns($diff);
                }
            } else {
                $this->createColumns($columns);
            }
        }
    }

    function createColumns($columns)
    {
        global $app;

        foreach ($columns as $column) {
            $file = $this->plugin_dir . "/sql/$column.sql";
            if (is_file($file)) {
                $sql = file_get_contents($file);
                $app->db->query($sql);
            }
        }
    }

    function mail_domain_edit($event_name, $page_form)
    {
        global $app;

        return;
    }

    function mail_user_edit($event_name, $page_form)
    {
        global $app;

        return;
    }

    function mail_domain_form($event_name, $page_form)
    {
        $this->loadLang($page_form);

        $page_form->formDef['tabs'] += array(
            'cloud' => array(
                'title' => 'Cloud',
                'width' => 100,
                'template' => $this->plugin_dir . '/templates/mail_domain_edit.htm',
                'fields' => array(
                    'cloud_enabled' => array(
                        'datatype' => 'VARCHAR',
                        'formtype' => 'CHECKBOX',
                        'default' => 'n',
                        'value' => array(1 => 'y', 0 => 'n')
                    ),
                    'cloud_group' => array(
                        'datatype' => 'VARCHAR',
                        'formtype' => 'TEXT',
                        'default' => '',
                        'validators' => array(
                            0 => array(
                                'type' => 'REGEX',
                                'regex' => '/^(?:[\w][\w\s,]+[\w\s])*$/u',
                                'errmsg' => 'cloud_group_error_regex'),
                        ),
                        'value' => '',
                        'width' => '40',
                        'maxlength' => '255'
                    ),
                )
            )
        );
    }

    function mail_user_form($event_name, $page_form)
    {
        $this->loadLang($page_form);

        $page_form->formDef['tabs'] += array(
            'cloud' => array(
                'title' => 'Cloud',
                'width' => 100,
                'template' => $this->plugin_dir . '/templates/mail_user_edit.htm',
                'fields' => array(
                    'cloud_user_enabled' => array(
                        'datatype' => 'VARCHAR',
                        'formtype' => 'CHECKBOX',
                        'default' => 'n',
                        'value' => array(1 => 'y', 0 => 'n')
                    ),
                    'cloud_user_quota' => array(
                        'datatype' => 'VARCHAR',
                        'formtype' => 'TEXT',
                        'validators' => array(
                            1 => array(
                                'type' => 'ISINT',
                                'errmsg' => 'cloud_quota_error_isint'),
                            0 => array(
                                'type' => 'REGEX',
                                'regex' => '/^([0-9]{1,})$/',
                                'errmsg' => 'cloud_quota_error_value'),
                        ),
                        'default' => '0',
                        'value' => '',
                        'width' => '30',
                        'maxlength' => '255'
                    ),
                    'cloud_user_group' => array(
                        'datatype' => 'VARCHAR',
                        'formtype' => 'CHECKBOX',
                        'default' => 'y',
                        'value' => array(1 => 'y', 0 => 'n')
                    ),
                )
            )
        );
    }

    function loadLang($page_form)
    {
        global $app, $conf;

        $language = $app->functions->check_language(
            isset($_SESSION['s']['user']['language']) ? $_SESSION['s']['user']['language'] : $conf['language']
        );

        $file = $this->plugin_dir . "/lib/lang/$language.lng";

        if (!is_file($file)) {
            $file = $this->plugin_dir . "/lib/lang/en.lng";
        }

        @include $file;

        if (isset($page_form->wordbook) && isset($wb) && is_array($wb)) {

            if (is_array($page_form->wordbook)) {
                $page_form->wordbook = array_merge($page_form->wordbook, $wb);
            } else {
                $page_form->wordbook = $wb;
            }
        }
    }
}

 

Fixed the problem with the "Domain" tab by restoring some missing vars.
Everything is working ok now.
@till you see any security flaw in this restored vars?. I think not because I'm using the on_before_insert/on_before_update and the custom plugin is running before the ISPConfig check/validations. What do you think?

PHP:

function onLoad():
        $app->plugin->registerEvent('mail:mail_domain:on_before_insert', 'mail_cloud_plugin', 'mail_domain_edit');
        $app->plugin->registerEvent('mail:mail_domain:on_before_update', 'mail_cloud_plugin', 'mail_domain_edit');
--------------------------------
    function mail_domain_edit($event_name, $page_form)
    {
        global $app;

        // Restore this vars in the form
        if (isset($page_form->dataRecord)) {
            $app->tpl->setVar(
                'server_id_value',
                (isset($page_form->dataRecord['server_id']) ? $page_form->dataRecord['server_id'] : ''),
                true
            );
            $app->tpl->setVar(
                'domain_id_value',
                (isset($page_form->dataRecord['domain']) ? $page_form->dataRecord['domain'] : ''),
                true
            );
            $app->tpl->setVar(
                'policy_id_value',
                (isset($page_form->dataRecord['policy']) ? $page_form->dataRecord['policy'] : ''),
                true
            );
        }
    }
--------------------------------
template:
<tmpl_if name='server_id_value' op="!=" value="">
<input type="hidden" name="server_id" value="{tmpl_var name='server_id_value'}" />
</tmpl_if>
<tmpl_if name='domain_id_value' op="!=" value="">
<input type="hidden" name="domain" value="{tmpl_var name='domain_id_value'}" />
</tmpl_if>
<tmpl_if name='policy_id_value' op="!=" value="">
<input type="hidden" name="policy" value="{tmpl_var name='policy_id_value'}" />
</tmpl_if>