I know that for conf, we have conf-custom to change the templates, both for installation/update and running. Is there a similar way for server plugin files or other ISPConfig core components?
You'll have to write your own plugin (or modify an existing ISPConfig plugin) and use a different name. Then disable the ispconfig default plugin by deleting its symlink in plugins-enabled folder and add a symlink for your plugin instead. ISPConfig server plugins are event-based, you can have as many plugins bind to the same events as you like. So e.g. when you want to have a plugin that adds some folders to a website after it has been created, create a plugin like x_web_plugin.inc.php, add some code to bind a function to the website update event (see existing apache plugin for an example) and in this function do whatever you like to do. using something like x_ in the name ensures that it is loaded at the end (sorted alphabetically) so your code is always executed after the website has been created.