Discussion: New documentation environment

The topic came up in another thread about options for writing documentation. I've been involved with many projects that faced the same question and all have made different choices. I think this means we can save a lot of time and accept that there is no "best" solution, it's really a matter of personal comfort of the few people who actually work on documentation. I'll note a number of related factors here, numbered only for reference (cuz you guys know I love numbered lists ), and might come back and edit this list as others add their notes. Then based on these factors, Till and other leads here can make their decisions.

1. Let's start with the elephant in the room: Free is good.
2. I think it's important to have the ability to share content between documents to avoid copy/paste or re-writing. I think text should be small and modular, not enclosed in massive, hard-coded "silo" documents. Think lots of DLLs rather than a huge EXE, or lots of OOP classes rather than one huge file/function.
   
3. I think this project could really use the ability to generate a document that is specialized for the reader. Right now there is a different "Perfect" document for a bunch of platforms, and there is another document published for each new version, like Ubuntu 16/18/20. I think the ideal scenario is to have intelligent documentation that is generated dynamically. So for example, if someone is using Courier rather that Dovecot, the document presented to them can include Courier-specific information and links to related discussions in this forum.
   
4. I hope we can agree that collaboration is a top requirement. An environment where people need to copy around a document is old school and inadequate. A single document maintained by a single person is not scalable.
5. Wiki docs in a Git repo with markdown is a great option for collaboration. (A GitHub Wiki repo is a bit weird. For this purpose I'm talking about a regular repo that happens to contain a markdown docs.) It can be versioned like software, branched, and published. A problem with generic markdown is that it doesn't support (either easily or at all) attractive UX features like tooltips, collapsing sections (spoiler/accordion), or other features common and helpful in other media. Tools are also required to export from markdown to other formats.
6. A wiki like MediaWiki is also a great option. Too many people think a wiki Must be open to the public. That is not true. People also believe a published wiki like this is exposed to bogus registrations, malicious content, etc. That is only the case when it is not secured - that is, when people want a wiki with public viewing and private collaboration, but they leave it open for public contributions. A hybrid option is possible where a read-only wiki is available to users with a private wiki for development. MediaWiki also has plugins for publishing documentation to PDF or HTML. So people don't need to get their information from a wiki - the wiki can be used just for development. In summary, there is no "a wiki does this". The more accurate statement is "a wiki does whatever we want it to do".
   Note that MediaWiki supports the modularity and conditional logic mentioned in #2 & #3.
   MediaWiki is just one wiki option. I focus on it here as just one of many wiki options that can be installed and fully supported internally.
   
7. Collaboration tools are much more readily available now than in the past. For example, documents are easily shared in Slack and Teams. Google docs can be used in a split second, and it contains adequate formatting for current purposes. Microsoft One Note collaboration is easy, as is Evernote and similar notebook applications.
   While some of these options allow for versioning, verision support isn't great.
   Many of these options claim real-time collaboration as a feature but in practice, especially for FOSS which is developed very asynchnously, that feature is almost entirely unimportant.
8. There are many subscription based websites. As examples, Confluence is popular and free for 10 users, and Bit.ai specializes in collaboration and is free for up to 5 users. Personally I don't like subscription sites because the documentation is on someone else's server and there are always limits to avoid.
9. Over the years WordPress has evolved into a full content management system. It's FOSS, free, all data is local, there are a ton of plugins that already support functionality described here, and it's very extensible. It's also written in PHP which is very familiar to ISPConfig devs if required.
   I personally recommend WordPress, though others might prefer Drupal or some other CMS. The point here is that for this specific application a general purpose CMS with a lot of features might be preferable to a documentation-specific platform. I tend not to think that way, that general-purpose tools might be better than tools for a specific purpose, but I see documentation as being "content", and each paragraph and chapter as being separate content, with a document just being a desired export format. Compared to forum software like this which requires forum-specific functionality.
   Note that a CMS, similar to markdown, can export content (a web page) in other formats like PDF.
   Note also that this medium faciliates user/role-based access, workflow, notifications, comments, and many other features. We can integrate it within the control panel for context-specific help. And we can probably also integrate it with this XenForo.
10. There are licensed products like HelpNDoc and Help&Manual. I've been using H&M professionally for the last year and have been quite happy with it. It conforms to almost all of the notes here so far except it is not free, though I don't know if there is a special license for FOSS projects. There are probably others in this category that might be full-featured And free.

Summary:
There are a lot of options with WordPress, so if I had to do this now for this project, that would be my first preference. If markdown a requirement then I'd go with MediaWiki - Second to that woudl be another wiki package - Third to that would be pages on GitHub. My overall preference would be H&M but I don't actually think that would be best for this project due to the cost. YMMV

I hope I haven't forgotten anything obvious. And I hope this is helpful in this process.
If the team decides on WordPress or MediaWiki, I would be happy to help setup, maintain, and sort of mentor the environment for these purposes. While others contribute with code, this would be my FOSSy contribution.

 

I'm not sure what the big deal is here but the documentation is afaik not free and has to be paid if one wants to read it.

If one wants to have nice looking PDFs I would suggest to use LaTeX because Word just sucks for documentation purposes or alternatively I would suggest to use reStructuredText which is another markup language, similar but not exactly the same as LaTeX and more targeted to direct online use. For both approaches you can use git repositories to maintain the docs.
However, I'm not sure yet how a community, or should I better say people who contribute to the documentation can benefit from contributing/writing the docs when the documentation is actually paid and as of today "closed source". I mean, shall the documentation become open so that anyone can contribute to it or are there plans for contributors to receive little benefits, or how do you think the system could work?

Or do you actually don't mean the real documentation but the How-To's which are published here on the frontpage?
Well, I don't know how the backend for adding the howtos looks like but I don't miss anything at least, when reading a howto here on this site, except that some of them are not very detailed and scratching only a little bit the surface of a software.

btw, Wordpress disqualifies itself when it comes to plugins which you've mentioned. The WP Core is actually quite fine or at least not such a huge security mess due to the automatic update function but the plugins and especially the 3rd party themes makes it unusable for any serious usage (i.e. business or professional).

 

This is not about the for-fee, 300+ page PDF. This is about all of the other docs that are now written as blogs, and any other KB type docs that would be helpful for this application. Ideally the same tools should be used to write all free/fee docs.

I mentioned Teams but did not suggest Word. Personally I've written a lot of fine manuals in Word and it is an industry standard. But it doesn't fit the requirements of a good collaborative project so I didn't mention it here. I think this means we agree.

I've been writing code and docs for about 40 years and under my little rock I think I've only heard of reStructuredText a couple times. IMO it's not mainstream enough to justify usage for a team that needs a general-purpose solution. Yes, LaTeX is probably the top typesetting system. But IMO a project like this doesn't need that much attention on UX - while it does need more ability to re-use, edit, and replace content modules that are used in many different documents. And I also don't know a single developer who actually knows LaTeX, but anyone who contributes to FOSS knows markdown. So anyway these are my reasons for not including these options in the list but if others agree with you then yes the options should be in the list.

The internet disagrees. You are correctly citing a recent blip in the news but in the big picture one blip doesn't characterize an entire platform forever. In my ongoing love/hate relationship with WP, I agree that the third-party independendent marketplace easily exposes too many sites to issues including basic client/visitor retention. There is also a lot of abandoned and insecure software that people are still using. But the same could be said for extensions for any IDE we use, any API, any browser plugin, any extensible framework, any SaaS, and every commercial offering we use. ALL FOSS has dependencies and ALL FOSS is subject to the weaknesses of the weakest link in the chain. If you cite WP for plugin/theme issues created by individual developers in a specific point-release then you'll need to include ALL FOSS as having the same potential vulnerabilities. In which case, ISPConfig and everything else we talk about here is "unusable for any serious usage". The problem with any of this software isn't with any of the platforms. None of them are inherently more subject to plugin/theme issues than others. The problem is our blind love of FOSS without a process to vet/certify it. That's a problem in the FOSS model. We're not going to solve that problem in this discussion.

 

Why not using only a Wiki like DocuWiki or Mediawiki, so people can share and edit Docus better.

My favorite is Dokuwiki or https://www.bookstackapp.com/

Edit: I never see any Wiki here on ispconfig. Must be only for v2.

 

Okay, maybe I just missed it.

I did documentation for years at i-MSCP and there I used Docuwiki. Spam was actually never a problem because accounts were activated manually after applying for them.

Mediawiki has good antispam plugins. I haven't tested Bookstackapp yet, but it looks very interesting, because topics are managed in so-called books

 

To me that wasn't clear on first read, that's why I asked you that in my post.

I know that Word is pretty often (mis)used for manuals because it is easy to start with but as soon as you hit a certain amount of pages it will become hell of a documentation tool.

Now I've the feeling that you misinterpret something. I was talking about LaTeX in the context of writing the manual and not the "standard how-tos" here on howtoforge.com. LaTeX isn't very popular, true because you have to learn it first eventhough it is NOT THAT difficult to learn but often one does lack the time to learn it that's why tools like Word are suddenly used for documentation purposes (manual).

And regarding reStructuredText: Yes, it does exists since ~18 years but it is getting (lately) more and more popular since the ongoing python hype & growing git use. Bareos is documenting their software with RST, the Linux Kernel, and even non-FOSS services like HornetSecurity document their API with RST. That's why you probably didn't hear any of it but just recently...

I'm not citing any news. I'm talking about my own experience and what I see almost everyday in my life of someone who is maintaining web servers and mailservers for organizations, smaller & bigger companies and also public customers for almost 20 years. Yes, I agree with you that every CMS is sharing the same problem with insecure extensions but WP has the problem that it doesn't just have extensions. It does also have themes which include PHP code and that's an issue which some other CMS definitely don't share. In the right hands of an experienced user WP will most likely don't produce any problems but most people use it because it is easy to set up and you don't need to know much about PHP & templating.

I think we are done here but in case we are not we both can continue this discussion by pm.

 

Docbook.
https://en.wikipedia.org/wiki/DocBook

 

@michelangelo - we're on the same page now - I completely agree. My approach to themes is not to get too fond of them because over time they easily fall off the maintained/recommended list. But apparently we're also agreed that this is just the state of the industry - people easily get into the same debates over Linux distros, browsers, IDEs, frameworks, etc... So all other concerns being equal, IMO, I choose to acknowledge and deal with concerns as they come. YYMV

 

@Taleman - In one sense I believe DocBook is ultimately the "best" solution for any documentation. Unfortunately it's a best solution that still has not quite taken off. There are many pros and cons, clarified pretty well in this Q&A.

DocBook does the right thing, completely abstracting presentation from content in a very nice MVC way. The problem is that there are so few C=Controller and V=View tools for it. We write content in DocBook without a WYSIWYG GUI to faciliates the writing process. That means we spend a lot of time in DocBook text focused on XML semantics for content rather than the content itself. For better tooling we need DTD/XTD for writing, and XSL for rendering - and all of this could be done in VS Code, but we're still editing XML and then going through a build process to render it. Other options described here don't have this burden, and that allows non-programmers to participate. To the well-initiated, in-lining formatting is wrong for many reasons. But it's easy to do and helps us to produce and maintain documentation very quickly. If ISPConfig had an audience where some people could write content, others could format into DocBook, and others could focus on presentation, I'd think differently about this. But for this specific audience, I don't think this is the right solution - unless someone here can propose some decent tooling...

Note that the Help & Manual software that I mentioned (not recommended for this project due to cost) does store all data in a custom dialect of XML. It in-lines formatting into the code. Because the doc is code, it's easy to use macros, snippets, if/then dynamic content, and other features that I really love. I see it as an OK balance, with XML that can be manipulated by code, but not the UI-free purity of DocBook.

I found a couple interesting pages as I was verifying my current understanding:
https://opensource.com/article/17/9/docbook
https://el-tramo.be/blog/docbook-kit/

 

@till - I was just looking around at https://docs.ispconfig.org/. The site is minimal, clean, free, effective, standard, accessible, and secure. It's been updated to the latest WordPress version and is not exposing a significant attack surface. From my perspective as a content contributor I see no reason why that couldn't serve at least as a base for aggregating and providing public content.

Role-based access provides control against abuse: Subscribers verified in-forum PM can commment ; Contributors can recommend but not publish changes ; Authors can write pages and blogs ; And all content can be verified before publication by an Editor or higher level role.

This thread has at least two focuses : Content, and Presentation. I think it's possible, perhaps desirable, for content providers to contribute with minimal concern for presentation. Just focus on what you know, not how it looks. Leave "pretty" to someone else. I think if we follow this principle we won't need to be concerned about abuse of markup, specialized blocks, and other visual details. As noted earlier, I think too much focus on features provided by Markdown, LaTeX, etc, moves us away from separation of MVC-style concerns. And WordPress content editing is WYSIWYG - no knowledge of formatting code required. That eliminates the need for content contributors to have any formatting-fu skills, and opens the door for more content providers, which is really the goal here.

I don't think the tech audience for this software cares much at all about nice formatting and presentation anyway. They/we need solid information, which is the main problem that I'm trying to solve with this initiative, and "pretty" can be an annoyance to create and maintain. Most of us are used to terse JavaDoc, PHPDoc, SandCastle, and even the almost purely functional styling of sites like GitHub and StackOverflow. Formatting isn't a big deal to this audience. The less formatting we have in-line, the easier it will be to render content, what we actually care about, in different media, whether PDF or even an export into DocBook. I'm not saying docs should be terse and ugly. I'm saying it's not as important as common discussion implies, and the WordPress environment is versatile enough to allow for elegant presentation without forcing contributors to understand it.

I'm hoping we can move toward establishing existing https://docs.ispconfig.org/ site as at least a start. If it doesn't work, we're no worse off than we are now.

 

What about the tutorials? I feel writing those is in need of improvements. Maybe start a requirements document so if and when a better documentation system is being created some targets are known.

• It looks to me the tutorial preview uses different CSS than the published version. So when I preview the tutorial while writing, what I see does not match the published version.
• Even better if I can write only markup and someone or something else formats it
• it is complicated to give tutorial to a proofreader, he/she should be able to mark corrections and I then decide what to do with those corrections.
• I do not like writing in browser. I much prefer getting a set of text files on my own workstation and working on them with my favorite editor.

 

I'm writing another (long-winded) thread about content, including tutorials.

About content vs presentation, I agree that it's ideal for some people to provide nothing but content while others format. While rarely discussed, I believe formatting of content is a completely valid contribution in the spirit of FOSS, especially for those who say "I can't code, what else can I do?"

Indeed. Quite simply, some people are qualified to verify/modify tutorials and others are not. Some people might correct a statement you make and effectively change the subtlety of your intent. You need the opportunity to reject the change and replace it with what you really want. That said, the process is important. When someone changes text that I write, I take it as a hint that maybe what I wrote wasn't clear or complete enough to convey the facts. "Corrections" via editing/redacting are helpful to the author, like a beta is to software. We get the opportunity to understand how someone actually reads our content compared to how we write it. It's a valuable process. Right now we kinda get that with comments in the blog postings for Perfect tutorials, except with that we see comments which the author can then choose to accept, ignore, perhaps re-implement in his own words. That's an OK process, just different from content editing.

What you're describing is point #4 of my OP: "I hope we can agree that collaboration is a top requirement. An environment where people need to copy around a document is old school and inadequate. A single document maintained by a single person is not scalable." That dove-tails with your note here:

Yeah, I get that too. But the concept of "I want to do this on my system with my tools" is at odds with centralized collaboration and standardization. I would suggest that initial writing can be done with whatever tools one prefers, but when content enters the collaborative phase then we need to come together with common tools. I've personally argued against many browser-oriented applications, but that IS the standard today and it DOES satisfy many of the needs. You personally could choose to edit your text in collaboration with others through a GitHub repo, with the final verified text then getting posted into WordPress for formatting and publication. But that binds technical expertise of site management with another skillset for Git project collaboration. While we can reasonably assume that most people who are skilled with one should be skilled with the other, that's not a universal truth. Simply stated, I don't think it's good for the user community to have to be bound to external dependencies and personal protocols like working on docs in text/Git first before moving into the WP/Wiki/whatever environment.

In an environment like this we need to give a lot of freedom to contributions. I'm usually very careful about what I say and how I say it. I don't want anyone to rephrase something, especially into a technical inaccuracy, and blame me for it. But if the meaning is clear and the text is still accurate after someone else messes with my docs, I often choose to just accept it. Consider these forums, StackOverlow, blogs, and every other documentation resource in our industry : people frequently botch English, but we know what they mean and we accept it. (BTW, I understand it's usually very difficult to speak English as a second language. As someone who speaks several languages (poorly) I appreciate when people speak English as a common language and I especially give no grief for language issues.) In "professional" documentation I think it's important to accept some of this while still correcting basic grammar, spelling, tense, and other subtleties that can affect the reader experience.

In other words, people Will change our words, and while we have a great deal of control over it, sometimes it's better to let the little things go so that we can focus on the bigger things. Forgiveness like this is also encouraging to those who are generous enough to contribute. Rejecting of too many minor details is really discouraging to someone who is trying hard to do the little part that they can - that's bad for the community, limits the proliferation of the offering, and quite often leads to the death of FOSS that's otherwise really good software.

Bottom line for now : We're never going to be entirely happy with what someone else does with our code, docs, or anything else we create. Some jackass will always put salt on what the chef believes is a perfectly seasoned meal. The chef has the final say about what leaves the kitchen, but after that he has to trust others to serve it properly. The big picture is important. We make some concessions in the interest of democracy and collaboration. We should establish what we believe is best for the most people, and if we get it wrong, fix it. That's what I'm hoping you guys will establish here - the high level, big picture, best for the masses solution.