Storing generated PDF creates rebase conflicts

[thoni56]

I know there are some advantages of storing a generated version of the PDF:s in the repo, e.g. there is always a version that can be pointed to from the website.

But just having been through the hazzle of rebasing the dev-man branch onto a recent chunk of text changes, and been forced to handle a conflict for each and every commit (and probably also forcing a lot of that onto whoever wants to pull now), I think we need to re-think this. (I had no other problems...)

I propose that we figure out a way to upload the generated files to repo releases or somewhere else (website), so that we can get rid of the generated binary files from the repo to streamline the editing of the two branches.

I quickly looked for a git option to always ignore conflicts for some files but didn't find much. Possibly we could use a (custom?) merge-driver. Never heard about it before so that's why I added the "research" tag...

[thoni56]

After some googling I found https://gist.github.com/tmaybe/4c9d94712711229cd506 which explains merge driver a bit, but also indicates how to use it for exactly this purpose. So I'm gonna try this out on the dev-man branch and see what I can learn from it.

[tajmone]

I propose that we figure out a way to upload the generated files to repo releases or somewhere else (website), so that we can get rid of the generated binary files from the repo to streamline the editing of the two branches.

We could keep the generated PDFs only in master branch. I remember that the reason we were also updating them in the dev branches was to check that everything was working fine, but maybe now we no longer need that.

I quickly looked for a git option to always ignore conflicts for some files but didn't find much.

I think that git rerere (reuse recorded resolution) might do the trick:

• https://git-scm.com/docs/git-rerere
• https://www.git-scm.com/book/en/v2/Git-Tools-Rerere

[thoni56]

Thanks for the ideas. Yeah, we could do away with the PDF in the dev-branch.

The "branching" model we decided upon was, as least how I see it, to maintain two version of the documentation

• master that is aligned with the current official release but that would be updated with better and better content continously
• dev-man that is aligned with the development snapshots, as new features are added this version of the documentation would also add it

We would continuously rebase dev-man onto master to ensure that at release a smooth merge of it onto master could be done. Thanks to asciidoc that is practically feasible. And working.

(Maybe that needs to go somewhere, for posterity, and maybe myself ,-), the wiki?)

Removing the PDF from one, or both, branches would help. But that leaves us with no "dev documents" to point to.

There is a CI-job, which currently only does asciidoc validation as far as I can see. Would it be a possible route to expand on that to build the documentation too? Maybe that was our intention all along. Setting up the toolchain in Travis would be a hazzel, esp. with asciidoctor-fopub, but it would be interesting to try.

Then we could possibly upload the result from the branches to two different "releases", like "Official" and "Snapshot". Then those would be stable places to point to. And would contain corresponding, and updated, versions of the various documents. (Ok, only the manual would actually need the "Snapshot" version...)

I think I tried something like this (uploading to github) in some other project but did not finish it, probably because of lack of time. I would be interested to dig into this.

Maybe a new issue for it?

[tajmone]

ALAN Manual: Dev Snapshots

The "branching" model we decided upon was, as least how I see it, to maintain two version of the documentation

• master that is aligned with the current official release but that would be updated with better and better content continously
• dev-man that is aligned with the development snapshots, as new features are added this version of the documentation would also add it

Exactly. The point is that not every commit to dev-man would require rebuilding the docs; only significant changes to the documentation demand a rebuild, i.e. in order to grant developers access to bleeding edge features, or just benefit from updated contents. This judgement being arbitrary in nature, we didn't so far attempt to automate the process but just handled it by hand.

Removing the PDF from one, or both, branches would help. But that leaves us with no "dev documents" to point to.

The PDF is really only required in master branch, the dev docs could be just in HTML, after all they are just a pre-view courtesy, so one format should suffice (also, the HTML is fully standalone, so it can be downloaded directly and consulted locally).

ALAN Manual: Dev Strategies

We would continuously rebase dev-man onto master to ensure that at release a smooth merge of it onto master could be done. Thanks to asciidoc that is practically feasible. And working.

After the first conflicts problems, we've been careful to ensure that the dev branches are always rebaseable unto master, and we also did rebase dev-man whenever master was changed. So, it should be fine if we keep going this way.

Workflow Notes

(Maybe that needs to go somewhere, for posterity, and maybe myself ,-), the wiki?)

We did jot down some notes in CONTRIBUTING.md, but maybe the Wiki could be a better place for a full blown article on the strategies for specific documents maintenance. It seems that we often loose track of our knowledgebase contents (e.g. the Wiki pages on the problems with ALAN Italian have been there for years); as a general rule, every problem we faced and solved was documented somehow and somewhere, the problem is remembering where!

Since we've been using diligently Issue labels, milestones and Dashboard Projects, it should be fairly easy to find out any topic via filtered searches. But this only covers Issue, not in-repo documents or Wiki pages...

Indexing the Existing Knowledgebase

We should strengthen the Wiki of each repository, creating some sort of smart index that allows us to retrieve all these guidelines and notes. The Wiki of the alan-if/alan repository might deserve more attention in this respect, and could become the main "official Wiki", making it easier to use it a directory to find all knowledgebase articles (or links to Issues), otherwise this knowledge will be scattered across many repositories (and their Wikis). The only problem here is redundancy — i.e. some articles naturally belong to the Wiki of their repository, but having to keep an Index in every Wiki of the alan-if org might become cumbersome, unless we can come up with some sort of databse solution than can auto-generate and update all these Indexes.

It might seem overkill, but the problem here is that our projects are actually over documented (whereas usually the opposite is true). Writing guidelines and tutorials take time, so it's a pity if these articles are not brought to fruition due to lack of visibility.

CI Automation

There is a CI-job, which currently only does asciidoc validation as far as I can see.

It only validates code styles consistency via EditorConfig, actually.

Would it be a possible route to expand on that to build the documentation too? Maybe that was our intention all along. Setting up the toolchain in Travis would be a hazzel, esp. with asciidoctor-fopub, but it would be interesting to try.

I remember the problem being the asciidoctor-fopub part, which only works on Windows (see #66) due to Win-specific paths in our custom configuration (for fonts or styles, don't remember). The fopub configurations are not very friendly, and quite hostile toward collaborative editing and version control. Probably the best solution would be to check Asciidoctor's native PDF backend, which has been updated a lot in the meantime and probably solved all the issues that were preventing us from using it here (problems with footnotes inside tables, and a few other missing features, see: #9)

ALAN Manual Dev Snapshots: Release Strategy

Then we could possibly upload the result from the branches to two different "releases", like "Official" and "Snapshot". Then those would be stable places to point to. And would contain corresponding, and updated, versions of the various documents. (Ok, only the manual would actually need the "Snapshot" version...)

To achieve this we'd need to define a snapshot release strategy.

Sometime only one of the two format needs updating (e.g. because we improved the CSS of the HTML doc, or the template of the PDF); other times there might be just typo fixes, we don't necessarily call for an updated version.

Possible solutions could be:

1. Regular updated every so often — e.g. every three or six months.
2. Update the snapshot docs whenever a new Alpha version of ALAN is released.
3. Others?

The problem with (1) is that end users might have to wait to gain access to new juicy stuff; whereas with (2) the problem is ensuring that we update the contents before releasing the new Alpha, since the build jobs would be automated via some CI cross-repo communications, base on release tags or branch merges.

I think I tried something like this (uploading to github) in some other project but did not finish it, probably because of lack of time. I would be interested to dig into this.

Personally, I'm for the manual approach, after all we are always aware when new juicy contents are added to the Manual, so each one of us is free to update the HTML snapshot (and even the PDF) based on whether he thinks it's worth sharing it with bleeding-edge authors — it just boils down to deciding whether to add the built docs to the commit stage or not, so no big deal there. Since the snapshot preview links are pointing to the dev-man branch, and docs name don't change, these links will always show the latest document that was pushed.

Maybe a new issue for it?

We've had a long discussion on this in #6 (now closed), we could reuse some of the text material from that thread if it saves us some typing.

About PDF Rebase Conflicts

Back to the original question of this Issue, which type of conflicts does rebasing on master bring about?

What I usually do when these conflicts come up (inevitably for both HTML and PDF builds, due to date changes or just Asciidoctor template changes) is to simply solve the conflict with either "our" or "theirs" and then rebuild both docs on the fly and add them to the stage before committing the rebase — bear in mind that when rebasing or merging we should always rebuilt the docs from scratch (if we include them) because of the way dates and version numbers are handled, and because Asciidoctor updates often introduce CSS changes.

So the best strategy is to always manually rebuild the docs.

This can be probably achieved by some Git hooks/filters (when carrying out specific operations on dev-man, for example).