Has anyone used Git submodules with the Git plugin?

[chrispitude]

Hi all,

I tried to use the Git submodule feature of the Git plugin today, but didn't work as I expected.

When I switch to a submodule, the plugin shows every file in the submodule as having an uncommited local change:

image.png

If I select all files and Discard them, the list becomes empty - but only for several seconds, then all files are listed as changed again.

If I pick any file and choose "Open in compare editor", the diff windows always shows zero changes.

Because all files are seen as having uncommitted local changes, I can't pull any new updates down from the submodule's remote repository:

image.png

Is anyone using Git submodules in their projects?

[alex_jitianu]

Hi Chris,

I haven't encountered this situation, myself. What you describe, might be a line separator issue. You can open one of those files that appear as modified in an Oxygen editor and then go to Tools->Hex Viewer. If you see 0D 0A sequence then it might be something related with the core.autocrlf setting. You can check to see if this setting is set:

Code: Select all

git config --get core.autocrlf

and try to disable it to see if anything changes. Perhaps it is better to do it it on that specific project in order not to affect the global setting:

Code: Select all

git config --local core.autocrlf false

Best regards,
Alex

[chrispitude]

crlf-test.tgz

Hi Alex,

So you have pointed me down the right path, and it looks like it's not related specifically to submodules at all.

We have a mixed environment with multiple Git clients:

• Oxygen for Windows Git client
• Git for WIndows
• Git command line in Ubuntu linux

I created LF and CRLF versions of some DITA files:

Code: Select all

crlf-test$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:16 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:16 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:16 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:16 unix/topic_1.dita

I used them to upload an Oxygen test project (attached) on our Git server.

crlf-test.tgz

If I clone this repo in a linux shell, the initial checkout preserves the original format and nothing is shown as modified:

Code: Select all

$ git clone git@gitsnps.internal.synopsys.com:chrispy/crlf-test.git
Cloning into 'crlf-test'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Resolving deltas: 100% (2/2), done.
$ cd crlf-test/
$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:32 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:32 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:32 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:32 unix/topic_1.dita
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$

But when I use the Oxygen Git plugin to clone this project on my Windows laptop, the files in the dos directory are immediately listed as modified, despite the fact that I haven't changed anything at all:

image.png

What's strange is that (1) those files are in dos (CRLF) format so I don't understand why it's complaining, and (2) the Git plugin converted the unix files to dos upon the initial commit:

Code: Select all

-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 unix/topic_1.dita

If Oxygen's Git plugin wants to convert everything to CRLF format upon checkout, why would it list the files in the dos directory as modified?

And why would it immediately list files as modified after cloning a repo, no matter what they are? That seems internally inconsistent to me.

[chrispitude]

Here's another strange finding - if I discard the files, they come back listed as "modified" when I switch focus away from Oxygen and switch back!

crlf.gif

[chrispitude]

Okay, I think I figured it out.

On my Windows 10 machine, I have Oxygen for Windows installed, and I have Ubuntu linux installed via WSL (Windows Subsystem for Linux).

• By default, Oxygen's Git plugin converts text files to CRLF upon checkout.
• In that same repo directory, if I commit a CRLF file via WSL, I get the bad behavior.

Rather than look at solutions using the .git/config file, I decided to explore the .gitattributes file because it allows more detailed configuration of the files in the repo. This is what I'm testing now:

Code: Select all

# For files with no extensions, let Git detect if they're binary
* text=auto

# By default, all files with extensions are converted to LF only, so Windows does not break them
*.* text eol=lf

# Override the following filetypes to be binary
*.[gG][iI][fF] binary
*.[jJ][pP]*[gG] binary
*.[pP][nN][gG] binary
*.[eE][xX][eE] binary
*.[pP][dD][fF] binary
*.[sS][nN][aA][gG]* binary
*.[vV][sS][dD]* binary

I chose to configure binary file extensions explicitly and let everything else fall back to ASCII, but I could have done it the other way around, or I could list both ASCII and binary extensions explicitly.

There are two places this file can be placed:

• <repo>/.git/info/attributes - affects only your local repo, nobody else
• <repo>/.gitattributes - stored in repo, affects everyone

For now, I put it in .git/info/ to see if it breaks anything in my daily use. When I trust it, I can roll it out to the wider team.

Placing the attributes file in the repo affects future file pulls, but does not alter the current files on-disk. I was able to test the settings on the existing files by doing a "Remove from Disk" in Oxygen, then doing a "Discard" of all the file deletions in the Git plugin, which caused the files to get pulled in LF form.

Now that I have a solution to this problem, I'll go back and update my Git submodule repo accordingly and see how it goes!

[chrispitude]

Hi Alex,

I did some experimenting with Git submodules through the Oxygen Git plugin. The basic functionality is there, but some of our writers are nontechnical and anything beyond a pushbutton interface will invite confusion.

Accordingly, I have the following enhancement requests:


1. When you clone a repo in the plugin (the "+" button), Oxygen should populate submodules too. This can be done in two separate steps:

Code: Select all

# clone repo, THEN clone submodules -- in two separate steps
git clone <repo>
git submodule update --init --recursive

or a single step:

Code: Select all

# clone repo, plus its submodules
git clone <repo> --recurse-submodules

2. Submodule updates should be easier (or automatic). Currently when a submodule needs an update, the submodule root shows up in the modified files list:

image.png

This can be confusing because the submodule was changed externally, unlike other files in the list that were changed locally. In addition, the mechanics of switching to the submodule, updating, and switching back are intimidating for someone nontechnical.


I see the value in allowing explicit recursion into the submodule, for cases where multiple people are working on the submodule contents, there might be conflicts, you might need to perform conflict resolution, etc. But for the simple case where shared files are modified by one administrator and rolled out read-only to a team (DITAVAL files, CSS files, etc.), a pushbutton update would make this much easier.

Perhaps like our other discussions on auto-updating, we could provide more automation for submodule updates that have no conflicts?

[alex_jitianu]

Hi Chris,

1. I agree, chances are that, for our audience, initializing the submodules automatically is better. Otherwise, you currently get an empty directory in your file system and you don't know what to do. I've added an issue to add an option for this, with the default being to automatically initialize them.

2. Yes, working with submodules can be a challenge.

Currently when a submodule needs an update, the submodule root shows up in the modified files list:

As far as I know, the submodule appears as modified after the user entered it and run git pull (or similar), right? I'll add an issue to see how we can make things easier.

[jhanna]

Please add my vote for implementing changes that would make working with submodules easier.

On a related note, has any consideration been given to creating a tool for working with subtrees? From what I have seen, subtrees are easier to work with than submodules for pulling updates.

Regards,
Jonathan

[sorin_carbunaru]

Hi Jonathan,

I added your vote to the issues related to working with submodules. I also created a new feature request for adding support for subtrees. By the way, I would say that perhaps adding support for subtrees has a higher priority than improving submodules. What do you think?

@Chris, do you have anything to say about subtrees? Would you prefer them instead of submodules?

Best wishes,
Sorin Carbunaru
Oxygen XML Editor

[jhanna]

Hi Sorin,

Since the submodule feature already works, I would agree that adding support for subtrees would take priority over submodule improvements.

Regards,
Jonathan

[sorin_carbunaru]

Hi all,

Just wanted to say that we released version 2.5.0 of the Git Client, and this version received some improvements on submodules. You can find them in the release notes from https://github.com/oxygenxml/oxygen-git ... /tag/2.5.0.

Best wishes,
Sorin Carbunaru
Oxygen XML Editor

[jhanna]

Hi Sorin,

Thank you for the update and for continuing to add improvements for working with submodules! I am particularly intrigued by the following improvement that was added:

When pulling the remote changes for a repository that contains submodules, the submodules are updated as well (by default). This behavior depends on the Update all submodules after pulling changes from the remote repository option from the Git Client preferences page in Oxygen.

How exactly is this supposed to work? Could you provide an example?

I tried testing this feature out by making a change to a remote repository, then pulling the changes. However, the submodules did not update. I still had to manually open the submodule (using the Submodules button in the Git Staging plugin) to update the submodule. I double-checked the preferences and the "update all submodules" option was selected.

Interestingly, I noticed that when I opened the submodule, it was in a detached HEAD state, pointing to a specific commit. I previously had the submodule pointing at the main branch. It switched to the detached HEAD state automatically after I pulled changes to the parent repository.

Best regards,
Jonathan

[chrispitude]

Thanks Sorin, I can't wait to try the improvements!

Sorry I didn't see your question about subtrees until now. For our purposes, I think submodules are more suitable. We have three Git repos of DITA documentation. Each of them has a prj/ directory that contains all our Oxygen configuration data - frameworks, DITA plugins, external tools, and all that fun stuff. I want to create a Git repo for this prj/ stuff, then instantiate it (as a submodule) in each of the repos. This way, I can make Oxygen environment changes in one place, then roll them out to all three teams. I believe submodules are the best fit for this.

[alex_jitianu]

Hi Jonathan,

How exactly is this supposed to work? Could you provide an example?

First of all, the option Update all submodules after pulling changes must be set. The scenario is as follows. Lets assume we have a main repository with a second submodule:
1. User1 open main in the Git Staginf view. It uses the Submodules action on the toolbar and goes into the second submodule
2. User1 moves the submodule to a newer commit. In Oxygen it has to move from DETACHED HEAD to a brach, pull changes and maybe use the History view to switch to another commit
3. User1 returns to main and commits/pushes the submodule binding
4. User2 also works in the main repository. When it Pulls the change we also run a git submodule update. Depending on the module update behavior, its second submodule might jump to the DETACHED HEAD state, on the commit bound by User1.

Best regards,
Alex

[jhanna]

First of all, the option Update all submodules after pulling changes must be set. The scenario is as follows. Lets assume we have a main repository with a second submodule:
1. User1 open main in the Git Staginf view. It uses the Submodules action on the toolbar and goes into the second submodule
2. User1 moves the submodule to a newer commit. In Oxygen it has to move from DETACHED HEAD to a brach, pull changes and maybe use the History view to switch to another commit
3. User1 returns to main and commits/pushes the submodule binding
4. User2 also works in the main repository. When it Pulls the change we also run a git submodule update. Depending on the module update behavior, its second submodule might jump to the DETACHED HEAD state, on the commit bound by User1.

Hi Alex,

Thank you for the example. Unfortunately, I still cannot get the submodules to automatically update when pulling changes to the parent repository. Either I am doing something wrong, I am not understanding how this function is supposed to work, or there is a bug.

If I understand the Stack Overflow article you provided, I am supposed to add update = merge to the submodule in the .gitmodules file. I tried doing this and it appears to not have any effect. When I open the submodule, I still see it in a detached HEAD state, even though I previously checked out the main branch. I never experienced this issue in previous versions of the git plugin, which is why I am wondering if there is a bug. Another item to note, in previous versions of the git plugin, I would get a notification that the submodule was in a detached HEAD state. I no longer get these notifications in the current version of the plugin.

Would it be possible for you to provide a video demonstration on how this feature is supposed to work?

Best regards,
Jonathan

[alex_jitianu]

Hi Jonathan,

I, too, see a difference beween our Git client and the command line git client. I've setup a .gitmodule file in the main repository like this:

Code: Select all

[submodule "git_test"]
	path = git_test
	url = https://github.com/AlexJitianu/git_test.git
	branch = master
	update = merge

A git submodule update leaves the submodule in its current state, on a local branch. Meanwhile, Oxygen will switch the module to the commit recorded in the main repository. I'll add an issue to investigate this different behavior. You can disable Update all submodules after pulling changes from the remote repository in preferences until we understand why it behaves differently.

Another item to note, in previous versions of the git plugin, I would get a notification that the submodule was in a detached HEAD state. I no longer get these notifications in the current version of the plugin.

We removed that warning because a submodule is many time by design in a detached HEAD state. This is what you do if the main repo and the module have different release cycles and you want to control what content gets pulled in the main repo.


Best regards,
Alex

[jhanna]

Hi Alex,

Thank you for the additional information. I will disable the Update Submodules feature for now. Thank you again for all the work you are putting in to support submodules!

Best regards,
Jonathan

[chrispitude]

We removed that warning because a submodule is many time by design in a detached HEAD state. This is what you do if the main repo and the module have different release cycles and you want to control what content gets pulled in the main repo.

Hi Alex,

I haven't had a chance to dig into submodules yet (busy with my "day job" as technical writer these days), but I wanted to note that this will likely be the case for us. Our submodule will contain all our Oxygen project/framework stuff. That submodule repo will accumulate fixes/enhancements over time, but we will control how we roll these out to the full Oxygen/DITA repo environment.

Agreed with Jonathan - thank you for all the work you are putting into submodules, and to Git support in general!

[jhanna]

A useful feature to add would be the ability to update all submodules with a single click of a button. Currently, updating submodules is a bit cumbersome, especially if a repository contains several submodules. It is also a bit confusing since it is not a straightforward process to change the working copy back to the main repository after checking out a submodule.

Basically what I am looking for is a button that will run the command:

Code: Select all

git submodule update --recursive --remote

After that, users can stage and commit the changes as they usually would.

Would this be possible or is there a better solution?

[alex_jitianu]

Hello,

We had a prior request for such a feature, but we didn't get the chance to implement it yet. I will add your vote for it and increase its priority. When we manage to do it and a release it, we will notify you on this thread.
Right now, I'm afraid that there is not an alternative to opening each project and performing a pull.

Best regards,
Alex

[pieterjan_vdw]

Hi,
Please add my vote for this feature too. Would be very handy to update all submodules at once.

Kind regards,
Pieterjan

[alex_smarandache]

Hi Pieterjan,

Thank you for your request.
I added your vote to our internal issue to increase its priority.

Best regards,
Alex Smarandache