Git problems with private subodule alias

Question

git version 1.7.1 (I have no power over this, on this server)

I have a private repo with several submodules, some of which are also private. Submodules that are not private come in without problems. However, the first closed repo returned ssh: Could not resolve hostname bitbucket.org:username: Name or service not known , and the process stops here.

As documents , I configured the deployment key for the staging server and created an alias ~/.ssh/config to point the .org bitpack to this key. This works because the private parent repo just gets pulled in. The staging server has several locked files in the environment that I have to work in, so I initialize the following:

 git init git remote add origin ssh:// [e-mail] /user/parent-repo.git git fetch git checkout -t origin/master 

So far so good, but the submodules are empty:

 # Update submodules git submodule update --init --recursive # git submodule update --remote # Not available in 1.7.1 git submodule foreach git pull origin master ## git fetch --recurse-submodules=yes # introduced 1.7.3 

Everything goes wrong here, and ssh cannot connect, and the process stops at the first private repo. This is not surprising, because to access private repositories we need to associate a different key, which may not be the same as the one associated with the account (scroll down to the heading “Private repositories” for some interesting comments in this article by Luke Woodward )

So, after creating separate (I could only create one) keys and creating ssh aliases for each of them (on an intermediate server), I got stuck on how to correctly refer to this key.

To be sure that aliases work, we can test them with:

 ssh -Tv git@ <alias> 

What if the successful return of a large number of results ending in:

 .... authenticated via a deploy key. You can use git or hg to connect to Bitbucket. Shell access is disabled. This deploy key has read access to the following repositories: XXXXX 

Here is what I have tried so far. After each change, Im working:

 git submodule update --init --recursive git submodule foreach git pull origin master 

EDIT: as suggested in the comments on updating the links, at the moment a git submodule sync will synchronize all the submodules that have already completed the pull. None of the private repositories will be listed in this operation. Unfortunately, this operation does not seem to address issues with missing private repositories.

• change paths to aliases ssh .gitmodules

Result: ssh: Could not resolve hostname bitbucket.org:username

It looks like an alias is resolving, but is something blocking the key?

2. change the paths to ssh aliases in .git/config .

Result: ssh: Could not resolve hostname <alias>:username

Obviously, the alias does not allow at all.

---

As an option 1, at least I made an ssh request, I manipulated the ssh alias, which I thought sent the request to the corresponding key. There must be some kind of caching, because changing the host on an alias to something impossible, for example bitbucketXXX.org . The result was the same error: ssh: Could not resolve hostname bitbucket.org:username . I would think that the error would read ssh: Could not resolve hostname *bitbucketXXX.org:username* --HUMMMM

---

Update: after a little thought

So far, my approach has been to create an alias for the ssh key that associates bitbucket.org with the parent key that I set for this server in my personal bitpack settings.

If this did not work for the subtitles, I configure the keys for each of them, and then on the server, the aliases for each of them too, but keeping the parent repo key the same as before. So far, I have been looking for ways to reference these aliases in .gitmodules and .git/config . But this just does not work, the parent key is recognized, but aliases for submodules are not.

My last approach was to make one shared key (different from my account key, they will not allow you to do this) and assign it to the parent repo and each submodule. On the server, an alias for bitbucket.org points to this shared key.

Now, when I test it with ssh -Tv [e-mail] after a lot of output, we get:

 authenticated via a deploy key. You can use git or hg to connect to Bitbucket. Shell access is disabled. This deploy key has read access to the following repositories: *A list of all the repos associated with that key* 

Although this looks promising, the result is the same ssh: Could not resolve hostname . . . ssh: Could not resolve hostname . . .

I even tried to absorb the username into an alias like so

 Host bitbucket.org:<user_name> HostName bitbucket.org IdentityFile ~/.ssh/shared_bb_key 

but this leads to conq: invalid repository syntax.

---

At this moment I do not see the forest, not to mention the trees. Any thoughts?

Answer 1

How to use public and private submodules in repo

I needed to create a deployment strategy for wordpress that used repositories of private submodules. There is some good source in interwebs on this already, but they do not directly address private submodules. Therefore, I am documenting the steps that I have taken to complete this work with Bitbucket. Some of them may apply to other services, although I tested them only with Bitbucket. If you use Github, you may be wondering that I n this answer too .

Just for the preface, I was stuck using git 1.7.1 on my intermediate server, which, as far as I can tell, was released in early 2011, so it's only a few years. This may have complicated my situation, but if you see that I use command x instead of command Y , that’s probably why. I tried to note that I understand more modern approaches, and, of course, I will do updates with any reviews. I also tried to identify the dangers along the way, but I do not think that the problem or solution is unique to my particular host.

I assume that you already know or created submodules in your project. If not these articles, this is the place to start: https://www.atlassian.com/ git / tutorials /
http://blogs.atlassian.com/2013/03/git-submodules-workflows-tips/

For those who already know

If you already have keys and aliases, and are wondering why this does not work, skip ahead. The answer is to identify git as a user in your ssh aliases.

Ground Works: Configuring SSH Keys

First, you will need to be able to ssh on your server using the terminal, and after that you will need to set the key (or keys) for the modules that you will pull. Bitbucket allows you to share ssh keys between repositories, so more than one may be required for your use case. For simplicity, however, I will illustrate the "one key for all" approach.

Note. This key is NOT a key installed in your user account so that you can click on your repositories. Bitbucket will not allow you to use this key as deployment keys in repositories for security reasons.

In addition, since this process will only use the deployment key, we can only pull it out. Pushing back upstream is possible for the parent repo if you own it, but this repo requires a separate alias. My use case, however, is an intermediate server for testing the website, so for me it was good enough.

Atlassian provides complete instructions for generating keys here: https://confluence.atlassian.com/x/YwV9E

In short, keys are usually stored in ~/.ssh/ , and running ssh-keygen will tell you how to create a key - just remember: do not specify a passphrase .

Note I found that running keygen from my ~/.ssh caused permission problems later. This may be unique to my server, but to be safe, just start the process from your home directory.

Once the key has been created, you can see it if you are li -a ~/.ssh/ . Assuming you saved your key, cat ~/.ssh/your_bitbucket_submod.pub will return the main part of the key, so copy this (taking care not to add extra characters) and paste it into the deployment key section of each of your private repositories.

Instructions here: https://confluence.atlassian.com/display/BITBUCKET/Use+deployment+keys

Creating SSH Alises

After you have assigned the key to the repositories on Bitbucket, you will need to create ssh aliases on your server. These aliases will provide the public key for bitbucket.org when ssh is called to connect to these private repositories.

As you might have suggested, we will use ssh rather than https for all of our private repo connections.

A regular git ssh path should look like this: ssh:// [e-mail] :<USER-NAME>/repo-name , although Bitbucket will accept a path with or without ssh://

SSH aliases are relatively straightforward and are listed in ~/.ssh/config/ .

If this configuration file exists, we will modify it if we do not create it. Using your text editor, add the following lines:

 Host your_bitbucket_submod.pub HostName bitbucket.org IdentityFile ~/.ssh/your_bitbucket_submod User git 

This is the meat of what we need, or at least one important piece of it.

Just run:

• Host is the alias keyword ssh will look for in paths sent to it. It can be anything, but I often call it the same as the generated key.

• HostName is the domain that the alias points to, in this case bitbucket.org.

• IdentityFile points to the private part of your key (note no .pub ) and with two parts, ssh can verify the session.

• User indicates user ssh http://unixhelp.ed.ac.uk/CGI/man-cgi?ssh+1 See ENVIRONMENT

User ? Now this is a touch different from some of the other instructions that you will find. It turns out using ssh git points to user and puts the git account that hosts the repo (i.e. Bitbucket username) after the semicolon. This one line cost me several days of head scratches until I re-read this article by Jochen Cooperschmidt and finally understood what he was pushing for. This is the only article I have found that specifically refers to the User parameter in this context, therefore the hat tip for Jochen.

Testing Your Aliases

It is worth checking if your alias is connected, and if the key is assigned to all the repositories you expect it to be. To check, follow these steps:

 ssh -Tv your_bitbucket_submod 

If you're lucky, you should see a lot of words that end with something like this:

 authenticated via a deploy key. You can use git or hg to connect to Bitbucket. Shell access is disabled. This deploy key has read access to the following repositories: * A list of all the repos associated with this key . . . * 

No joy? Here is some help: https://confluence.atlassian.com/display/BITBUCKET/Troubleshoot+SSH+Issues

Repo setup:

Now I am not whistling with git, so there may be different opinions on how best to do this next part. My option was to place the repo in a directory with existing content that was not tracked, but should remain . I suspect this is one area that will see a bit of finesse as my best customers offer reviews.

First we will cd into the directory where the repo will be located. After that we will do the following:

 git init git remote add origin your_bitbucket_submod:<USER-NAME>/your-fav-repo.git git fetch git checkout -t origin/master 

Note that we use an alias here, since my parent repo is also private, and I have a deployment key associated with it.

Here you can configure a second alias that points to your primary account key. Although I have not tested this, this should allow the stream to push everything except submodules, even if they are under the same name as your main profile ssh key.

So, if we get to this, our alias and key work, and now the parent repo is created. You can be tempted at this point to pull out submodules with something like the following:

 # Update submodules without risk of recursion git submodule update --init --recursive ## git submodule update --remote # Not available in 1.7.1 git submodule foreach git pull origin master # git fetch --recurse-submodules=yes # introduced 1.7.3 

If you have done this, you should see successful transfers of any public submodules that you might have, but as soon as a private repo occurs, the process will stop and you will see.

If you have submodules inside your submodules, there is a risk of an infinite cloning loop. This two-step upgrade method should prevent this .

 ssh: Could not resolve hostname bitbucket.org:<USER-NAME> 

Hu ?!

Alternatively, you may need to enter a password, in which case your submodules will be https , and they will need to be converted to ssh paths.

Configuring .bitbucket/config :

When you first checked the parent repo, if git found submodules, it will do a number if hidden files and directories keep track of them ( ls -a to open them). We will need to edit (with vim, nano, etc.) the .git/config file and change the paths listed here with our alias to the previously created development key.

The configuration file will look something like this:

 [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = your_bitbucket_submod:<USER-NAME>/parent-repo.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master [submodule "plugins/akismet"] url = https://github.com/git-mirror/wordpress-akismet.git [submodule "plugins/your-famous-plugin"] url = ssh:// [e-mail] :<USER-NAME>/your-famous-plugin.git . . . 

Note that the url under [core] already smoothed from the moment you specify remote add origin . Later, for the akismet plugin, we see its public repo on github connecting via https, and finally, the private repo on Bitbucket. Here you will change all the links, for example:

ssh:// [e-mail] :<USER-NAME>/your-famous-plugin.git

to: your_bitbucket_submod:<USER-NAME>/your-famous-plugin.git

Once this is done, we can re-update the submodules:

 git submodule update --init --recursive git submodule foreach git pull origin master 

And watch them come.,.

Any failures at this point will either be in some distorted way when you specify your submodules, or when setting up a configuration file. Or perhaps the deployment key is actually not tied to the repo in the bitpack.

Hope this helps a few people.

Further reading:

• https://chrisjean.com/git-submodules-adding-using-removing-and-updating/
• https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407

Answer 2

Concerning

 ssh: Could not resolve hostname 

part of this problem that many are facing, SSH has two different configuration files: a system-wide configuration file and a user configuration file. In the instructions that I read, including Shannon Chou's, everyone says to add aliases to the configuration file for the user , which is located in ~ / .ssh / .config

In my case, I needed to add aliases to the system configuration file, which when using Git in Windows 10 is usually located here: C: \ Program Files \ Git \ etc \ ssh \ ssh_config, in the Git directory.

You can determine which SSH configuration file is used by running this command, "myalias" can be any line that interests us, this is the path to the configuration file that will be displayed:

 ssh -vv myalias OpenSSH_7.1p2, OpenSSL 1.0.2d 9 Jul 2015 debug1: Reading configuration data /etc/ssh/ssh_config 

Please note that the output file contains the path to the file "/ etc / ssh / ssh_config". This tells us that SSH is looking for aliases there, and not in the ~ / .ssh / .config file.