Advanced Git Commands for Troubleshooting and Debugging

Posted by

When you use Git, things don’t always go as planned or as you would expect. Here are some pointers for identifying and fixing Git problems.

Debugging

When troubleshooting problems with Git, try these debugging techniques.

Use a custom SSH key for a Git command

Debug problems with cloning

For Git over SSH:

For Git over HTTPS:

Debug Git with traces

Git includes a complete set of traces for debugging Git commands, for example:

  • GIT_TRACE_PERFORMANCE=1: enables tracing of performance data, showing how long each particular git invocation takes.
  • GIT_TRACE_SETUP=1: enables tracing of what git is discovering about the repository and environment it’s interacting with.
  • GIT_TRACE_PACKET=1: enables packet-level tracing for network operations.

Broken pipe errors on git push

It is possible to encounter “broken pipe” problems while trying to push to a remote repository. When you push, you typically observe:

Write failed: Broken pipe
fatal: The remote end hung up unexpectedly

To fix this issue, here are some possible solutions.

Increase the POST buffer size in Git

You may want to try setting the POST buffer size larger in the Git configuration if you’re using Git over HTTP rather than SSH.

An illustration of a clone error might be: fatal: inflate returned -5; pack has bad object at offset XXXXXXXXX.

Launch a terminal, then type in:

Since the amount is given in bytes, the buffer size has been set to 50 MB in the example above. It starts at 1 MB.

RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

A sluggish internet connection could be the source of this issue. Try one of the following workarounds if you use Git over HTTP rather than SSH:

  • Increase the POST buffer size in the Git configuration with git config http.postBuffer 52428800.
  • Switch to the HTTP/1.1 protocol with git config http.version HTTP/1.1.

If none of the methods resolves the issue, you could require a different internet service provider.

Check your SSH configuration

Before pushing via SSH, make sure everything is configured correctly because underlying SSH problems, like login problems, can occasionally be the source of “broken pipe” errors. Follow the steps in the SSH troubleshooting handbook to ensure that SSH is configured correctly.

You may also stop session timeouts if you’re a GitLab administrator with server access by setting up SSH keep-alive on the client or server.

Configuring both the client and the server is unnecessary.

To configure SSH on the client side:

  • On UNIX, edit ~/.ssh/config (create the file if it doesn’t exist) and add or edit:

  • Set the value of Seconds between keepalives (0 to turn off) to 60 while utilising PuTTY on Windows. To do this, navigate to your session options, select “Connection,” and then click on “Sending of null packets to keep session active.”

To configure SSH on the server side, edit /etc/ssh/sshd_config and add:

Running a git repack

If ‘pack-objects’ type errors are also being displayed, you can try to run a git repack before attempting to push to the remote repository again:

git repack
git push

Upgrade your Git client

If you are using Git less than 2.9, you should think about updating to >= 2.9 (see Broken pipe while pushing to Git repository).

ssh_exchange_identification error

Users may experience the following error when attempting to push or pull using Git over SSH:

Please make sure you have the correct access rights
and the repository exists.
...
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

or

ssh_exchange_identification: Connection closed by remote host
fatal: The remote end hung up unexpectedly

or

kex_exchange_identification: Connection closed by remote host Connection closed by x.x.x.x port 22

This error typically indicates that SSH connections are being throttled by the MaxStartups parameter of the SSH daemon. The maximum number of simultaneous, unauthenticated connections to the SSH daemon is specified by this option. Because every connection is initially “unauthenticated,” this affects users who have the correct authentication credentials (SSH keys). Ten is the default value.

Increase MaxStartups on the GitLab server by adding or modifying the value in /etc/ssh/sshd_config:

MaxStartups 100:30:200

Up to 100 SSH sessions are permitted without limitation, after which 30% of connections are dropped until a maximum of 200 is reached. This configuration is represented as 100:30:200.

Once you’ve changed the MaxStartups setting, review the configuration for any mistakes.

sudo sshd -t -f /etc/ssh/sshd_config

Restarting the SSH daemon should be safe if the configuration check completes without any issues and the update takes effect.

# Debian/Ubuntu
sudo systemctl restart ssh

# CentOS/RHEL
sudo service sshd restart

Timeout during git push / git pull

A timeout is triggered if retrieving or pushing data from/to your repository takes longer than fifty seconds. It includes a record of the quantity of operations carried out along with the times of each, like in the following example:

remote: Running checks for branch: master
remote: Scanning for LFS objects... (153ms)
remote: Calculating new repository size... (cancelled after 729ms)

This might be used to look into the specific operation that isn’t working well and give GitLab more details on how to make the service better.

git clone over HTTP fails with transfer closed with outstanding read data remaining error.

Sometimes, when cloning old or large repositories, the following error is thrown:

Git’s inability to manage large files or vast volumes of files is a common cause of this issue. To address this issue, Git LFS was developed, however even it has its drawbacks. Usually, one of the following causes it:

  • The number of files in the repository.
  • The number of revisions in the history.
  • The existence of large files in the repository.

The root causes vary, so multiple potential solutions exist, and you may need to apply more than one:

You can set the cloning depth to 1 if this problem happens when cloning a large repository. As an illustration:

variables:
  GIT_DEPTH: 1

You can increase the http.postBuffer value in your local Git configuration from the default 1 MB value to a value greater than the repository size. For example, if git clone fails when cloning a 500 MB repository, you should set http.postBuffer to 524288000:

# Set the http.postBuffer size, in bytes
git config http.postBuffer 524288000

You can increase the http.postBuffer on the server side:

  1. Modify the GitLab instance’s gitlab.rb file:

2. After applying this change, apply the configuration change:

sudo gitlab-ctl reconfigure

For instance, adjusting the depth should resolve issues with repositories having a lengthy history without many huge files. But even a depth of 1 could be too big in a repository with a lot of huge files, necessitating the postBuffer modification. The problem remains even after you raise your local postBuffer if the NGINX value on the backend is still too little.

It’s not always possible to modify the server, and doing so increases the potential risk. Make local adjustments first.

Password expired error on Git fetch via SSH for LDAP user

If git fetch returns this HTTP 403 Forbidden error on a self-managed instance of GitLab, the password expiration date (users.password_expires_at) for this user in the GitLab database is a date in the past:

Your password expired. Please access GitLab from a web browser to update your password.

Requests made with a SSO account and where password_expires_at is not null return this error:

"403 Forbidden - Your password expired. Please access GitLab from a web browser to update your password."

To resolve this issue, you can update the password expiration by either:

  • Using the gitlab-rails console:
gitlab-rails console
user.update!(password_expires_at: nil)

Using gitlab-psql:

 # gitlab-psql
 UPDATE users SET password_expires_at = null WHERE username='<USERNAME>';

The bug was reported in this issue.

Error on Git fetch: “HTTP Basic: Access Denied

If you receive an HTTP Basic: Access denied error when using Git over HTTP(S), refer to the two-factor authentication troubleshooting guide.

401 errors logged during successful git clone

When cloning a repository via HTTP, the production_json.log file may show an initial status of 401 (unauthorized), quickly followed by a 200.

This initial 401 log entry is expected because of the way that HTTP Basic authentication functions for Git operations carried out over HTTP.

The initial request sent to GitLab by the Git client does not include any authentication information when it starts a clone. For that request, GitLab returns a 401 Unauthorised response. The Git client sends a follow-up request with authentication information a few milliseconds later. The outcome of this second request ought to be a 200 OK log entry.

If a 401 log entry lacks a corresponding 200 log entry, the Git client is likely using either:

  • An incorrect password.
  • An expired or revoked token.

If not rectified, you could encounter 403 (Forbidden) errors instead.

A 403 (Forbidden) error when attempting to use Git over HTTP means that the failed-authentication ban has blocked your IP address:

fatal: unable to access 'https://gitlab.com/group/project.git/': The requested URL returned error: 403

The 403 can be seen in the production_json.log:

If your IP address has been blocked, a corresponding log entry exists in the auth_json.log:

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x