Trouble with initial push of a very large repo
TheHarvesteR
on Tue 08 Jan, 2013 21:19:10 +0000
Hi,
We’re currently in the process of moving our entire project over to a git repo here. I’ve got the local repo all set up, and ready to go, but I’m having issues pushing it to xpdev.
The repo currently holds 18,650 objects, and totals in around 16GB (project files plus tracking metadata). Quite a large thing, no question about it, but it needs to go up otherwise we’re dead in the water.
So, here’s what’s happening. I do a git push origin master, and it starts counting files, and then proceeds to compress them, which takes a few minutes. At any point during that time, I get the message “Fatal: the remote end hung up unexpectedly”. The compression process continues though, and once it’s done, the push fails completely with this:
fatal: sha1 file '<stdout>' write error: Invalid argument
error: failed to push some refs to 'ssh://git.xp-dev.com/KSPGit'
This is the complete output for a failed push attempt:
$ git push origin master
Counting objects: 18650, done.
fatal: The remote end hung up unexpectedly
Delta compression using up to 8 threads.
Compressing objects: 100% (18488/18488), done.
fatal: sha1 file '<stdout>' write error: Invalid argument
error: failed to push some refs to 'ssh://git.xp-dev.com/KSPGit'
Anyhow, I looked around, and saw another thread that said the requests are failing because of a load balancer issue, which suggested bumping up the http.postBuffer value to something quite high, like 500MB. I set it to 18GB, which seems rather extreme, and even so, it didn’t help. The connection drops out before the compression step is done, and then everything fails.
Needless to say, this is a project-critical issue, so I appreciate any help very much!
Thanks in advance,
Cheers
Forum Topic and Comments
Reply
Finished. Yeah, seems the write process just didn’t get told the push had failed and kept going anyway.
So, if the connection failed at around 25% of the way in (and on a previous attempt it failed around 23%), it seems there is some sort of timeout issue happening here, which cuts off the connection before the push can complete. I figure that’s probably because these processes are not supposed to take as long as they’re taking now (because this is a huge commit), and I think it wasn’t even able to start uploading anything by then...
So, that’s back to square one again... Any help at this point would be greatly appreciated.
Cheers
Another follow-up:
I gave up on trying to upload everything in one push here. I deleted all the git metadata, and restarted the local repo. Now I’m in the process of doing many small (under 500mb) commits. It’s a tedious process, but at least we’re making progress.
Cheers
About 18GB, counting project files and version control metadata.
Well, after a day of doing small bite-sized pushes, we’re almost done uploading. Just a few more gigs to go.
BTW, I’m back to using ssh now, https is downloading faster it seems, but it’s not uploading very well, and ssh doens’t ask for the password all the time. :)
Cheers
Ok, after three days of uploading, we’ve got our entire project uploaded to the new git repo.
There’s just a bit of an issue today, we’re getting very, very slow speeds when pulling here... Around 20-30kb/s. Is that a known issue, or is there something we can try to speed things up?
Cheers
That’s definitely not normal
Do you mind raising a support ticket with details of your IP address, and a tracepath/tracert to xp-dev.com please ? Also, please do mention the repository name as well.
Filed a support ticket with tracerts from my PC and our build server PC, which is having the most trouble.
Thank you for the very quick response.
Cheers
You do not have sufficient permissions to comment
Comment: About 18GB, counting project files and version control metadata. Well, after a day of doing small bite-sized pushes, we're almost done uploading. Just a few more gigs to go. Cheers → About 18GB, counting project files and version control metadata. Well, after a day of doing small bite-sized pushes, we're almost done uploading. Just a few more gigs to go. BTW, I'm back to using ssh now, https is downloading faster it seems, but it's not uploading very well, and ssh doens't ask for the password all the time. :) Cheers
View full history
TheHarvesteR on Tue 08 Jan, 2013
Ok, here’s a follow up:
I was using ssh so far, and tried switching over to https. This did produce a different result. Now, it’s able to get past compression, and it starts “writing objects”.
It seems the data it needs to write adds up to around 4GB. As of right now, I have it 64% done, but around 25% of the way in, I got a message saying the remote end hung up. This time with some more information. It says: "RPC Failed; result=22, HTTP code = 411"
However, the writing objects process is still going. I think it’s probably going to fail once it ends though, but I’m waiting to see what happens. It migth succeed somehow, or if it fails, at least provide some new information.
Cheers