Can't git clone from Gitlab from within a grain #3644
Comments
|
@zenhack Do you have any suggestions on this one? |
|
Is the code for this publicly available? Is there anything of interest in sandstorm.log? Because of the way the HTTP proxying over capnp works, sometimes information about actual errors gets obscured by the time the grain sees a response, but errors in logs can be illuminating. (It is possible that GitLab is not giving you a 500, but somewhere inside sandstorm some error is occurring). |
No, it's very WIP at the moment.
I checked the grains log and I was just seeing I've managed to get Using the So I know the main use case for the |
|
We have multiple apps which use Git repos, none of which use IpNetwork, so you should be okay there. You may want to look at our GitWeb package. (We have GitLab too, but it is much older, I believe.) |
|
@ocdtrekkie, I believe @moyamo is trying to use a git client from inside a grain, rather than serve a git repo from a grain -- so we don't actually afaik have existing apps that do this. I would hazard a guess the issue is either in our implementation of ApiSession (it seems entirely possible that git clone does something our current implementation isn't handling correctly), or in the way you're trying to use it in the app code. Do you have an objection to publishing the code (or some simplified version that exhibits the problem)? I feel like it would save a lot of time for me to just be able to see what you're doing. |
|
So I checked the sandstorm logs and it didn't log any errors so I'm 90% sure that the 500 error is coming from Gitlab and not from the Here's some snippets of the code: First we create a token that we can claim to access Gitlab. @0x9759ad011d40ab4c; # generated using `capnp id`
using Powerbox = import "/sandstorm/powerbox.capnp";
using ApiSession = import "/sandstorm/api-session.capnp".ApiSession;
const myTagValue :ApiSession.PowerboxTag = (
canonicalUrl = "https://gitlab.com/api/v4",
authentication = "basic",
);
const myDescriptor :Powerbox.PowerboxDescriptor = (
tags = [
(id = 0xc879e379c625cdc7, value = .myTagValue)
],
);NOTE we set We do the thingy described in the docs to get the base64-encoded version of this constant. descriptor = "EA5QAQEAABEBF1EEAQH/x80lxnnjecgAQAMRCdIAABERMv9odHRwczovLwJnaXRsYWIuY29tL2FwaS92ATQfYmFzaWM="Then we use a jinja2 template to put the token in the HTML we do the thingy in the docs again to get the claim token. <!doctype HTML>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<button onclick="connectGitlab()">Connect to Gitlab</button>
<button onclick="clone()">Clone</button>
<script>
function connectGitlab() {
window.parent.postMessage({
powerboxRequest: {
rpcId: 1,
query: [
"{{descriptor}}"
],
saveLabel: {defaultText: "gitlab API access"},
}
}, "*");
}
window.addEventListener("message", function (event) {
if (event.source !== window.parent) {
// SECURITY: ignore postMessages that didn't come from the parent frame.
return;
}
var response = event.data;
if (response.rpcId !== 1) {
// Ignore RPC ID that dosen't match our request. (In real code you'd
// probably have a table of outstanding RPCs so that you don't have to
// register a separate handler for each one.)
return;
}
if (response.error) {
// Oops, something went wrong.
alert(response.error);
return;
}
if (response.canceled) {
// The user closed the Powerbox without making a selection.
return;
}
// We now have a claim token. We need to send this to our server
// where we can exchange it for access to the remote API!
doClaimToken(response.token);
});
async function doClaimToken(token) {
r = await fetch("/token", {method: "POST", body: token})
}
async function clone() {
r = await fetch("/clone", {method: "POST"})
}
</script>
</body>
</html>Then we claim the token and store it in the file @app.route("/token", methods=["POST"])
def token():
tok = request.data.decode('utf-8')
session_id = request.headers.get("X-Sandstorm-Session-Id")
r = requests.post(f"http://http-bridge/session/{session_id}/claim",
headers={"Content-Type": "application/json"},
json={"requestToken": tok, "requiredPermissions": ["read"]}
)
gitlab_cap = r.json()['cap']
with open('/var/bearer.txt', 'w') as f:
f.write(gitlab_cap)
return ''Now comes the interesting part. Unfortunately HOME=/var/ mitmdump --mode upstream:$HTTP_PROXY -s /opt/app/gitproxy.py &NOTE
#!/usr/env/bin python
def request(flow):
try:
with open("/var/bearer.txt") as f:
bear = f.read().strip()
flow.request.headers["Authorization"] = "Bearer " + bear
except Exception:
passNow we try to git clone the repository @app.route("/clone", methods=["POST"])
def go():
os.chdir('/var')
proxy = "http://localhost:8080" # This is mitmdump
subprocess.run(['rm', '-r', 'myrepo'])
# Try with --depth=1 (succeeds for some reason)
subprocess.run(["git", "clone", "http://http-proxy/moyamo/myrepo.git", "--depth=1"], env={"http_proxy": proxy, "HTTP_PROXY": proxy})
subprocess.run(['ls', '-l']) # Verify repo is cloned
subprocess.run(['rm', '-r', 'myrepo'])
# Try again without --depth=1 (fails for some reason)
subprocess.run(["git", "clone", "http://http-proxy/moyamo/myrepo.git"], env={"http_proxy": proxy, "HTTP_PROXY": proxy})
subprocess.run(['ls', '-l']) # Show repo is not cloned
return ''The output in the grains logs is Given that the |
|
Yeah, if it works with |
|
So if I set If I set |
|
On the other hand maybe this is a different error. Maybe when I do a |
Hm, I know we've had problems with larger requests in the other direction, but I can never remember if those got solved (@ocdtrekkie, ring any bells?) |
|
Only thing I know of with large transfers was the whole range request thing, which is still an open PR, and I am turned around enough to not know which way that was or which way this is, which may not be much help to anyone. |
I'm trying to write an app that
git clonefrom a private Gitlab repo into a grain. I've used the powerbox to pass the URL of the repo and the Personal Access Token to my grain. If Igit clone --depth=1, everything works fine, so I've setup everything correctly.However if I
git clonewithout the--depthflag I get an500 Internal Server Errorfrom Gitlab. The 500 error is in response to git doing aPOST <repo>/git-upload-pack.Is there anything weird that the sandstorm-http-bridge could be doing that could cause git clone to fail?
P.S. I've written an http proxy that sits between git and sandstorm-http-bridge to add the
Authorization: Bearer <token>but otherwise passes the requests verbatim to the sandstorm-http-bridge.The text was updated successfully, but these errors were encountered: