-
Notifications
You must be signed in to change notification settings - Fork 420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add git-lfs to image #654
Comments
I would want an e2e test for this - how do I force this to be used so I know it works? |
Do existing tests not cover this? Can I not just add git-lfs package in a Dockerfile using apt-get then run tests on new image? My background is sysadmin. I'm new to devops. I know some bash, docker, and k8s. You want me to write a test to prove that it works? Not sure how to do that. I forked the repo. I'm looking at Dockerfile.in and test_e2e.sh. I'm not sure where to add git-lfs package in Dockerfile.in. Do I pass it an image:tag as an argument to run tests? |
If you can show me how to craft an e2e setup which specifically proves
LFS works, that would be great. You can see in that file that each
testcase is a little snip of shell code which sets up a scenario.
What I don't quiote grok is how to set up a repo to use LFS and how to
*prove* that it works.
…On Thu, Dec 22, 2022 at 4:56 PM James Ringer ***@***.***> wrote:
Do existing tests not cover this? My background is sysadmin. I'm new to devops. I know some bash. You want me to write a test to prove that it works? Not sure how to do that. I went ahead and forked the repo. I'm looking at test_e2e.sh. How can I help?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
Does this help? GitHub - Configuring Git LFS GitHub provides some storage and bandwidth for LFS for free. GitHub - About Storage and Bandwidth We can do a call and work through it together if you'd like. I set it up on BitBucket, but it's probably similar on GitHub. The BitBucket repo I have setup is private, but I could setup a public repo if that helps. Afaik the only thing that will be different after adding the git-lfs package will be that if a repo contains lfs files they will be synced rather than the text files that point to them. This makes me think existing tests should be fine, but I'm far from and expert. |
To do an e2e I need some way to replicate it locally or at least without
any auth to an external service. Possible?
…On Thu, Dec 22, 2022, 5:23 PM James Ringer ***@***.***> wrote:
Does this help? GitHub - Configuring Git LFS
<https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage>
We can do a call and work through it together if that helps. I set it up
on BitBucket, but it's probably similar on GitHub.
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVG5E3GCH32QE7KMKMDWOT5I7ANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
If you setup a public GitHub repo with LFS I don't think you need authentication. |
Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo? |
The existing e2e tests use a `file://` repo or an SSH repo (still backed by
files).
It looks like we can build an LFS test-server binary (
https://github.com/git-lfs/lfs-test-server) to run tests against, but
there's not a standard apt pkg for it, so we'll have to pull and build it
as part of the test. Look in _test_tools. That's not impossible, it just
makes this a little more involved. At least it is written in Go, so we can
drop it into tools/go.mod. :)
If you have bandwidth, it's appreciated, otherwise I'll get to this
eventually, I hope. :)
…On Fri, Dec 23, 2022 at 7:10 AM James Ringer ***@***.***> wrote:
Do the existing tests use a git repo to test? Maybe just add an LFS file
to that repo?
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVFGCCNKCP5ONWDXENDWOW6E7ANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
In my fork I have setup lfs-test-server using the sshd folder as an example.
I am using env vars LFS_USERNAME and LFS_PASSWORD to configure lfs-test-server management username and password. Looking at sshd example I'm assuming we don't want these committed to the repository. Not sure how I'm going to do that. If you don't set these the management interface is not accessible. Also, it looks like you have to login and create a user to use lfs-test-server. Not how I'm going to work around that either. Any ideas? |
Ugh, what a cumbersome piece of software to automate. How about something like this Dockerfile:
I did a quick test and that SEEMS to work, as long as lfs-test-server is built with Now you can run the image in docker, get the IP (see ssh examples) and us username "e2e" password "e2e". |
I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed. |
That's because it's not easy to make "trivial" SSH keys, whereas this case
is super easy :)
…On Mon, Dec 26, 2022 at 4:34 PM James Ringer ***@***.***> wrote:
I take it from your comment committing the credentials in git is not an
issue? I saw in sshd that the certs were in a volume that was not committed.
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVFVFQ4LEKCP6F7JCZLWPI2TDANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
That makes sense. I just wanted to check before moving forward. Btw adding the user with curl that way is just what I was looking for. I'll keep working on this. Thanks. |
Haven't circled back to this yet, but I will. Please don't let the issue go stale and close. Thanks! |
Hi. I am hoping to have a v4.0.0 release candidate some time in the next month or so - would love to have this in. No pressure, we can always add it later. |
I'll put some time toward this over the next few weeks. |
I'm working on test_e2e.sh. I added a section to init lfs-test-server.
The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.
To confirm the lfs tracked file is sent to lfs-test-server we need to |
I don't think you need to do this in the common init_repo, but in the
specific LFS test case?
…On Sun, Mar 12, 2023, 3:28 AM James Ringer ***@***.***> wrote:
I'm working on test_e2e.sh. I added a section to init lfs-test-server.
# Init Git LFS Test Server
# Clone lfs-test-server to _test_tools
git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server
# Overwrite lfs-test-server Dockerfile
tee ./_test_tools/lfs-test-server/Dockerfile <<EOF
FROM golang:1.18
WORKDIR /go/src/github.com/git-lfs/lfs-test-server
COPY . .
RUN go build
EXPOSE 8080
ENV LFS_ADMINUSER=admin
ENV LFS_ADMINPASS=admin
RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \
sleep 0.5 \
&& curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e
ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"]
EOF
# Build the lfs-test-server image
docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest
# Run the lfs-test-server container
docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest
The shell snippet works. Idk if that's what you had in mind. I'm also
looking at modifying the init_repo function.
function init_repo() {
rm -rf "$REPO"
mkdir -p "$REPO"
git -C "$REPO" init -q -b "$MAIN_BRANCH"
touch "$REPO"/file
git -C "$REPO" add file
git -C "$REPO" commit -aqm "init file"
cd "$REPO"
git lfs install
git lfs track "*.jpg"
cd ..
echo "[lfs]" > "$REPO"/.lfsconfig
echo " url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig
git -C "$REPO" add .lfsconfig .gitattributes
git -C "$REPO" commit -aqm "lfs config"
touch "$REPO"/lfs.jpg
git -C "$REPO" add lfs.jpg
git -C "$REPO" commit -aqm "add lfs.jpg"
rm -rf "$REPO2"
cp -r "$REPO" "$REPO2"
}
To confirm the lfs tracked file is sent to lfs-test-server we need to git
push, but for that we need a remote.
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVAHFJNHWUSATBMSGLDW3WQMDANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh? |
Do you have a branch pushed to GitHub? I can clone it later this week and
see what I see.
…On Mon, Mar 13, 2023, 12:00 PM James Ringer ***@***.***> wrote:
I've been looking at the repo trying to understand how to do this, but I
don't quite follow how everything works. Would it be better if I put the
script at /test-lfs.sh?
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVDC4EU4YWEQ22QIPHDW35VF3ANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Tbh, there's not much to see at this point. After working on it a bit last weekend I have a better idea of which direction to go. I'm going to work on it again this weekend a bit more, so we'll see how it goes. |
I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server. I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo. Are we on the same page with that goal? |
I think that's right - enough to prove that LFS was activated and works.
…On Sun, Mar 19, 2023 at 9:40 PM James Ringer ***@***.***> wrote:
I worked with it a bit more and made a little progress. I got a
lfs-test-server container along with a git-server-docker container setup
and was able to push a commit to git-server-docker containing a jpg that
was pushed to lfs-test-server.
I'll throw a bit more time at it next weekend. I feel it's starting to
come together, but there is still a ways to go. Ultimately, I want to
automate the setup of an environment where we have a repo in
git-server-docker and a large file in lfs-test-server such that when we
clone the repo with git-sync it pulls the large file with the repo.
Are we on the same page with that goal?
—
Reply to this email directly, view it on GitHub
<#654 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVEEDNZECMWQG2CD7UTW47NTLANCNFSM6AAAAAATHFYSJ4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Circling around to open topics - any progress here? |
I have not had time the past couple of weeks. Tomorrow I'll have some time to pick this up again. Idk how far I'll get, but I'll keep at it until it's done. |
I talked with my manager and received approval to start dedicating some of my work time to OSS. This is at the top of my list! |
That's awesome! Congrats. |
I updated my fork. It's not working yet and it's rough, but if you want to take a look. |
@thockin I have not forgotten about this, but I'm on a modified schedule until the end of June with no time for oss. I'll circle back to this in July. Sorry I'm slow and inexperienced. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Bad bot. |
I can re-open if you still want to tackle this, but the bot just sweeps up debris which is often left behind by well-intentioned people who just can't find/make the time to finish (IOW, me, frequently). |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
@thockin thanks Tim. I'll get to this someday. I haven't forgotten! |
I'm requesting git-lfs be added to the image. This issue came up back in 2019 #174. The issue was closed without adding git-lfs to the image. I tested v3.6.2 by adding git-lfs to the image and I'm syncing LFS files from BitBucket without any additional configuration.
This is my dockerfile.
This is my container spec within my deployment in k8s.
The text was updated successfully, but these errors were encountered: