Recent issue with wget2, json and the future of the scripts.

[Samueru-sama]

Because json formatting doesn't care about whitespaces, we need to be careful when using grep and sed on it.

That means that all wget need to be piped first into sed 's/"/ /g; s/ /\n/g' which will get rid of the quotations and turn every space into a new line. Then grep can be run to get the desired link

We can't just rely on wget always, because even the websites can change the formatting without changing the actual data and break stuff.

This is what I'm using now to get go-appimage/appimagetool in my scripts:

wget -q https://api.github.com/repos/probonopd/go-appimage/releases -O - | sed 's/"/ /g; s/ /\n/g' | grep -o 'https.*continuous.*tool.*86_64.*mage$'

It works with both wget and wget2.

The more ideal solution would be using jq. But that introduces a new dependency.

EDIT: This other combination is useful for finding the URL, as it get rids of the parenthesis, curly brackets, double quotes and commas before turning each space into one line.

sed 's/[()",{}]/ /g; s/ /\n/g' | grep https

[Samueru-sama]

Also @ivan-hc just in case you haven't notice, the issue was not wget2 itself, it was that wget2 used a different user agent that wget and that causes github to give the data in a different format, it is still the same data since it is json formatted but since awk, grep, sed, etc do take whitespaces into account it broke them.

So yeah, any day if github has an update all 2000 scripts can suddenly stop working.

[ivan-hc]

I've done some tests, I had to remove/add patches in install.am

function _install_common_patch(){
						if ! grep -q -- "--debug" $AMPATH/.cache/install-args; then
							# Patch "wget" to have a progress bar and fix errors when applying patches
							sed -i 's#$(wget -q#$(curl -Ls#g' ./$arg
							sed -i 's#-O -##g' ./$arg
							#sed -i "s#wget #wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #g" ./$arg
							#sed -i 's#$(wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #$(wget -q #g' ./$arg
							sed -i "s# -q -q # -q #g" ./$arg
							# Silence "rm" errors by adding "-f"
							sed -i 's#^rm /#rm -f /#g' ./$arg; sed -i 's#^rm ./#rm -f ./#g' ./$arg
							# Silence "mv", "sed" errors and "squashfs-root" messages
							sed -i 's/ \&& mv/ | grep -v "squashfs-root" | grep -v ":" | grep -v "help"; mv/g' ./$arg
							sed -i '/appimage-extract/ s/$/ 2>\&1 | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/^mv / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.png) #./$APP --appimage-extract \*.png 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.svg) #./$APP --appimage-extract \*.svg 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i '/^sed / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/squashfs-root/ s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
						fi
						sed -i "s# https://api.github.com#$HeaderAuthWithGITPAT https://api.github.com#g" ./$arg
					}

the only issue is to hide the big wget output

simplescreenrecorder-2024-04-28_14.02.20.mkv.mp4

[ivan-hc]

@Samueru-sama as you can see, in the patch was enough to add this

sed -i 's#$(wget -q#$(curl -Ls#g' ./$arg
sed -i 's#-O -##g' ./$arg

while all wget patches were been removed.

This is how a simple wget $version look alike with wget2

[ivan-hc]

note, I've tested this also on apps not hosted on github

[ivan-hc]

@Samueru-sama your proposal to add a further patch to the existing scripts is wasteful, as we would end up inflating already very complex commands regardless.

The least common divisor here are "wget -q" and "-O -", the first one needs to be replaced, the other needs to be deleted. And we solved it.

[ivan-hc]

I just realized that I will have to make this change to all my scripts in all my repositories... HOLY SHIT! THOSE CURSED BASTARDS!

EDIT: just finished to change to update all my scripts using your patch

[Samueru-sama]

@ivan-hc Using curl instead of wget is NOT A FIX. The site still gives you json formatted data. The issue is with parsing json. Once again the ideal solution would be start to use jq.

However the sed 's/[()",{}]/ /g; s/ /\n/g' I shared is useful to get links from json formatted data, as doesn't matter in which order it gets printed because it splits all links into individual lines. But anything more complex will need jq.

Both of those are json, both contain the same information, a json parser will have no issue getting the right info you want from any. however using grep, sed, awk, etc will have issues depending on how you use them.

[ivan-hc]

today I added a patch to the install.md module, now merged in main and I've tested this with many installation scripts

however, as we have seen these two days, the main issue is to keep all these changes permanent

in this moment I don't know where to start

[Samueru-sama]

in this moment I don't know where to start

@ivan-hc Any new script has to start this way for wget, curl, etc:

wget/curl + sed 's/[()",{}]/ /g; s/ /\n/g' + ideally just one grep with wildcards to get the url.

sed 's/[()",{}]/ /g; s/ /\n/g' | grep https spits out all the https links on the terminal, so it is also easier to work with this way as well.

Once again the issue wasn't wget2 itself, it was a different user agent causing github to spit the data in non human readable json, while for wget github does spit the data in human readable json.

Github and other websites can start giving the json data in a different format any time, so even though the maintainer of wget2 changed the user agent to wget, I wouldn't rest easily hoping that webistes never change how they format their json data.

[ivan-hc]

all scripts need to be reviewed, especially those that search for items from api.github.com.

if you think your function is ideal for bypassing the use of jq, so be it

[ivan-hc]

Today I've also tested the differences between wget and wget2 in terms of options and the ones we use now are not compatible at all.

					function _install_common_patch(){
						if ! grep -q -- "--debug" $AMPATH/.cache/install-args; then
							# Use curl instead of wget to manage github APIs
							sed -i 's#$(wget -q#$(curl -Ls#g' ./$arg
							sed -i 's#-O -##g' ./$arg
							# Patch "wget" to have a progress bar and fix errors when applying patches
							sed -i "s#wget #wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #g" ./$arg
							# Silence "rm" errors by adding "-f"
							sed -i 's#^rm /#rm -f /#g' ./$arg; sed -i 's#^rm ./#rm -f ./#g' ./$arg
							# Silence "mv", "sed" errors and "squashfs-root" messages
							sed -i 's/ \&& mv/ | grep -v "squashfs-root" | grep -v ":" | grep -v "help"; mv/g' ./$arg
							sed -i '/appimage-extract/ s/$/ 2>\&1 | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/^mv / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.png) #./$APP --appimage-extract \*.png 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.svg) #./$APP --appimage-extract \*.svg 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i '/^sed / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/squashfs-root/ s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
						fi
						sed -i "s# https://api.github.com#$HeaderAuthWithGITPAT https://api.github.com#g" ./$arg
					}

this is what I've changed for now

# Use curl instead of wget to manage github APIs
sed -i 's#$(wget -q#$(curl -Ls#g' ./$arg
sed -i 's#-O -##g' ./$arg
# Patch "wget" to have a progress bar and fix errors when applying patches
sed -i "s#wget #wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #g" ./$arg

As long as api.github.com maintains compatibility with "curl" at the user agent level (since it recognizes "curl" and "wget 1.x" calls, but not yet "wget2", which is not yet an established CLI) we can rest assured.

As for the installation interface available since version 6.6.2 (and for which I fought so hard), I will certainly have to adapt to that of "wget2", which is not different from:

wget -q --no-verbose --show-progress --progress=bar

[Samueru-sama]

Btw @ivan-hc have you done anything with $CACHEDIR? just asking because I will begin to take a look at that again. Will likely rebase it again from dev.

[ivan-hc]

well, if you tell me with that "face" (imagining that that is your real face right now), you don't seem so convinced

[Samueru-sama]

well, if you tell me with that "face" (imagining that that is your real face right now), you don't seem so convinced

Haha, No no! I'm actually happy for real.

I tried to use jq to get the go-appimage link example alone and that was so painful that I gave up lol.

This isn't the first time I've dealt with jq, I also had my nightmares with amdgpu_top that uses json for its formatting. jq is also used with i3wm.

[ivan-hc]

is it that bad to use jq?

[ivan-hc]

tell me, what kind of issues you had with them?

[Samueru-sama]

is it that bad to use jq?

No, but I gave you a longer reply here so you know what you are getting yourself into: #524 (comment)

[Samueru-sama]

@ivan-hc I will reply here for better viewing:

is it that bad to use jq?

For example this is what I use with amdgpu_top to get the GPU USAGE and VRAM USAGE. Just two values + it calculates the results in GiB.

amdgpu_top -J -n 1 | jq -c -r '(.devices[] | ((.gpu_activity.GFX.value|tostring) + "% " + ((.VRAM."Total VRAM Usage".value/1024*10|round/10|tostring) + "/" + (.VRAM."Total VRAM".value/1024*10|round/10|tostring)) + "GB"))'

Something that if the data was given as plain text would be a very simple awk command. For example here is the equivalent with nvidia-smi back when I was using nvidia:

nvidia-smi | awk 'FNR == 10 {printf "%s %0.1f/%0.0fGB\n", $13, $9/1024, $11/1024; exit}'

Of course jq doesn't have to be used for everything, But I'm not sure at which point I can stop using jq and switch back to the coreutils to do the rest of operation.

tell me, what kind of issues you had with them?

Here is the jq equivalent to get the appimagetool link:

wget -q https://api.github.com/repos/probonopd/go-appimage/releases -O - | jq '.[] | select (.tag_name == "continuous") | .assets[] | select(.name | test("appimagetool") and test("-x86_64") and test("AppImage$", "is")) | .browser_download_url'

I tried for 3 hours to get it to work, in the end I had to ask for help and someone I know that knows a lot of js did it for me.

Honestly I really hope my sed trick never breaks so that I don't have to deal with jq or even gron in the future.

[Samueru-sama]

@ivan-hc Alright I got back, yes what you are doing is still a hack however you just have to change the jq for this!

jq -r '.[].assets[].browser_download_url'

And it should give you all the urls that you can then grep, sed, etc

Phew!

The end result is this:

wget -q https://api.github.com/repos/probonopd/go-appimage/releases -O - | jq -r '.[].assets[].browser_download_url' | grep -v zsync | grep -i continuous | grep -i appimagetool | grep -i x86_64    

You have to remove the grep browser_download_url | cut -d '"' -f 4 from the old scripts basically.

[ivan-hc]

this is why its called a patch lol

[ivan-hc]

if the script contains api.github.io / if the wget version is 1.x

					# Patch to apply to installation scripts
					function _install_common_patch(){
						if ! grep -q -- "--debug" $AMPATH/.cache/install-args; then
							# Add "jq" to manage github APIs
							if grep -q "api.github.com" ./$arg; then
								sed -i "s#-O -#-O - | jq '.'#g" ./$arg
							fi
							# Patch "wget" (version 1.x) to have a progress bar and fix errors when applying patches
							if wget --version | head -1 | grep -q ' 1.'; then
								sed -i "s#wget #wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #g" ./$arg
								sed -i 's#$(wget -q --no-verbose --show-progress --progress=dot -e dotbytes=500K #$(wget -q #g' ./$arg
								sed -i "s# -q -q # -q #g" ./$arg
							fi
							# Silence "rm" errors by adding "-f"
							sed -i 's#^rm /#rm -f /#g' ./$arg; sed -i 's#^rm ./#rm -f ./#g' ./$arg
							# Silence "mv", "sed" errors and "squashfs-root" messages
							sed -i 's/ \&& mv/ | grep -v "squashfs-root" | grep -v ":" | grep -v "help"; mv/g' ./$arg
							sed -i '/appimage-extract/ s/$/ 2>\&1 | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/^mv / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.png) #./$APP --appimage-extract \*.png 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i 's#mv $(./$APP --appimage-extract \*.svg) #./$APP --appimage-extract \*.svg 2>\&1 | grep -v "squashfs-root"; mv ./squashfs-root/*$app* #g' ./$arg
							sed -i '/^sed / s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
							sed -i '/squashfs-root/ s/$/ 2>\/dev\/null | grep -v "squashfs-root" | grep -v ":" | grep -v "help"/' ./$arg
						fi
						sed -i "s# https://api.github.com#$HeaderAuthWithGITPAT https://api.github.com#g" ./$arg
					}

[ivan-hc]

I'm ready for the new release

[ivan-hc]

@Samueru-sama why that face?

[ivan-hc]

@Samueru-sama @nazdridoy what about this?

In APP-MANAGER

# DOWNLOAD PORTABLE DEPENDENCES
function _portable_dependences_setup() {
	mkdir -p "$AMPATH"/bin

	function _jq_installer() {
		if command -v jq 1>/dev/null; then
			amjq="jq"
		else
			amjq="$AMPATH/bin/jq"
			jqversion=$(curl -Ls https://api.github.com/repos/jqlang/jq/releases | sed 's/[()",{}]/ /g; s/ /\n/g' | grep -o "https.*linux.*$jqarch" | head -1)
			if ! test -f "$AMPATH"/bin/jq; then
				if [ "$arch" == x86_64 ]; then
					jqarch="amd64"
				else
					jqarch="$arch"
				fi
				echo -ne "Downloading jq to $AMPATH/bin...\r"
				wget -q "$jqversion" -O "$AMPATH"/bin/jq && echo "$jqversion" >> "$AMPATH"/bin/jq-version && chmod a+x "$AMPATH"/bin/jq
			else
				jqversion0=$(cat "$AMPATH/bin/jq-version")
				if [ "$jqversion" != "$jqversion0" ]; then
					rm -f "$AMPATH"/bin/jq "$AMPATH/bin/jq-version"
					echo -ne "Updating $AMPATH/bin/jq...\r"
					wget -q "$jqversion" -O "$AMPATH"/bin/jq && echo "$jqversion" >> "$AMPATH"/bin/jq-version && chmod a+x "$AMPATH"/bin/jq
				fi
			fi
		fi
	}
	_jq_installer
}

the function is called when running the option -i

Maybe I can include it in install.am directly, I've changed it like this

sed -i "s#-O -#-O - | $amjq '.'#g" ./$arg

[nazdridoy]

how about you make a PR then we can code review.. if anything need to changes we can add commentes.. please don't push major update in rush. take your time. let other test ant and then push

[ivan-hc]

OK

[nazdridoy]

i'm not talking about jq. as jq is prerequisite to parse github api response. i'm suggesting you not to host any binary in the repo rather pull from upstream, be it via install script or hard coded

[ivan-hc]

sure, the wget binary was only for testing purpouses... its als not listed in the apps list for this

[zen0bit]

everything is ready on my PC, I only want to know your opinion @nazdridoy @Samueru-sama @zen0bit

What? Why?

[nazdridoy]

how about you make a PR then we can code review.. if anything need to changes we can add commentes.. please don't push major update in rush. take your time. let other test ant and then push

@ivan-hc make a general beta testing schedule for pushing update. (for example 1 week for major update, 3 days for minoj version, 1 day for minor fix, and hot fix as is ). this will help other contributor (even if they are busy they can certainly spare a few hours in a week) to code review and find bugs before breaking updates.

[ivan-hc]

My changes are ready to be tested:

https://github.com/ivan-hc/AM/tree/dev

Commits main...dev

[ivan-hc]

@nazdridoy sorry for my ignorance, but... for beta testing... should I submit the PR and tag people?

[nazdridoy]

yes

[nazdridoy]

@ivan-hc
https://www.youtube.com/watch?v=lSnbOtw4izI

[nazdridoy]

@ivan-hc make it necessary that every PR need to be have at least 1 review (by you or any other contributor) before it can be merged.

https://www.youtube.com/watch?v=Igx8w95cakU