Possible issue with SFTP + AWS and postData.sh + check_allsky.sh

[astrobrothers]

I started seeing two issues pop up about 2 months ago (running Version: v2023.05.01_03 at the time). Initially what I think happened is I was saving too many images on local allsky (RPi) and ran out of disk space. Two things started happening after that:

1. Modules were "corrupted" and overlays stopped appearing on remote (website on AWS).
2. started getting the postData.sh X days old error for the first time

After resetting # of days to keep to far less and restoring the modules to last "working" version, making an edit (clicking any of the check boxes on then off) and finally saving - it fixed #1.

The postData.sh continued to have issues. Images/startrails etc were uploading fine. It was just the .json files as far as I can tell from the logs. So, today I upgraded to Version: v2023.05.01_05. With the new check_allsky.sh I got this:
PROTOCOL (sftp) set but not 'REMOTE_PASSWORD'. Uploads will not work.

Note - we use a .pem file, not a password to connect to AWS and this has worked just fine - so, I think this error is incorrect? That said, when I ran postData.sh after the upgrade, I get this (same as v_03):
Upload output from '/home/pi/allsky/config/settings.json:
rm of settings.json failed! Trying again...
rm: Access failed: No such file (settings.json)
The difference after upgrade of allsky (v_05) is that postData.sh did make the postData.sh XXX days old message go away, where as it did not in v_03. So, we'll see if this is works automatically going forward... It could also be that the allsky local files became corrupted when I ran out of disk space and the upgrade simply fixed that by installing uncorrupted files??

Looking at the lftp log, there still seems to be an issue related to file removal? In case this helps:

set sftp:connect-program 'ssh -i /home/pi/Documents/.pem'
set dns:fatal-timeout 10
set net:max-retries 2
set net:timeout 10
open --user '' --password '' 'sftp://********'
cd . || exit 99
quote PWD | grep current
ls
debug 5
cd '../bitnami/htdocs/allsky/' || exit 1
echo 'In DIRECTORY=../bitnami/htdocs/allsky/:'
ls
glob --exist 'PostData-3964*' && echo 'PostData-3964 exists; removing...' && rm 'PostData-3964'
put '/home/pi/allsky/tmp/data.json' -o 'PostData-3964'
|| (echo 'put of /home/pi/allsky/tmp/data.json to PostData-3964 failed! Trying again...'; (!sleep 3); put '/home/pi/allsky/tmp/data.json' -o 'PostData-3964' && echo 'WORKED' && exit 0)
|| (echo '2nd put failed again, quitting'; exit 1)
|| exit 2
rm -f 'data.json'
glob --exist 'data.json*'
&& (echo 'rm of data.json failed! Trying again...'; (!sleep 3); rm 'data.json' && echo 'WORKED' && exit 1)
&& (echo '2nd rm failed, quiting.'; rm -f 'PostData-3964'; exit 1)
&& exit 3
mv 'PostData-3964' 'data.json'
|| (echo 'mv of PostData-3964 to data.json failed! Trying again...'; (!sleep 3); mv 'PostData-3964' 'data.json' && echo 'WORKED' && exit 0)
|| (echo '2nd mv failed, quitting.'; rm -f 'PostData-3964'; exit 1)
|| exit 4
exit 0

[EricClaeys]

@astrobrothers,

Modules were "corrupted" and overlays stopped appearing on remote (website on AWS).

Do you mean the output of the modules, or the actual module source files, e.g., *.py files?
Running out of disk space can really hose up almost everything, but it shouldn't change any of the Allsky source files since nothing writes to them, other than an Allsky release upgrade. Allsky data files, including *.json files, could get corrupted since they are, by nature, written to often.

Please double check your PROTOCOL in ftp-settings.sh. If you are using AWS the PROTOCOL should be S3, which doesn't use the REMOTE_PASSWORD, as you noted.

What ftp log are you referring to? There are several. The one you show just contains the commands passed to the lftp command, and not the output of running lftp. The commands run include many checks for failure. In some cases, if something fails it tries a second time, and if that also failed, it displays an error message and quits.

[astrobrothers]

Since the last post, it hasn't happened again (either the old data.json or the "corrupted" modules). I think for one or both of these it was just that disk space was zero and it couldn't write the new .json files or update the modules. Once I cleared out the image folders, it seems to be good again. I saw a few other users post that their modules got this error when they ran out of disk space as well. Just something to keep in mind I guess when setting up the # of days to keep images.

To prevent this in the future I changed the retention days from 30 to 14. I'm going to write a script to FTP the Pi's image folders to an NAS so I don't have to remember to do this manually once a week.

If it does happen again, I'll post the other ftp logs.. Thanks!