668 joblogs

[vlerkin]

Added logic to remove files:

1. After file was successfully uploaded to the S3-like-storage the file in the volume will be removed;
2. When code receives events from other existing jobs on the cluster it checks if files with its logs are present in the S3-like-storage, if not the code will attempt to collect logs, if the stdout is still available, if the file exists the code now checks if that file also still present in the volume and removes it since it is present in the S3-like-storage.

I think this functionality addresses your concerns.

[wvengen]

Thanks! Can you please rebase this on the main branch?

Regarding your second point, I don't fully understand this. Does it mean that if a job is found running or finished, and also a log on container storage, it will do nothing? When finished, I think this makes sense, but when running, this is actually an error condition, I would think. Or can you think of cases when a file uploaded to container storage exists, and a job is still found running?

(Also, I was thinking that this will generate a lot of checking with container storage to see if files exists. That is fine. A later improvement could be to add metadata to the job indicating that its logs have been stored (potentially with the full container storage path, so it can be easily located e.g. by #12). But that's for later.)

[vlerkin]

The second point I outlined is about Completed or Failed jobs that still exist on the cluster, if they actively run then their logs are still being actively collected and when finished, uploaded. But if for some reason the managing pod died, and its replacement in the process of creation, then jobs will finish running without it, and all the logs they produce need to be collected by a new managing pod. It will append logs from the moment it was disrupted, upload the file and delete it from an associated volume, or if there are complete logs, jobs are not deleted and the uploading process was interrupted by unexpected pod death, then a new pod, when receives an event that there are still jobs on a cluster in Completed/Failed state, checks if a corresponding log file exists in the S3-like-storage, if not checks if it's in the volume, if yes, uploads it to the S3-like-storage and deletes it from volume.

Code lines: 257-301 (handle_events method) in the log_handler_k8s.py file.