Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update crc_squeue #231

Closed
wants to merge 52 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
f15d626
Start on building a usage table with info from keystone instead of cr…
Comeani May 3, 2024
904b71c
utility functions for interacting with keystone
Comeani May 3, 2024
9a0d8ff
pull function for getting allocations by primary key
Comeani May 3, 2024
4d6afa5
add requests
Comeani May 6, 2024
5469d59
verify provided slurm account against keystone as well as slurm
Comeani May 6, 2024
3589ba4
Merge branch 'keystone_update' of https://github.com/pitt-crc/wrapper…
Comeani May 6, 2024
6b1ca32
properly handle dates and pretty table rows
Comeani May 6, 2024
96a1c72
split output into summary and usage tables, keep track of per cluster…
Comeani May 6, 2024
db6de4d
PEP fixes and comments
Comeani May 6, 2024
dc9c260
Update crc_sus.py
chnixi May 6, 2024
d4ea369
adjustments from meeting
Comeani May 6, 2024
e04a982
Merge branch 'keystone_update' of https://github.com/pitt-crc/wrapper…
Comeani May 6, 2024
fb14de0
filter on active and expiration date in request for allocations
Comeani May 6, 2024
1ee9341
add and move slurm functions for gathering usage data into utils.syst…
Comeani May 6, 2024
c9eac94
move slurm functions out of utils.keystone
Comeani May 6, 2024
23db644
add usage table setup
Comeani May 6, 2024
546220b
small fix with field_names
Comeani May 6, 2024
68af066
fix imports, use account_name in checking for whether slurm account e…
Comeani May 6, 2024
e0e154a
fix adding total to out_data
Comeani May 6, 2024
0d7b701
clean up usage output formatting
Comeani May 6, 2024
4968723
Update crc_squeue
chnixi May 7, 2024
faa405e
small formatting changes to crc_sus
Comeani May 7, 2024
08d4018
formatting adjustment
Comeani May 7, 2024
2a85a64
Update crc_squeue.py
chnixi May 7, 2024
84c59f0
add request primary key as query param for requests
Comeani May 7, 2024
4d28a85
use cluster totals instead of allocation totals, other review items
Comeani May 7, 2024
cf49a73
use request pk when gathering allocations
Comeani May 7, 2024
85a5326
Merge branch 'keystone_update' of https://github.com/pitt-crc/wrapper…
Comeani May 7, 2024
1ee2af8
use check_slurm_account_exists
Comeani May 7, 2024
6172572
working state for crc-sus and crc-usage
Comeani May 7, 2024
626cb8d
fix crc_usage test that wa failing
Comeani May 7, 2024
89c6b39
Pull comment regarding format of request json
Comeani May 7, 2024
4df347a
Update crc_squeue.py
chnixi May 7, 2024
532ed77
pull duplicate functionality for running subprocesses
Comeani May 7, 2024
1bbcfc1
small typo in crc_usage test string
Comeani May 7, 2024
a88ae28
Update crc_squeue.py
chnixi May 7, 2024
406ed7a
Update crc_squeue.py
chnixi May 7, 2024
9d0103f
crc-proposal-end (#232)
chnixi May 7, 2024
268f071
Merge branch 'keystone_update' into crc-squeue
Comeani May 7, 2024
6176557
Merge branch 'keystone_update' into crc-squeue
Comeani May 7, 2024
9eef5cb
Update crc_squeue.py
chnixi May 17, 2024
347e17c
Merge branch 'crc-squeue' of https://github.com/pitt-crc/wrappers int…
Comeani May 17, 2024
997d718
fix missing imports and syntax errors
Comeani May 17, 2024
1fefffb
Merge branch 'main' into crc-squeue
Comeani May 17, 2024
4bc90f5
Update crc_squeue.py
chnixi May 22, 2024
d91e95f
Update crc_squeue.py
chnixi May 22, 2024
8bf2e15
fix missing imports
Comeani May 22, 2024
8418769
Remove step to create banking DB location
Comeani May 22, 2024
8ac1933
Attempt to fix crc_squeue test so it works in CI
Comeani May 22, 2024
88057ca
use getuser instead of os.environ
Comeani May 22, 2024
bdbf80c
Merge branch 'crc-squeue' of https://github.com/pitt-crc/wrappers int…
Comeani May 22, 2024
919fecd
Merge branch 'main' into crc-squeue
Comeani Jun 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/PackageTest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ jobs:
- name: Setup environment
run: /usr/local/bin/entrypoint.sh

- name: Set Up Bank DB Location
run: mkdir -p /ihome/crc/bank

- name: Checkout repository
uses: actions/checkout@v4

Expand Down
35 changes: 32 additions & 3 deletions apps/crc_squeue.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
"""A simple wrapper around the Slurm ``squeue`` command."""

import getpass
import os
import grp
from getpass import getpass, getuser
from argparse import Namespace
from datetime import datetime, date

Check warning on line 7 in apps/crc_squeue.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

apps/crc_squeue.py#L7

Unused datetime imported from datetime
from time import sleep

from .utils.keystone import *
from .utils.system_info import Slurm, Shell
from .utils.cli import BaseParser
from .utils.system_info import Shell


class CrcSqueue(BaseParser):
Expand Down Expand Up @@ -40,7 +44,7 @@
command_options.append(cls.output_format_all)

else:
user = f'-u {getpass.getuser()}'
user = f'-u {getuser()}'
command_options.append(user)
command_options.append(cls.output_format_user)

Expand All @@ -53,6 +57,31 @@
args: Parsed command line arguments
"""

default_group = grp.getgrgid(os.getgid()).gr_name
Slurm.check_slurm_account_exists(account_name=default_group)

auth_header = get_auth_header(KEYSTONE_URL,
{'username': getuser(),
'password': getpass("Please enter your CRC login password:\n")})

keystone_group_id = get_researchgroup_id(KEYSTONE_URL, default_group, auth_header)

if not keystone_group_id:
print(f"No allocation data found in accounting system for '{default_group}'")
exit()

requests = get_active_requests(KEYSTONE_URL, keystone_group_id, auth_header)

if not requests:
print(f"No active resource allocation requests found in accounting system for '{default_group}'")
exit()

for request in requests:
# Check if proposal will expire within 30 days. If yes, print a message to inform the user
if (date.fromisoformat(request['expire']) - date.today()).days < 30:
print(f"The active proposal for account {default_group} will expire soon on {request['expire']}."
"Please begin working on a new Resource Allocation Request if you want to run jobs beyond that date.")

Check notice on line 83 in apps/crc_squeue.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

apps/crc_squeue.py#L83

Line too long (124/120)

command = self.build_slurm_command(args)
if args.print_command:
print(command)
Expand Down
2 changes: 1 addition & 1 deletion tests/test_crc_squeue.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def test_user_option(self) -> None:
"""Test the ``--all`` argument toggles the slurm ``-u`` option in the returned command"""

app = CrcSqueue()
slurm_user_argument = f'-u {getpass.getuser()}'
slurm_user_argument = "-u"

# The application should default to showing information for the current user
args, _ = app.parse_known_args([])
Expand Down
Loading