Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Add node setup&teardown #16

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

WIP: Add node setup&teardown #16

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fe2b70d
merged
Aug 4, 2022
3ec9656
Merge branch 'master' of github.com:petuum/pirlib into setup-teardown
Aug 5, 2022
fda1a6a
WIP: added setup & teardown method
Aug 5, 2022
1360b21
fixed lint
Aug 5, 2022
321411e
hid handler context from users
Aug 8, 2022
1551517
fixed lint
Aug 8, 2022
d591722
removed unused import and metric report
Aug 8, 2022
6395eab
changed pipeline decorator mechanism and cleaned up context methods
Aug 9, 2022
1601280
fixed lint
Aug 9, 2022
8947ef9
added context log
Aug 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 16 additions & 18 deletions example/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,10 @@ services:
condition: service_completed_successfully
train_pipeline.train:
condition: service_completed_successfully
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
- ${INPUT_train_dataset:?err}:/mnt/graph_inputs/train_dataset
- ${INPUT_translate_model:?err}:/mnt/graph_inputs/translate_model
- ${INPUT_sentences:?err}:/mnt/graph_inputs/sentences
- ${OUTPUT:?err}:/mnt/graph_outputs
train_pipeline.clean:
Expand All @@ -30,9 +29,9 @@ services:
- -m
- pirlib.backends.docker_batch
- node
- gASVWwIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMBWNsZWFulIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBVleGFtcGxlLmV4YW1wbGU6Y2xlYW6UjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6MTJhYmVmZmYtODlkNy00ODNhLTliMjMtNzgxNTNkZDUxMGRjlHVic4wJZnJhbWV3b3JrlE6MBmNvbmZpZ5R9lIwGaW5wdXRzlF2UaACMBUlucHV0lJOUKYGUfZQoaAWMB2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjAZzb3VyY2WUaACMCkRhdGFTb3VyY2WUk5QpgZR9lCiMB25vZGVfaWSUTowLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklE6MDmdyYXBoX2lucHV0X2lklIwNdHJhaW5fZGF0YXNldJR1YowEbWV0YZRoAIwITWV0YWRhdGGUk5QpgZR9lCiMBG5hbWWUjAdkYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJhjAdvdXRwdXRzlF2UaACMBk91dHB1dJSTlCmBlH2UKGgFjAZyZXR1cm6UaCGMCURJUkVDVE9SWZRoLWgvKYGUfZQoaDKMBnJldHVybpRoNE51YnViYWgtaC8pgZR9lChoMowFY2xlYW6UaDROdWJ1Yi4=
- gASVJAEAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMD3RyYW5zbGF0ZV9tb2RlbJRoCIwERklMRZRoCmgMKYGUfZQoaA+MD3RyYW5zbGF0ZV9tb2RlbJRoEU51YnViaAMpgZR9lChoBowJc2VudGVuY2VzlGgIjAlESVJFQ1RPUlmUaApoDCmBlH2UKGgPjAlzZW50ZW5jZXOUaBFOdWJ1YmUu
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
- gASVWwIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMBWNsZWFulIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBVleGFtcGxlLmV4YW1wbGU6Y2xlYW6UjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6YjhmN2ViYjQtMmI0OS00Zjg3LThjMWEtYWZkYjg1Yjk1NmFjlHVic4wJZnJhbWV3b3JrlE6MBmNvbmZpZ5R9lIwGaW5wdXRzlF2UaACMBUlucHV0lJOUKYGUfZQoaAWMB2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjAZzb3VyY2WUaACMCkRhdGFTb3VyY2WUk5QpgZR9lCiMB25vZGVfaWSUTowLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklE6MDmdyYXBoX2lucHV0X2lklIwNdHJhaW5fZGF0YXNldJR1YowEbWV0YZRoAIwITWV0YWRhdGGUk5QpgZR9lCiMBG5hbWWUjAdkYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJhjAdvdXRwdXRzlF2UaACMBk91dHB1dJSTlCmBlH2UKGgFjAZyZXR1cm6UaCGMCURJUkVDVE9SWZRoLWgvKYGUfZQoaDKMBnJldHVybpRoNE51YnViYWgtaC8pgZR9lChoMowFY2xlYW6UaDROdWJ1Yi4=
- gASV2gAAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMCXNlbnRlbmNlc5RoCIwJRElSRUNUT1JZlGgKaAwpgZR9lChoD4wJc2VudGVuY2VzlGgRTnVidWJlLg==
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
- ${INPUT_train_dataset:?err}:/mnt/graph_inputs/train_dataset
Expand All @@ -42,12 +41,12 @@ services:
- -m
- pirlib.backends.docker_batch
- node
- gASVCAMAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMCGV2YWx1YXRllIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBhleGFtcGxlLmV4YW1wbGU6ZXZhbHVhdGWUjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6MTJhYmVmZmYtODlkNy00ODNhLTliMjMtNzgxNTNkZDUxMGRjlHVic4wJZnJhbWV3b3JrlE6MBmNvbmZpZ5R9lIwGaW5wdXRzlF2UKGgAjAVJbnB1dJSTlCmBlH2UKGgFjBNrd2FyZ3MudGVzdF9kYXRhc2V0lIwGaW90eXBllIwJRElSRUNUT1JZlIwGc291cmNllGgAjApEYXRhU291cmNllJOUKYGUfZQojAdub2RlX2lklE6MC3N1YmdyYXBoX2lklE6MCW91dHB1dF9pZJROjA5ncmFwaF9pbnB1dF9pZJSMCXNlbnRlbmNlc5R1YowEbWV0YZRoAIwITWV0YWRhdGGUk5QpgZR9lCiMBG5hbWWUjBNrd2FyZ3MudGVzdF9kYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJoHSmBlH2UKGgFjBJrd2FyZ3MucHJlZGljdGlvbnOUaCGMCURJUkVDVE9SWZRoI2glKYGUfZQoaCiMGGluZmVyX3BpcGVsaW5lLnNlbnRpbWVudJRoKU5oKowGcmV0dXJulGgrTnViaC1oLymBlH2UKGgyjBJrd2FyZ3MucHJlZGljdGlvbnOUaDROdWJ1YmWMB291dHB1dHOUXZRoAIwGT3V0cHV0lJOUKYGUfZQoaAWMBnJldHVybpRoIYwJREFUQUZSQU1FlGgtaC8pgZR9lChoMowGcmV0dXJulGg0TnVidWJhaC1oLymBlH2UKGgyjAhldmFsdWF0ZZRoNE51YnViLg==
- gASVJAEAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMD3RyYW5zbGF0ZV9tb2RlbJRoCIwERklMRZRoCmgMKYGUfZQoaA+MD3RyYW5zbGF0ZV9tb2RlbJRoEU51YnViaAMpgZR9lChoBowJc2VudGVuY2VzlGgIjAlESVJFQ1RPUlmUaApoDCmBlH2UKGgPjAlzZW50ZW5jZXOUaBFOdWJ1YmUu
- gASVCAMAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMCGV2YWx1YXRllIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBhleGFtcGxlLmV4YW1wbGU6ZXZhbHVhdGWUjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6YjhmN2ViYjQtMmI0OS00Zjg3LThjMWEtYWZkYjg1Yjk1NmFjlHVic4wJZnJhbWV3b3JrlE6MBmNvbmZpZ5R9lIwGaW5wdXRzlF2UKGgAjAVJbnB1dJSTlCmBlH2UKGgFjBNrd2FyZ3MudGVzdF9kYXRhc2V0lIwGaW90eXBllIwJRElSRUNUT1JZlIwGc291cmNllGgAjApEYXRhU291cmNllJOUKYGUfZQojAdub2RlX2lklE6MC3N1YmdyYXBoX2lklE6MCW91dHB1dF9pZJROjA5ncmFwaF9pbnB1dF9pZJSMCXNlbnRlbmNlc5R1YowEbWV0YZRoAIwITWV0YWRhdGGUk5QpgZR9lCiMBG5hbWWUjBNrd2FyZ3MudGVzdF9kYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJoHSmBlH2UKGgFjBJrd2FyZ3MucHJlZGljdGlvbnOUaCGMCURJUkVDVE9SWZRoI2glKYGUfZQoaCiMGGluZmVyX3BpcGVsaW5lLnNlbnRpbWVudJRoKU5oKowGcmV0dXJulGgrTnViaC1oLymBlH2UKGgyjBJrd2FyZ3MucHJlZGljdGlvbnOUaDROdWJ1YmWMB291dHB1dHOUXZRoAIwGT3V0cHV0lJOUKYGUfZQoaAWMBnJldHVybpRoIYwJREFUQUZSQU1FlGgtaC8pgZR9lChoMowGcmV0dXJulGg0TnVidWJhaC1oLymBlH2UKGgyjAhldmFsdWF0ZZRoNE51YnViLg==
- gASV2gAAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMCXNlbnRlbmNlc5RoCIwJRElSRUNUT1JZlGgKaAwpgZR9lChoD4wJc2VudGVuY2VzlGgRTnVidWJlLg==
depends_on:
train_pipeline.infer_pipeline.sentiment:
condition: service_completed_successfully
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
- ${INPUT_sentences:?err}:/mnt/graph_inputs/sentences
Expand All @@ -57,14 +56,14 @@ services:
- -m
- pirlib.backends.docker_batch
- node
- gASV7QIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMGGluZmVyX3BpcGVsaW5lLnNlbnRpbWVudJSMC2VudHJ5cG9pbnRzlH2UjARtYWlulGgAjApFbnRyeXBvaW50lJOUKYGUfZQojAd2ZXJzaW9ulIwCdjGUjAdoYW5kbGVylIwZZXhhbXBsZS5leGFtcGxlOnNlbnRpbWVudJSMB3J1bnRpbWWUjApweXRob246My45lIwHY29kZXVybJROjAVpbWFnZZSMMXBpcmNsaS1idWlsZDoxMmFiZWZmZi04OWQ3LTQ4M2EtOWIyMy03ODE1M2RkNTEwZGOUdWJzjAlmcmFtZXdvcmuUTowGY29uZmlnlH2UjAZpbnB1dHOUXZQoaACMBUlucHV0lJOUKYGUfZQoaAWMBW1vZGVslIwGaW90eXBllIwERklMRZSMBnNvdXJjZZRoAIwKRGF0YVNvdXJjZZSTlCmBlH2UKIwHbm9kZV9pZJSMBXRyYWlulIwLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklIwGcmV0dXJulIwOZ3JhcGhfaW5wdXRfaWSUTnVijARtZXRhlGgAjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMBW1vZGVslIwLYW5ub3RhdGlvbnOUTnVidWJoHSmBlH2UKGgFjAlzZW50ZW5jZXOUaCGMCURJUkVDVE9SWZRoI2glKYGUfZQoaCiMGmluZmVyX3BpcGVsaW5lLnRyYW5zbGF0ZV8xlGgqTmgrjAZyZXR1cm6UaC1OdWJoLmgwKYGUfZQoaDOMCXNlbnRlbmNlc5RoNU51YnViZYwHb3V0cHV0c5RdlGgAjAZPdXRwdXSUk5QpgZR9lChoBYwGcmV0dXJulGghjAlESVJFQ1RPUlmUaC5oMCmBlH2UKGgzjAZyZXR1cm6UaDVOdWJ1YmFoLmgwKYGUfZQoaDOMCXNlbnRpbWVudJRoNU51YnViLg==
- gASVJAEAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMD3RyYW5zbGF0ZV9tb2RlbJRoCIwERklMRZRoCmgMKYGUfZQoaA+MD3RyYW5zbGF0ZV9tb2RlbJRoEU51YnViaAMpgZR9lChoBowJc2VudGVuY2VzlGgIjAlESVJFQ1RPUlmUaApoDCmBlH2UKGgPjAlzZW50ZW5jZXOUaBFOdWJ1YmUu
- gASV7QIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMGGluZmVyX3BpcGVsaW5lLnNlbnRpbWVudJSMC2VudHJ5cG9pbnRzlH2UjARtYWlulGgAjApFbnRyeXBvaW50lJOUKYGUfZQojAd2ZXJzaW9ulIwCdjGUjAdoYW5kbGVylIwZZXhhbXBsZS5leGFtcGxlOnNlbnRpbWVudJSMB3J1bnRpbWWUjApweXRob246My45lIwHY29kZXVybJROjAVpbWFnZZSMMXBpcmNsaS1idWlsZDpiOGY3ZWJiNC0yYjQ5LTRmODctOGMxYS1hZmRiODViOTU2YWOUdWJzjAlmcmFtZXdvcmuUTowGY29uZmlnlH2UjAZpbnB1dHOUXZQoaACMBUlucHV0lJOUKYGUfZQoaAWMBW1vZGVslIwGaW90eXBllIwERklMRZSMBnNvdXJjZZRoAIwKRGF0YVNvdXJjZZSTlCmBlH2UKIwHbm9kZV9pZJSMBXRyYWlulIwLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklIwGcmV0dXJulIwOZ3JhcGhfaW5wdXRfaWSUTnVijARtZXRhlGgAjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMBW1vZGVslIwLYW5ub3RhdGlvbnOUTnVidWJoHSmBlH2UKGgFjAlzZW50ZW5jZXOUaCGMCURJUkVDVE9SWZRoI2glKYGUfZQoaCiMGmluZmVyX3BpcGVsaW5lLnRyYW5zbGF0ZV8xlGgqTmgrjAZyZXR1cm6UaC1OdWJoLmgwKYGUfZQoaDOMCXNlbnRlbmNlc5RoNU51YnViZYwHb3V0cHV0c5RdlGgAjAZPdXRwdXSUk5QpgZR9lChoBYwGcmV0dXJulGghjAlESVJFQ1RPUlmUaC5oMCmBlH2UKGgzjAZyZXR1cm6UaDVOdWJ1YmFoLmgwKYGUfZQoaDOMCXNlbnRpbWVudJRoNU51YnViLg==
- gASV2gAAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMCXNlbnRlbmNlc5RoCIwJRElSRUNUT1JZlGgKaAwpgZR9lChoD4wJc2VudGVuY2VzlGgRTnVidWJlLg==
depends_on:
train_pipeline.infer_pipeline.translate_1:
condition: service_completed_successfully
train_pipeline.train:
condition: service_completed_successfully
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
train_pipeline.infer_pipeline.translate_1:
Expand All @@ -73,25 +72,24 @@ services:
- -m
- pirlib.backends.docker_batch
- node
- gASV5QIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMGmluZmVyX3BpcGVsaW5lLnRyYW5zbGF0ZV8xlIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBlleGFtcGxlLmV4YW1wbGU6dHJhbnNsYXRllIwHcnVudGltZZSMCnB5dGhvbjozLjmUjAdjb2RldXJslE6MBWltYWdllIwxcGlyY2xpLWJ1aWxkOjEyYWJlZmZmLTg5ZDctNDgzYS05YjIzLTc4MTUzZGQ1MTBkY5R1YnOMCWZyYW1ld29ya5ROjAZjb25maWeUfZSMA2tleZSMBXZhbHVllHOMBmlucHV0c5RdlChoAIwFSW5wdXSUk5QpgZR9lChoBYwGYXJncy4wlIwGaW90eXBllIwERklMRZSMBnNvdXJjZZRoAIwKRGF0YVNvdXJjZZSTlCmBlH2UKIwHbm9kZV9pZJROjAtzdWJncmFwaF9pZJROjAlvdXRwdXRfaWSUTowOZ3JhcGhfaW5wdXRfaWSUjA90cmFuc2xhdGVfbW9kZWyUdWKMBG1ldGGUaACMCE1ldGFkYXRhlJOUKYGUfZQojARuYW1llIwGYXJncy4wlIwLYW5ub3RhdGlvbnOUTnVidWJoHymBlH2UKGgFjAZhcmdzLjGUaCOMCURJUkVDVE9SWZRoJWgnKYGUfZQoaCpOaCtOaCxOaC2MCXNlbnRlbmNlc5R1YmgvaDEpgZR9lChoNIwGYXJncy4xlGg2TnVidWJljAdvdXRwdXRzlF2UaACMBk91dHB1dJSTlCmBlH2UKGgFjAZyZXR1cm6UaCOMCURJUkVDVE9SWZRoL2gxKYGUfZQoaDSMBnJldHVybpRoNk51YnViYWgvaDEpgZR9lChoNIwLdHJhbnNsYXRlXzGUaDZOdWJ1Yi4=
- gASVJAEAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMD3RyYW5zbGF0ZV9tb2RlbJRoCIwERklMRZRoCmgMKYGUfZQoaA+MD3RyYW5zbGF0ZV9tb2RlbJRoEU51YnViaAMpgZR9lChoBowJc2VudGVuY2VzlGgIjAlESVJFQ1RPUlmUaApoDCmBlH2UKGgPjAlzZW50ZW5jZXOUaBFOdWJ1YmUu
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
- gASViQIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMGmluZmVyX3BpcGVsaW5lLnRyYW5zbGF0ZV8xlIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBlleGFtcGxlLmV4YW1wbGU6dHJhbnNsYXRllIwHcnVudGltZZSMCnB5dGhvbjozLjmUjAdjb2RldXJslE6MBWltYWdllIwxcGlyY2xpLWJ1aWxkOmI4ZjdlYmI0LTJiNDktNGY4Ny04YzFhLWFmZGI4NWI5NTZhY5R1YnOMCWZyYW1ld29ya5ROjAZjb25maWeUfZSMA2tleZSMBXZhbHVllHOMBmlucHV0c5RdlGgAjAVJbnB1dJSTlCmBlH2UKGgFjAlzZW50ZW5jZXOUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjAZzb3VyY2WUaACMCkRhdGFTb3VyY2WUk5QpgZR9lCiMB25vZGVfaWSUTowLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklE6MDmdyYXBoX2lucHV0X2lklIwJc2VudGVuY2VzlHVijARtZXRhlGgAjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMCXNlbnRlbmNlc5SMC2Fubm90YXRpb25zlE51YnViYYwHb3V0cHV0c5RdlGgAjAZPdXRwdXSUk5QpgZR9lChoBYwGcmV0dXJulGgjjAlESVJFQ1RPUlmUaC9oMSmBlH2UKGg0jAZyZXR1cm6UaDZOdWJ1YmFoL2gxKYGUfZQoaDSMC3RyYW5zbGF0ZV8xlGg2TnVidWIu
- gASV2gAAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMCXNlbnRlbmNlc5RoCIwJRElSRUNUT1JZlGgKaAwpgZR9lChoD4wJc2VudGVuY2VzlGgRTnVidWJlLg==
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
- ${INPUT_translate_model:?err}:/mnt/graph_inputs/translate_model
- ${INPUT_sentences:?err}:/mnt/graph_inputs/sentences
train_pipeline.train:
command:
- python
- -m
- pirlib.backends.docker_batch
- node
- gASVswIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMBXRyYWlulIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBVleGFtcGxlLmV4YW1wbGU6dHJhaW6UjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6MTJhYmVmZmYtODlkNy00ODNhLTliMjMtNzgxNTNkZDUxMGRjlHVic4wJZnJhbWV3b3JrlGgAjAlGcmFtZXdvcmuUk5QpgZR9lCiMBG5hbWWUjAdhZGFwdGRslGgOTowGY29uZmlnlE51YmgefZQojBRhZGFwdGRsL21pbl9yZXBsaWNhc5RLAYwUYWRhcHRkbC9tYXhfcmVwbGljYXOUSwR1jAZpbnB1dHOUXZRoAIwFSW5wdXSUk5QpgZR9lChoBYwHZGF0YXNldJSMBmlvdHlwZZSMCURJUkVDVE9SWZSMBnNvdXJjZZRoAIwKRGF0YVNvdXJjZZSTlCmBlH2UKIwHbm9kZV9pZJSMBWNsZWFulIwLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklIwGcmV0dXJulIwOZ3JhcGhfaW5wdXRfaWSUTnVijARtZXRhlGgAjAhNZXRhZGF0YZSTlCmBlH2UKGgcjAdkYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJhjAdvdXRwdXRzlF2UaACMBk91dHB1dJSTlCmBlH2UKGgFjAZyZXR1cm6UaCmMBEZJTEWUaDZoOCmBlH2UKGgcjAZyZXR1cm6UaDxOdWJ1YmFoNmg4KYGUfZQoaByMBXRyYWlulGg8TnVidWIu
- gASVJAEAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMD3RyYW5zbGF0ZV9tb2RlbJRoCIwERklMRZRoCmgMKYGUfZQoaA+MD3RyYW5zbGF0ZV9tb2RlbJRoEU51YnViaAMpgZR9lChoBowJc2VudGVuY2VzlGgIjAlESVJFQ1RPUlmUaApoDCmBlH2UKGgPjAlzZW50ZW5jZXOUaBFOdWJ1YmUu
- gASVswIAAAAAAACMCnBpcmxpYi5waXKUjAROb2RllJOUKYGUfZQojAJpZJSMBXRyYWlulIwLZW50cnlwb2ludHOUfZSMBG1haW6UaACMCkVudHJ5cG9pbnSUk5QpgZR9lCiMB3ZlcnNpb26UjAJ2MZSMB2hhbmRsZXKUjBVleGFtcGxlLmV4YW1wbGU6dHJhaW6UjAdydW50aW1llIwKcHl0aG9uOjMuOZSMB2NvZGV1cmyUTowFaW1hZ2WUjDFwaXJjbGktYnVpbGQ6YjhmN2ViYjQtMmI0OS00Zjg3LThjMWEtYWZkYjg1Yjk1NmFjlHVic4wJZnJhbWV3b3JrlGgAjAlGcmFtZXdvcmuUk5QpgZR9lCiMBG5hbWWUjAdhZGFwdGRslGgOTowGY29uZmlnlE51YmgefZQojBRhZGFwdGRsL21pbl9yZXBsaWNhc5RLAYwUYWRhcHRkbC9tYXhfcmVwbGljYXOUSwR1jAZpbnB1dHOUXZRoAIwFSW5wdXSUk5QpgZR9lChoBYwHZGF0YXNldJSMBmlvdHlwZZSMCURJUkVDVE9SWZSMBnNvdXJjZZRoAIwKRGF0YVNvdXJjZZSTlCmBlH2UKIwHbm9kZV9pZJSMBWNsZWFulIwLc3ViZ3JhcGhfaWSUTowJb3V0cHV0X2lklIwGcmV0dXJulIwOZ3JhcGhfaW5wdXRfaWSUTnVijARtZXRhlGgAjAhNZXRhZGF0YZSTlCmBlH2UKGgcjAdkYXRhc2V0lIwLYW5ub3RhdGlvbnOUTnVidWJhjAdvdXRwdXRzlF2UaACMBk91dHB1dJSTlCmBlH2UKGgFjAZyZXR1cm6UaCmMBEZJTEWUaDZoOCmBlH2UKGgcjAZyZXR1cm6UaDxOdWJ1YmFoNmg4KYGUfZQoaByMBXRyYWlulGg8TnVidWIu
- gASV2gAAAAAAAABdlCiMCnBpcmxpYi5waXKUjApHcmFwaElucHV0lJOUKYGUfZQojAJpZJSMDXRyYWluX2RhdGFzZXSUjAZpb3R5cGWUjAlESVJFQ1RPUlmUjARtZXRhlGgBjAhNZXRhZGF0YZSTlCmBlH2UKIwEbmFtZZSMDXRyYWluX2RhdGFzZXSUjAthbm5vdGF0aW9uc5ROdWJ1YmgDKYGUfZQoaAaMCXNlbnRlbmNlc5RoCIwJRElSRUNUT1JZlGgKaAwpgZR9lChoD4wJc2VudGVuY2VzlGgRTnVidWJlLg==
depends_on:
train_pipeline.clean:
condition: service_completed_successfully
image: pircli-build:12abefff-89d7-483a-9b23-78153dd510dc
image: pircli-build:b8f7ebb4-2b49-4f87-8c1a-afdb85b956ac
volumes:
- node_outputs:/mnt/node_outputs
version: '3.9'
Expand Down
47 changes: 31 additions & 16 deletions example/example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,23 +46,44 @@ def evaluate(kwargs: EvaluateInput) -> pandas.DataFrame:
return df


class TranslateModel(object):
def translate(self, inp: str) -> str:
output = f"translation: {inp}"
return output


@task
def translate(args: Tuple[FilePath, DirectoryPath]) -> DirectoryPath:
model, sentences = args
def translate(sentences: DirectoryPath) -> DirectoryPath:
task_ctx = task.context()
with open(model) as f, open(sentences / "file.txt") as g:
model = task_ctx.get("translate_model")
with open(sentences / "file.txt") as g:
inp = g.read().strip()
print(
"translate({}, {}, config={})".format(
f.read().strip(),
g.read().strip(),
"translate({}, config={})".format(
inp,
task_ctx.config)
)
translate_result = model.translate(inp)
outdir = task_ctx.output
with open(outdir / "file.txt", "w") as f:
f.write("translate_result")
f.write(translate_result)
return outdir


@translate.setup
def translate_setup() -> None:
task_ctx = task.context()
task_ctx.set("translate_model", TranslateModel())
zhanyuanucb marked this conversation as resolved.
Show resolved Hide resolved
print(">>> Initialized translation model.")


@translate.teardown
def translate_teardown() -> None:
task_ctx = task.context()
task_ctx.reset("translate_model")
print(">>> Cleaned up translation model.")


@task
def sentiment(model: FilePath, sentences: DirectoryPath) -> DirectoryPath:
with open(model) as f, open(sentences / "file.txt") as g:
Expand All @@ -74,21 +95,19 @@ def sentiment(model: FilePath, sentences: DirectoryPath) -> DirectoryPath:


@pipeline
def infer_pipeline(translate_model: FilePath,
sentiment_model: FilePath,
def infer_pipeline(sentiment_model: FilePath,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not be changing the structure of the example DAG with this PR. Should maintain backward compatibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so make a new example for new features?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I mean, this PR adds the setup and teardown methods. The only change needed in the example should be to add a setup and teardown step to one of the tasks. Not sure why we need to make these other changes like to the task arguments, removing a file input, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the file input for translation model since now the translation model is initialized by task's setup method. If we need to keep it, then I guess it should mean something else? For example, model checkpoint.

sentences: DirectoryPath) -> DirectoryPath:
translate_1 = translate.instance("translate_1")
translate_1.config["key"] = "value"
return sentiment(sentiment_model, translate_1((translate_model, sentences)))
return sentiment(sentiment_model, translate_1(sentences))


@pipeline
def train_pipeline(
train_dataset: DirectoryPath,
translate_model: FilePath,
sentences: DirectoryPath) -> Tuple[FilePath, pandas.DataFrame]:
sentiment_model = train(clean(train_dataset))
sentiment = infer_pipeline(translate_model, sentiment_model, sentences)
sentiment = infer_pipeline(sentiment_model, sentences)
eval_input = {"test_dataset": sentences, "predictions": sentiment}
return sentiment_model, evaluate(eval_input)

Expand All @@ -98,17 +117,13 @@ def train_pipeline(
print(yaml.dump(asdict(package), sort_keys=False))
# Prepare inputs.
dir_1 = tempfile.TemporaryDirectory()
file_2 = tempfile.NamedTemporaryFile()
dir_3 = tempfile.TemporaryDirectory()
with open(f"{dir_1.name}/file.txt", "w") as f:
f.write("train_dataset")
with open(f"{file_2.name}", "w") as f:
f.write("translate_model")
with open(f"{dir_3.name}/file.txt", "w") as f:
f.write("sentences")
# Test calling end-to-end pipeline.
model_path, metrics = train_pipeline(DirectoryPath(dir_1.name),
FilePath(file_2.name),
DirectoryPath(dir_3.name))
with open(model_path) as f:
print("pipeline model: {}".format(f.read().strip()))
Expand Down
Loading