Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PMI2 shim crashes #4

Open
jjhursey opened this issue Aug 9, 2019 · 1 comment
Open

PMI2 shim crashes #4

jjhursey opened this issue Aug 9, 2019 · 1 comment

Comments

@jjhursey
Copy link
Member

jjhursey commented Aug 9, 2019

Background information

What version of the PMIx Reference Library are you using? (e.g., v1.0, v2.1, git master @ hash, etc.)

Describe how PMIx was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

PMIx built with:

    ./configure --prefix=${PMIX_ROOT} \
                --with-hwloc=${HWLOC_INSTALL_PATH} \
                --with-libevent=${LIBEVENT_INSTALL_PATH} \
                --enable-debug

Open MPI built with:

    ./configure --prefix=${MPI_ROOT} \
                --with-hwloc=${HWLOC_INSTALL_PATH} \
                --with-libevent=${LIBEVENT_INSTALL_PATH} \
                --with-pmix=${PMIX_ROOT} \
                --enable-mpirun-prefix-by-default

Please describe the system on which you are running

  • Operating system/version: RHEL 7.6
  • Computer hardware: ppc64le
  • Network type: localhost

Details of the problem

I was testing PMI2 shim with the pmi2_client.c in the v3.1.4 release.

[mpiuser@3cd5ac47d23f test]$ env | grep PMIX
PMIX_ROOT=/home/mpiuser/local/pmix
[mpiuser@3cd5ac47d23f test]$ mpicc pmi2_client.c -I${PMIX_ROOT}/include -L${PMIX_ROOT}/lib -lpmi2 -Wall -g -O0 -o pmi2_client

If I don't specify a gds component it will crash with the stack below:

Program trace with no gds specified 
[mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client
INFO: spawned=0 size=20 rank=0 appnum=0
INFO: jobid=2406219777
INFO: TI1  : PASS
INFO: TI2  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4  : PASS
INFO: spawned=0 size=20 rank=1 appnum=0
INFO: jobid=2406219777
INFO: TI1  : PASS
INFO: TI2  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4  : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0
INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0
FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341
FATAL: value does not meet expectation at test_item5:342
FATAL: value does not meet expectation at test_item5:343
INFO: TI5  : FAIL
FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341
FATAL: value does not meet expectation at test_item5:342
FATAL: value does not meet expectation at test_item5:343
INFO: TI5  : FAIL
INFO: TI6  : PASS
INFO: TI6  : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: TI7  : PASS
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: TI7  : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8  : FAIL
INFO: Rank 1 executing Put of key KEY-1
[3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8  : FAIL
INFO: Rank 0 executing Put of key KEY-0
INFO: Rank 1 executing Fence
INFO: Rank 0 executing Fence
INFO: Rank 0 executing Get of key KEY-0
INFO: Rank 1 executing Get of key KEY-0
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 0 executing Get of key KEY-1
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 1 executing Get of key KEY-1
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 0 executing Get of key KEY-2
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 1 executing Get of key KEY-2
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9  : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9  : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72
INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
[3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10  : FAIL
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10  : FAIL
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[36716,1],1]
  Exit code:    4
--------------------------------------------------------------------------

I noticed a slightly different behavior if I forced the hash component.

Program trace with `hash` gds specified 
[mpiuser@3cd5ac47d23f test]$ export PMIX_MCA_gds=hash
[mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client
INFO: spawned=0 size=20 rank=0 appnum=0
INFO: jobid=2406678529
INFO: TI1  : PASS
INFO: TI2  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4  : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1
INFO: TI5  : PASS
INFO: TI6  : PASS
INFO: spawned=0 size=20 rank=1 appnum=0
INFO: jobid=2406678529
INFO: TI1  : PASS
INFO: TI2  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3  : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4  : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1
INFO: TI5  : PASS
INFO: TI6  : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: TI7  : PASS
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: TI7  : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
[3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8  : FAIL
INFO: Rank 1 executing Put of key KEY-1
[3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8  : FAIL
INFO: Rank 0 executing Put of key KEY-0
INFO: Rank 1 executing Fence
INFO: Rank 0 executing Fence
INFO: Rank 1 executing Get of key KEY-0
INFO: Rank 0 executing Get of key KEY-0
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 1 executing Get of key KEY-1
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 0 executing Get of key KEY-1
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 1 executing Get of key KEY-2
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 0 executing Get of key KEY-2
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9  : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15
INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaa] 0
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9  : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15
INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxx] 0
[3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10  : FAIL
[3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10  : FAIL
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[36723,1],0]
  Exit code:    3
--------------------------------------------------------------------------

I'm not sure if this is a PMIx library issue or something with Open MPI as I see a mix of ORTE and PMIx error messages in there.
@rhc54
Copy link
Contributor

rhc54 commented Oct 15, 2020

This should/will be shifted to the new PMI-1/2 backward compatibility repo: https://github.com/openpmix/pmi-shim

@rhc54 rhc54 closed this as completed Oct 15, 2020
@rhc54 rhc54 reopened this Oct 15, 2020
@rhc54 rhc54 transferred this issue from openpmix/openpmix Oct 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhc54 @jjhursey