Dynamic host management/communicators still broken

[CalugaruVaxile]

Hi Dev team,

With OpenMPI 3.0.0 the code crashes when I try two MPI_Comm_spawn successive directives, while one single spawn works perfectly. I have downloaded and built the head master OpenMPI, same issue. OS = Debian 7.

The full code is intended to dynamically add/remove hosts to/from a "uber" intracomm (which will be used by the main computation). Therefore I am trying to spawn+merge new communicators and separate/detach nodes in various sequences which will hopefully cover all the situations.

"Prototype" Master code:

`
#include
#include
#include <mpi.h>
#include <unistd.h>
#include <limits.h>

using namespace std;

int main() {
char slavejobtospawn[500];
strcpy(slavejobtospawn, "./bug_slave");
char localhost[HOST_NAME_MAX];
gethostname(localhost, HOST_NAME_MAX);

// == comms
MPI_Comm worker_comm1_;
MPI_Comm worker_comm2_;

// == init MPI
int provided;
MPI_Init_thread(0, 0, MPI_THREAD_MULTIPLE, &provided);
if (provided < MPI_THREAD_MULTIPLE) {
cout << "ERROR: The MPI library does not have full thread support" << endl;
MPI_Abort(MPI_COMM_WORLD, 1);
};

// == MPI_info obj
MPI_Info minfo;
MPI_Info_create(&minfo);

// == first spawn
MPI_Info_set(minfo, "add-host", "houprg118070,houprg118071");
MPI_Comm_spawn(slavejobtospawn, MPI_ARGV_NULL,
2, minfo, 0, MPI_COMM_WORLD, &worker_comm1_, MPI_ERRCODES_IGNORE);

// == second spawn
MPI_Info_set(minfo, "add-host", "houprg118072");
MPI_Comm_spawn(slavejobtospawn, MPI_ARGV_NULL,
1, minfo, 0, MPI_COMM_WORLD, &worker_comm2_, MPI_ERRCODES_IGNORE);

usleep(5000000);

// == stop MPI
MPI_Finalize();
cout << "MASTER " << localhost << " SHUTTING DOWN" << endl;
return 0;
}
`

"Prototype" Slave code
`
#include
#include
#include <mpi.h>
#include <unistd.h>
#include <limits.h>

using namespace std;

int main() {
char localhost[HOST_NAME_MAX];
gethostname(localhost, HOST_NAME_MAX);

// == comms
MPI_Comm slave_Comm_;

// == init MPI, get parent
int provided;
MPI_Init_thread(0, 0, MPI_THREAD_MULTIPLE, &provided);
if (provided < MPI_THREAD_MULTIPLE) {
cout << "ERROR: The MPI library does not have full thread support" << endl;
MPI_Abort(MPI_COMM_WORLD, 1);
};
MPI_Comm_get_parent(&slave_Comm_);

// == test spawning
cout << "SLAVE " << localhost << " GETS PARENT" << slave_Comm_ << endl;

usleep(5000000);
// == stop MPI
MPI_Finalize();
cout << "SLAVE " << localhost << " SHUTTING DOWN" <<endl;
return 0;
}
`

Error

`
%mpirun -np 1 ./bug_master

SLAVE houprg118071 GETS PARENT0x715b40
SLAVE houprg118070 GETS PARENT0x715f40
[houprg118061:23984] PACK-ORTE-ATTR: UNSUPPORTED TYPE
[houprg118061:23984] [[60201,0],0] ORTE_ERROR_LOG: Error in file runtime/data_type_support/orte_dt_unpacking_fns.c at line 109
[houprg118061:23984] [[60201,0],0] ORTE_ERROR_LOG: Error in file base/odls_base_default_fns.c at line 416
[houprg118070:07813] PACK-ORTE-ATTR: UNSUPPORTED TYPE
[houprg118070:07813] [[60201,0],1] ORTE_ERROR_LOG: Error in file runtime/data_type_support/orte_dt_unpacking_fns.c at line 109
[houprg118070:07813] [[60201,0],1] ORTE_ERROR_LOG: Error in file base/odls_base_default_fns.c at line 416
[houprg118071:09950] PACK-ORTE-ATTR: UNSUPPORTED TYPE
[houprg118071:09950] [[60201,0],2] ORTE_ERROR_LOG: Error in file runtime/data_type_support/orte_dt_unpacking_fns.c at line 109
[houprg118071:09950] [[60201,0],2] ORTE_ERROR_LOG: Error in file base/odls_base_default_fns.c at line 416
`

If I comment the second MPI_comm_spawn or if I replace the MPI_info object with MPI_INFO_NULL, everything works well.

I hope this can be fixed.

Cheers,
George

[rhc54]

Just curious - what happens if you use a different minfo (i.e., don't overwrite the first one) for the second call?

[CalugaruVaxile]

Hi Ralph, I tried that. Same error, as long as it's not MPI_INFO_NULL. In a different context (if I try to merge the spawned comm with MPI_COMM_WORLD first, then spawn f om the merged comm) I get an ssh error saying host cannot be resolved (on the new host I try to spawn to) and a small comment saying iptables may not be working. These two happen for mpirun -np 1 ./exec If I try mpirun -np 9 -hostfile lamhosts_tmp ./exec spawning on a host belonging to minfo fails as before (tcp connection can't be established). This is not that important for me, but maybe helps you Cheers, George

…

On Dec 22, 2017 23:30, "Ralph Castain" ***@***.***> wrote: Just curious - what happens if you use a different minfo (i.e., don't overwrite the first one) for the second call? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4665 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AgkAvdoE3w_MRX43s9AcR9tSa2SA8xJYks5tDI_vgaJpZM4RLcJk> .

[CalugaruVaxile]

Hi Ralph,

I see you have committed some changes. Should I try to rerun the tests with the head version?

Cheers,
George

[rhc54]

I believe there is still one remaining problem in the grpcomm code: #4718 . However, you are welcome to give it a try.

[CalugaruVaxile]

Hi Ralph,

preliminary tests look good (multiple spawns on subsequently-added hosts works now). I still need to run the successive (spwan + communicator merge) tests. I will come back to you with the full test results

Cheers,
George

[CalugaruVaxile]

Hi Ralph,

I confirm, it is now possible to do this

1. start with no slaves at all (just master)
2. spawn N slaves -> slaveCommunicator
3. merge slaveCommunicator with master into the intracomm spawnedCommunicator
4. repeat 2-3 and creating many intercomms

Great work, many thanks for the fix!

What I cannot do is to merge these spawned communicators into one global intercomm. I admit I do not know which is the correct way to do it, as I can't find relevant documentation. What I tried is

a. do the second MPI_COMM_SPAWN not from the MPI_COMM_WORLD (aka "master" in this case) but from the intracomm resulting from merging the first spawned comm with master
`
MPI_Info_set(minfo, "add-host", "houprg118070,houprg118070");
MPI_Comm_spawn(slavejobtospawn, MPI_ARGV_NULL, 2, minfo, 0, initial_comm_, &extra_comm1_, MPI_ERRCODES_IGNORE);
// == merging first spawned
MPI_Intercomm_merge(extra_comm1_,0, &merged_comm1_);
int tag =1;
MPI_Bcast(&tag, 1, MPI_INT, 0, merged_comm1_);
// == spawn second set
MPI_Info_set(minfo, "add-host", "houprg118073,houprg118073");
MPI_Comm_spawn(slavejobtospawn, MPI_ARGV_NULL, 1, minfo, 0, merged_comm1_,
&extra_comm2_, MPI_ERRCODES_IGNORE);
'
but this fails with TCP erors

b. spawn two comms, merge them with master, then create an intercomm between them (with the idea of merging this one with the master and getting a global intracomm). Unfortunately
'
MPI_Intercomm_create(merged_comm1_, 1, merged_comm2_, 1, 0, &intercommToMerge);
'
hangs indefinitely.

Please do let me know which is - conceptually - the correct way of creating a unified intracomm from two (or more) spawned communicators. If it happens that this doesn't currently work, I am more than happy to do all the tests needed to help you debug

Cheers,
George

[sidharths]

I face a similar problem with multiple spawning. Was this issue fixed in openmpi 3.1.2?

[rhc54]

The following program runs correctly on master:

intercomm create program

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <unistd.h>

#include <mpi.h>

#define NB_SPAWN 4

static void do_parent(char *argv[], int rank, int count);
static void do_target(char* argv[], MPI_Comm parent);

static char *cmd_argv1 = "b";
static char *cmd_argv2 = "c";
static char *whoami = "a";
static int tag = 201;

void ompitest_warning( char* filename, int lineno, const char* fmt, ... )
{
    char* buf = NULL;
    va_list va_list;

    va_start(va_list, fmt);
    opal_vasprintf( &buf, fmt, va_list );
    va_end(va_list);
    printf( "*warning* %s:%d %s\n", filename, lineno, buf );
    free(buf);
}

void ompitest_error( char* filename, int lineno, const char* fmt, ... )
{
    char* buf = NULL;
    va_list va_list;

    va_start(va_list, fmt);
    opal_vasprintf( &buf, fmt, va_list );
    va_end(va_list);
    printf( "*error* %s:%d %s\n", filename, lineno, buf );
    free(buf);
}

int
main(int argc, char *argv[])
{
    int rank, size;
    MPI_Comm parent;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    /* Check to see if we *were* spawned -- because this is a test, we
       can only assume the existence of this one executable.  Hence, we
       both mpirun it and spawn it. */

    parent = MPI_COMM_NULL;
    MPI_Comm_get_parent(&parent);
    if (parent != MPI_COMM_NULL) {
        whoami = argv[1];
        do_target(argv, parent);
    } else {
        do_parent(argv, rank, size);
    }

    /* All done */
    MPI_Finalize();
    return 0;
}

static int
spawn_and_merge( char* argv[], char* arg, int count,
                 MPI_Comm* inter, MPI_Comm* intra )
{
    int *errcode, err, i;
    char *spawn_argv[2];

    errcode = malloc(sizeof(int) * count);
    if (errcode == NULL)
        ompitest_error(__FILE__, __LINE__, "Doh!  Rank %d was not able to allocate enough memory.  MPI test aborted!\n", 0);
    memset(errcode, -1, count);
    /*MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);*/

    spawn_argv[0] = arg;
    spawn_argv[1] = NULL;
    err = MPI_Comm_spawn(argv[0], spawn_argv, count, MPI_INFO_NULL, 0,
                         MPI_COMM_WORLD, inter, errcode);
    for (i = 0; i < count; i++)
        if (errcode[i] != MPI_SUCCESS)
            ompitest_error(__FILE__, __LINE__,
                           "ERROR: MPI_Comm_spawn returned errcode[%d] = %d\n",
                           i, errcode[i]);
    if (err != MPI_SUCCESS)
        ompitest_error(__FILE__, __LINE__,
                       "ERROR: MPI_Comm_spawn returned errcode = %d\n", err);
    err = MPI_Intercomm_merge( *inter, 0, intra );
    free(errcode);
    return err;
}

static void
do_parent(char *argv[], int rank, int count)
{
    MPI_Comm ab_inter, ab_intra, ac_inter, ac_intra, ab_c_inter, abc_intra;
    int err;

    err = spawn_and_merge( argv, cmd_argv1, count, &ab_inter, &ab_intra );
    err = spawn_and_merge( argv, cmd_argv2, count, &ac_inter, &ac_intra );

    printf( "%s: MPI_Intercomm_create( ab_intra, 0, ac_intra, %d, %d, &inter) (%d)\n",
            whoami, count, tag, err );
    err = MPI_Intercomm_create( ab_intra, 0, ac_intra, count, tag, &ab_c_inter );
    printf( "%s: intercomm_create (%d)\n", whoami, err );

    printf( "%s: barrier on inter-comm - before\n", whoami );
    err = MPI_Barrier(ab_c_inter);
    printf( "%s: barrier on inter-comm - after\n", whoami );

    err = MPI_Intercomm_merge(ab_c_inter, 0, &abc_intra);
    printf( "%s: intercomm_merge(%d) (%d) [rank %d]\n", whoami, 0, err, rank );
    err = MPI_Barrier(abc_intra);
    printf( "%s: barrier (%d)\n", whoami, err );

    MPI_Comm_free(&abc_intra);
    MPI_Comm_free(&ab_c_inter);
    MPI_Comm_free(&ab_intra);
    MPI_Comm_free(&ac_intra);

    MPI_Comm_disconnect(&ab_inter);
    MPI_Comm_disconnect(&ac_inter);
}


static void
do_target(char* argv[], MPI_Comm parent)
{
    int rank, first = 0, err;
    MPI_Comm intra, inter, merge1;

    if( 0 == strcmp(argv[1], cmd_argv1) ) first = 1;

    /*MPI_Comm_set_errhandler(parent, MPI_ERRORS_RETURN);*/

    err = MPI_Intercomm_merge( parent, 1, &intra );
    MPI_Comm_rank(intra, &rank);

    if( first ) {
        printf( "%s: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, %d, &inter) [rank %d]\n", whoami, tag, rank );
        err = MPI_Intercomm_create( intra, 0, MPI_COMM_NULL, 0, tag, &inter);
        printf( "%s: intercomm_create (%d)\n", whoami, err );
    } else {
        printf( "%s: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, %d, &inter) [rank %d]\n", whoami, tag, rank );
        err = MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, tag, &inter);
        printf( "%s: intercomm_create (%d)\n", whoami, err );
    }
    printf( "%s: barrier on inter-comm - before\n", whoami );
    err = MPI_Barrier(inter);
    printf( "%s: barrier on inter-comm - after\n", whoami );

    err = MPI_Intercomm_merge( inter, 0, &merge1 );
    MPI_Comm_rank(merge1, &rank);
    printf( "%s: intercomm_merge(%d) (%d) [rank %d]\n", whoami, first, err, rank );
    err = MPI_Barrier(merge1);
    printf( "%s: barrier (%d)\n", whoami, err );

    MPI_Comm_free(&merge1);
    MPI_Comm_free(&inter);
    MPI_Comm_free(&intra);

    MPI_Comm_disconnect(&parent);
}

[rhc54]

This appears fixed on master and v5