Parallel Graph & Mesh Partitioning

I have spent the last day chasing down an "assertion failed" in parmetis that apparently occurred because every node in my graph was connected to itself.
I don't know if you call this a circular edge, or a 1-node edge, or what. Removing these edges from my adjacency vector caused the program to run correctly.

The assertion failed in Match_Global and appeared that certain ranks were asked for information about nodes that they did not own -- I can send on more information if this is helpful.

My real question is this: is it surprising that this kind of graph would cause ParMETIS to fail? It partitioned fine when run in serial.

thanks,
Craig Tanis

Hello,
I have been able to use the routine ParMETIS_V3_PartKway in parmetis's library, however I only managed to do so with one processor. In this case there is only one set of arrays (vtxdist, xadj, adjncy, ...) which is identical to the CSR format.
I just call the routine with the pointers to the single set of arrays that I have:

int j = ParMETIS_V3_PartKway(vtxdist, xadj, adjncy, vwgt, adjwgt, wgtflag, numflag, ncon, nparts, tpwgts, ubvec, options, edgecut, part, &comm);

I'm not clear about how to handle the different sets of arrays for more than one processor. Do I need to call the routine multiple times, each time with the data for that particular processor? Or do I need to call the routine once, using arrays of arrays with the data for each processor? Or is there a different procedure?
I haven't been able to find instructions on how to implement this. Do you have any sample code?
Thank you in advance for any help you can provide

Sorry if the question has already been asked (but I haven't found it).
I am a very beginner with ParMetis, and I use it through Petsc. I run it
with the (apperently) same installation (ParMetis-dev-p3), but I get different
partitioning depending on the machines I am using.
Is that the expected behaviour?

Thank you,
FL

Hello,

I'm using ParMETIS for mesh partitioning in a finite element code. I'm building the dual graph myself and using the ParMETIS_V3_PartKway routine for partitioning the graph.

For N_parts == N_MPIprocs and for N_parts != N_MPIprocs; N_MPIprocs == 1 everything is working fine.

The use case that I need is the following:
* N_MPIprocs = 1 (I can use METIS too, if that library is more appropriate for this)
* N_parts - arbitrary
* N_group < N_parts; N_group % N_parts == 0; such that the first N_group consecutively numbered parts form a contiguous mesh part, the next N_group parts form another contiguous mesh part and so no.

Is this possible with ParMETIS?

I'm adding a diagram, I'm not sure how well I've explained:

12 mesh parts, groups of 4 mesh parts

===================================
|| P0 | P1 || P4 | P5 || P8 | P9 ||
==---------==---------==---------==
|| P2 | P3 || P6 | P7 || P10| P11||
===================================

HI;
we are using Parmetis 3.2 and running on some issues.
Parmetis works quite well and balances perfectly the number of elements per partition.
However the number of corresponding nodes is quite inbalanced. Mostly we believe it comes from the non connex partitions and the way our dual mesh was numbered ( we use the Parmetis routine to generate the dual mesh by the way).
This is a major issue for us since a lot of operations are node based.

I was wondering if there is a way to stronly encourage Parmetis to provide connex partitions and balance the number of nodes per domain a little more ( variability is 20%).

thanks

Hi,

I was looking where was defined the type idx_t and found that I have to manually modify a file (metis.h) to change to 64 bit indices... Am I right?
If so, I would suggest to modify the "Makefile" to add this as a configuration parameter when one do a "make config". Moreover, with a default value, users won't be bothered by an obligation to manage this new parameter.

Thanks,

Eric

Hi,

I'm installing parMetis 4.02 onto our cluster for our users. I would like to verify that the compiled binaries work properly in our environment, but I'm not finding much in the documentation other than stating that there are files in the 'Graphs' directory which can be used for testing. It looks like mtest and ptest are the two binaries available for testing, so I've run those with the following files, and received their output (not sure if they're correct):

$ mtest bricks.hex3d
Nelements: 117649, Nnodes: 125000, EType: 3
MGCNUM: 4

...is this the right graph file, and is output correct? And for ptest:

$ptest rotor.graph
******begin output******
Testing ParMETIS_V3_PartKway with ncon: 1, nparts: 2
ParMETIS_V3_PartKway reported a cut of 2207 [OK:2207]

Testing ParMETIS_V3_RefineKway with ncon: 1, nparts: 2
Setup: Max: 0.017, Sum: 0.017, Balance: 1.000
Matching: Max: 0.052, Sum: 0.052, Balance: 1.000

Hello,

I am using Parmetis to partition a mesh used in a Direct Simulation Monte Carlo code. The code uses finite element terminology for the mesh, notably elements and nodes (or vertices). As I understand it, Parmetis (and Metis) uses a graph partitioning paradigm and corresponding terminology. I believe, in particular, the term 'vertex' in Parmetis can refer to different entities, depending on the situation, and should not be confused with the finite element term 'vertex'. Specifically I believe the description for parameter 'part' in ParMETIS_V3_PartMeshKway: ... 'Upon successful completion the partition vector of the locally-stored vertices is written to this array.' does not mean 'vertices' in the finite element sense, but actually refers to 'elements'. I would appreciate some confirmation of this and further clarification if possible - perhaps with reference to other documents describing the graph partitioning paradigm.

Hello,

I'm new with METIS library and just tried to run the function ParMETIS_V3_AdaptiveRepart, but it fails. I check a lot of variants and couldn't find out what is wrong with it.
Here is a source:

int mpisize, rank;
MPI_Comm_size(MPI_COMM_WORLD, &mpisize);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

MPI_Comm comm = MPI_COMM_WORLD;

idx_t* xadj = new idx_t[16];
idx_t* adjncy = new idx_t[43];

int xtmp[16] = {0, 2, 5, 8, 11, 13, 16, 20, 24, 28, 31, 33, 36, 39, 42, 44};
for(int i=0; i<16; i++) xadj[i] = xtmp[i];
int atmp[44] = {1, 5, 0, 2, 6, 1, 3, 7, 2, 4, 8, 3, 9, 0, 6, 10, 1, 5, 7, 11, 2, 6, 8, 12, 3, 7, 9, 13, 4, 8, 14, 5, 11, 6, 10, 12, 7, 11, 13, 8, 12, 14, 9, 13};
for(int i=0; i<43; i++) adjncy[i] = atmp[i];
idx_t* part = new idx_t[15];
for(int i=0; i<15; i++) part[i] = -1;

idx_t ncon = 1;
idx_t nparts = 2;
idx_t* options = new idx_t[4];
options [ 0 ] = 1; // set to zero for default

I have a very large FEM model ( No. of element: 19,294,342, No. of nodes: 3,403,564 ). I use parMetis to partiton the model(ParMETIS_V3_PartMeshKway). The code works very well when I use 512 processors. But it failed when I attempt to use 640 processors (same error when I use 1024 processors). The code did not return the partition. And the error is ( Maximum memory used: 53727992 bytes, Current memory used: 30653200 bytes, ***Memory realloc failed for cnbrpoolGet: cnbrpool. Requested size: 1696128 bytes ). And I located the function (idx_t cnbrpoolGetNext(ctrl_t *ctrl, idx_t nnbrs)).

I am wondering what causes the error, is it my own fem code, or the parMetis, or else?

Thanks in advance.