Skip to content

Conversation

@maxfirmbach
Copy link
Contributor

@maxfirmbach maxfirmbach commented Apr 2, 2025

Description and Context

This PR mainly consists of the first changes moving from Isorropia to Zoltan2 and for now mostly acts to view code changes and observe failing tests.

Current problems:

Related Issues and Pull Requests

@maxfirmbach maxfirmbach added status: in progress Issues that are currently being worked on taskforce: tpetra Issues related to the migration from Epetra to Tpetra team: discretization labels Apr 2, 2025
@maxfirmbach maxfirmbach added this to the Migration to Tpetra milestone Apr 2, 2025
@maxfirmbach maxfirmbach self-assigned this Apr 2, 2025
@maxfirmbach maxfirmbach marked this pull request as draft April 2, 2025 10:04
@isteinbrecher
Copy link
Contributor

isteinbrecher commented Apr 2, 2025

I took the liberty of collecting all failing tests:


 20 - beam3eb_backweuler_browndyn_singlefil.4C.yaml-p3 (Failed)
 31 - beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_first.4C.yaml-p2_for_beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_second.4C.yaml-p3-restart_step_1 (Failed)
 32 - beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_second.4C.yaml-p3_from_beam3eb_static_beam_to_solid_volume_meshtying_circ_restart_first.4C.yaml-p2-restart_step_1 (Not Run)
 47 - beam3r_herm2line3_backweuler_browndyn_crosslinking_beam3rline2_belloffrate.4C.yaml-p2 (Failed)
 52 - beam3r_herm2line3_backweuler_browndyn_randcrosslinking_beam3rline2.4C.yaml-p2 (Failed)
 53 - beam3r_herm2line3_backweuler_browndyn_randcrosslinking_beam3rline2_twobonds.4C.yaml-p3 (Failed)
 56 - beam3r_herm2line3_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 57 - beam3r_herm2line3_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 186 - beam3r_line2_backweuler_browndyn_initialcrosslinking_beam3rline2_largesys.4C.yaml-p3 (Failed)
 187 - beam3r_line2_backweuler_browndyn_largesys.4C.yaml-p3 (Failed)
 194 - beam3r_line2_backweuler_browndyn_randcrosslinking_beam3rline2_largesys.4C.yaml-p3 (Failed)
 197 - beam3r_line2_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 198 - beam3r_line2_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 228 - beam3r_rigidsphere_activecrosslinking_truss_browndyn_truss_catchslipbond.4C.yaml-p3 (Failed)
 229 - beam3r_rigidsphere_activecrosslinking_truss_browndyn_truss_forcedependetoffrate.4C.yaml-p3 (Failed)
 255 - beam3wk_tanline3_backweuler_browndyn_singlefil_ptc_elementbased_everydt.4C.yaml-p2 (Failed)
 256 - beam3wk_tanline3_backweuler_browndyn_singlefil_ptc_elementbased_everyiter.4C.yaml-p2 (Failed)
 327 - contact2D_parredist.4C.yaml-p2 (Failed)
 328 - contact2D_parredist_mineleproc.4C.yaml-p3 (Failed)
 471 - elch_1D_line2_multiscale_butlervolmer_electrolyte.4C.yaml-p3 (Failed)
 472 - elch_1D_line2_multiscale_butlervolmer_electrolyte.4C.yaml-p3-restart_from_same_input (Not Run)
 473 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro.4C.yaml-p3 (Failed)
 474 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro.4C.yaml-p3-restart_from_same_input (Not Run)
 475 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro_aitken.4C.yaml-p3 (Failed)
 476 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_macrotomicro_aitken_dofsplit.4C.yaml-p3 (Failed)
 477 - elch_1D_line2_multiscale_butlervolmer_electrolyte_partitioned_microtomacro.4C.yaml-p3 (Failed)
 539 - elch_3D_hex8_line2_scl_2dof_BGS_AMG_2x2.4C.yaml-p2-csv_comparison-xxx_micro_macro_coupling_coords.csv (Failed)
 547 - elch_3D_tet4_s2i_butlervolmer_AMG_pointbased.4C.yaml-p3 (Failed)
 548 - elch_3D_tet4_s2i_butlervolmer_AMG_vectorbased.4C.yaml-p3 (Failed)
 549 - elch_3D_tet4_s2i_butlervolmer_AMG-BGS_3x3.4C.yaml-p3 (Failed)
 550 - elch_3D_tet4_s2i_butlervolmer_BGS-AMG_3x3.4C.yaml-p3 (Failed)
 562 - elch_3D_tet4_s2i_butlervolmer_mortar_standard_redist.4C.yaml-p3 (Failed)
 563 - elch_3D_tet4_s2i_butlervolmer_mortar_standard_redist_BGS-AMG_3x3.4C.yaml-p3 (Failed)
 576 - elch_gaussian_hill_pbc_genalpha.4C.yaml-p2 (Failed)
 577 - elch_gaussian_hill_pbc_genalpha.4C.yaml-p2-restart_from_same_input (Not Run)
 579 - elch_gaussian_hill_pbc_NP_genalpha_calcInitDt_calcInitPot.4C.yaml-p2 (Failed)
 581 - elch_Kwok_Wu_BDF2.4C.yaml-p2 (Failed)
 582 - elch_Kwok_Wu_BDF2.4C.yaml-p2-restart_from_same_input (Not Run)
 602 - elch_pulseplating_hex27_moving_boundary.4C.yaml-p3 (Failed)
 603 - elch_pulseplating_hex27_moving_boundary.4C.yaml-p3-restart_from_same_input (Not Run)
 604 - elch_RCE_rotsymm_pbc.4C.yaml-p2 (Failed)
 709 - f3_bfs_xwall_8x8x8_bl1_turbinflow.4C.yaml-p3 (Failed)
 710 - f3_cha_8x8x8_recongradl2.4C.yaml-p2 (Failed)
 711 - f3_cha_8x8x8_recongradspr.4C.yaml-p2 (Failed)
 712 - f3_cha_xwall_6x8x6.4C.yaml-p3 (Failed)
 714 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp.4C.yaml-p2 (Failed)
 715 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp.4C.yaml-p2-restart_from_same_input (Not Run)
 716 - f3_cha_xwall_6x8x6_grad_ol2p_bs_ramp_ale_mfs.4C.yaml-p2 (Failed)
 718 - f3_cha_xwall_6x8x6_gradres_p_bs_ramp_ale.4C.yaml-p2 (Failed)
 741 - f3_channel_meshtying_DCmaster.4C.yaml-p2 (Failed)
 742 - f3_channel_meshtying_DCmaster.4C.yaml-p2-restart_from_same_input (Not Run)
 743 - f3_channel_mixedhybrid.4C.yaml-p3 (Failed)
 744 - f3_channel_mixedhybrid.4C.yaml-p3-restart_from_same_input (Not Run)
 751 - f3_drt_genalpha_initialfield_volforce_pbc.4C.yaml-p2 (Failed)
 752 - f3_drt_impl_initialfield_volforce_pbc.4C.yaml-p2 (Failed)
 756 - f3_nurbs27_AfGenAlpha.4C.yaml-p3 (Failed)
 757 - f3_periodic_hill_w_forcing_24x12x12.4C.yaml-p3 (Failed)
 760 - f3_shear_flow_linear_ramp_periodic_fix_pressure_new_ost.4C.yaml-p3 (Failed)
 762 - f3_turbinflow_doublechan.4C.yaml-p2 (Failed)
 763 - f3_turbinflow_doublechan.4C.yaml-p2-restart_from_same_input (Not Run)
 764 - f3_turbinflow_doublechan_transferdbc.4C.yaml-p2 (Failed)
 776 - f3_womersley.4C.yaml-p2 (Failed)
 777 - f3_womersley.4C.yaml-p2-restart_from_same_input (Not Run)
 806 - fsci_simp_salz_syst.4C.yaml-p3 (Failed)
 824 - fsi_dc_mono_ss_ada_euler_none.4C.yaml-p2 (Failed)
 839 - fsi_dc3D_part_ait_ga_ost_xwall.4C.yaml-p3 (Failed)
 874 - fsi_pw_mono_fs_ga_ga_contact.4C.yaml-p2 (Failed)
 925 - levelset_gaussian_hill_pbc.4C.yaml-p2 (Failed)
 926 - levelset_gaussian_hill_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 945 - loma_turbmodel_test_dynsmag.4C.yaml-p2 (Failed)
 946 - loma_turbmodel_test_dynsmag.4C.yaml-p3 (Failed)
 1402 - poro_3D_hex8_nopenetration.4C.yaml-p2 (Failed)
 1566 - scatra_1D_line2_multiscale_constperm_macro.4C.yaml-p3 (Failed)
 1567 - scatra_1D_line2_multiscale_constperm_macro.4C.yaml-p3-restart_from_same_input (Not Run)
 1568 - scatra_1D_line2_multiscale_constperm_macro_partitioned_macrotomicro.4C.yaml-p3 (Failed)
 1569 - scatra_1D_line2_multiscale_constperm_macro_partitioned_macrotomicro.4C.yaml-p3-restart_from_same_input (Not Run)
 1570 - scatra_1D_line2_multiscale_constperm_macro_partitioned_microtomacro.4C.yaml-p3 (Failed)
 1612 - scatra_3D_tet4_hex8_s2i_constperm_mortar_standard.4C.yaml-p3 (Failed)
 1613 - scatra_3D_tet4_hex8_s2i_constperm_mortar_standard.4C.yaml-p3-restart_from_same_input (Not Run)
 1627 - scatra_gaussian_hill_pbc.4C.yaml-p2 (Failed)
 1628 - scatra_gaussian_hill_pbc_gena_cons.4C.yaml-p2 (Failed)
 1629 - scatra_gaussian_hill_pbc_genalpha.4C.yaml-p2 (Failed)
 1630 - scatra_gaussian_hill_pbc_genalpha.4C.yaml-p2-restart_from_same_input (Not Run)
 1641 - scatra_gaussian_hill_pbc_reaBioFilm_noStab.4C.yaml-p2 (Failed)
 1642 - scatra_gaussian_hill_pbc_reaBioFilm_THZ.4C.yaml-p2 (Failed)
 1694 - scatra_turbmodel_test_dynsmag.4C.yaml-p3 (Failed)
 1699 - scatra_turbmodel_test_mfs_par.4C.yaml-p3 (Failed)
 1700 - scatra_turbmodel_test_mfs_par.4C.yaml-p3-restart_from_same_input (Not Run)
 1701 - scatra_turbmodel_test_vreman.4C.yaml-p3 (Failed)
 1739 - sohex8_multiscale_macro.4C.yaml-p2 (Failed)
 1740 - sohex8_multiscale_macro.4C.yaml-p2-restart_from_same_input (Not Run)
 1741 - sohex8_multiscale_macro_2micro.4C.yaml-p2 (Failed)
 1742 - sohex8_multiscale_macro_2micro.4C.yaml-p2-restart_from_same_input (Not Run)
 2076 - ssi_1wc_membrane_active_strain_scatrafromrestart_vol_nonmatching.4C.yaml-p2_from_ssi_1wc_membrane_active_strain_scatra_only.4C.yaml-p2-restart_step_0 (Timeout)
 2185 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_AMG_pointbased.4C.yaml-p3 (Failed)
 2187 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_BGS-AMG_2x2.4C.yaml-p3 (Failed)
 2188 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_BGS-AMG_2x2.4C.yaml-p3-restart_from_same_input (Not Run)
 2189 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_condensed_BGS-AMG_2x2.4C.yaml-p3 (Failed)
 2190 - sti_mono_2D_quad4_elch_s2i_butlervolmerpeltier_diabatic_condensed_BGS-AMG_2x2.4C.yaml-p3-restart_from_same_input (Not Run)
 2194 - sti_mono_3D_tet4_elch_s2i_butlervolmerpeltier_adiabatic_AMG-BGS_4x4.4C.yaml-p3 (Failed)
 2286 - tsi_heatflux_monolithic_pbc.4C.yaml-p2 (Failed)
 2287 - tsi_heatflux_monolithic_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 2335 - w1nurbs9_pbc.4C.yaml-p2 (Failed)
 2336 - w1nurbs9_pbc.4C.yaml-p2-restart_from_same_input (Not Run)
 2504 - xfluid_kimmoin_stat_24x24x1_wedge15_levelset.4C.yaml-p3 (Timeout)
 2539 - xfluid_taylor_couette_NavSlip_eps1.0_25x25_krylov_levelset_altgeogeneration.4C.yaml-p2 (Failed)
 1739 - sohex8_multiscale_macro.4C.yaml-p2 (Failed)
 2616 - beam3eb_static_beam_to_solid_volume_meshtying_monolithic_partitioning-vtk-beam.4C.yaml-p3 (Failed)
 2617 - beam3eb_static_beam_to_solid_volume_meshtying_monolithic_partitioning-vtk-structure.4C.yaml-p3 (Failed)
 2620 - f3_decaying_hit_8x8x8.4C.yaml-p2 (Failed)
 2621 - f3_decaying_hit_eb.4C.yaml-p2 (Failed)
 2622 - f3_decaying_hit_eb_revcond.4C.yaml-p2 (Failed)
 2624 - scatra_forced_hit_iso_8x8x8_dsm.4C.yaml-p3 (Failed)
 2625 - scatra_forced_hit_iso_8x8x8_dsm.4C.yaml-p3-restart_from_same_input (Not Run)
 2626 - scatra_forced_hit_mean_8x8x8.4C.yaml-p3 (Failed)
 2627 - scatra_forced_hit_mean_8x8x8.4C.yaml-p3-restart_from_same_input (Not Run)
 2630 - sohex8_multiscale_macro.4C.yaml-nestedPar (Timeout)
 2631 - sohex8_multiscale_macro.4C.yaml-nestedPar-restart (Not Run)
 2632 - sohex8_multiscale_macro_2micro.4C.yaml-nestedPar (Timeout)
 2633 - sohex8_multiscale_macro_2micro.4C.yaml-nestedPar-restart (Not Run)
 2634 - tsi_heatconvection_monolithic.4C.yaml-nestedPar (Failed)
 2745 - sohex8_multiscale_macro-vtk.4C.yaml-p2 (Not Run)
 2771 - unittests_deal_ii.np3 (Failed)
 2772 - unittests_deal_ii (Failed)
 2773 - unittests_deal_ii.np2 (Failed)
 2782 - unittests_mat.np3 (Failed)

@maxfirmbach
Copy link
Contributor Author

@mayrmt Would you mind taking a look at the tests related to fsi and contact? Why they fail and if they are fixable in an easy way?
@isteinbrecher Would you mind taking a look at the beam related tests?
@m-frey Would you mind taking a look at the tests related to micro-macro and nestedPar?

@mayrmt
Copy link
Member

mayrmt commented Apr 3, 2025

@maxfirmbach I'll have a look. Not today, but hopefully tomorrow.

@mayrmt
Copy link
Member

mayrmt commented Apr 4, 2025

@maxfirmbach FSI tests seem to just violate the result tolerance a little bit. Given the changes in subdomains combined with additive Schwarz components in the preconditioner, I am inclined to accept these changes as ok and just adapt the test result or the tolerance accordingly.

Contact tests are a different story and need more investigation.

@mayrmt
Copy link
Member

mayrmt commented Apr 4, 2025

I took care of the FSI and FS3I tests. I have removed them from the list of failing tests above.

@mayrmt mayrmt changed the title Draft: [Trilinos] Move from Isorropia to Zoltan2 [Trilinos] Move from Isorropia to Zoltan2 Apr 8, 2025
@mayrmt
Copy link
Member

mayrmt commented Apr 10, 2025

Contact tests with rebalancing require a closer look, since they fail with a run time error in the graph partitioner.

@m-frey
Copy link
Contributor

m-frey commented Apr 11, 2025

Ill look into the micro-macro and the nestesPar

@maxfirmbach maxfirmbach force-pushed the 927-Move-from-isorropia-to-zoltan2 branch from 1a483fd to ce66a2b Compare April 22, 2025 09:35
@maxfirmbach
Copy link
Contributor Author

maxfirmbach commented Apr 25, 2025

@eulovi Seems like the implementation is correct from a conceptual point of view (even though the results of the partitioning differ). Would you mind working on the output related tests and fix them e.g. vtk? I think you are faster with this than I am ... 😉

@eulovi
Copy link
Contributor

eulovi commented Apr 29, 2025

I fixed the following vtk output tests:

  • beam3eb_static_beam_to_solid_volume_meshtying_beam_to_beam_contact_boundingvolume-vtk-beam-interaction-geometric-search.dat-p2
  • embeddedmesh_parallel_bending_beam_fine-vtk-structure.dat-p3
  • beam3r_herm2line3_static_beam_to_solid_surface_contact_ironing_penalty_gap_variation_segmentation_lin_quad-vtk-averaged-normals.dat-p3
  • beam3r_herm2line3_static_beam_to_solid_surface_contact_mortar_line2_gap_variation_reference_segmentation_penalty_lin_quad-vtk-averaged-normals.dat-p3
  • elch_3D_tet4_s2i_butlervolmer_plating_monolithic_reg_none_vtk.dat-p3
  • membrane_patch_new_struct_vtk.dat-p2
  • poromultielastscatra_2D_quad4_tumor-vtk-porofluid.dat-p3
  • solid_runtime_ghosting_output-vtk-solid.dat-p3
  • solid_runtime_nurbs-vtk-solid.dat-p3
  • solid_runtime_velocity_output-vtk-solid.dat-p2
  • solid_runtime_vtk_2d-vtk.dat-p2
  • solid_runtime_vtk_offset-vtk.dat-p2

For the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 I wasn't sure if I should change it or not as the results of the beams are
different:
beam-results

If this is expected with the current changes, let me know to add the current VTK results.

@maxfirmbach
Copy link
Contributor Author

@eulovi Thanks for fixing some of the vtk related tests! Did you already remove the tests from the list above? Would you mind doing that?

@maxfirmbach
Copy link
Contributor Author

@isteinbrecher Any opinion on:

For the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 I wasn't sure if I should change it or not as the results of the beams are
different

@isteinbrecher
Copy link
Contributor

I think the changes in the test beam3r_herm2line3_static_beam_to_solid_surface_coupling_mortar_reference_forced_line3-vtk-structure-beams.dat-p3 are too large. Do the result test of the "main" simulation pass?

@maxfirmbach
Copy link
Contributor Author

maxfirmbach commented Apr 30, 2025

@isteinbrecher @eulovi The results in the "main" simulation pass, but there's a Epetra ERROR -1, packages/epetra/src/Epetra_CrsMatrix.cpp feature appearing ... so not sure what influence this has (also happens on the current main branch). Every time we skip these errors something wrong is happening, so I assume that also here we do something wrong e.g. accessing a index not appearing in a map or not locally available ... etc.

@isteinbrecher
Copy link
Contributor

Is it possible to identify where this error occurs?

mayrmt and others added 25 commits October 17, 2025 11:20
Tolerances can be loosened, since the solver tolerances are actually a
couple of orders of magnitudes above the result tolerances.
Tolerances can be loosened, since the solver tolerances are actually a
couple of orders of magnitudes above the result tolerances.
These tests target the contact preconditioner, thus their nonlinear
solver tolerance is way to loose. Hence, results are affected by changes
in the subdomain. Slightly loosening the result tolerances makes the
test more robust.

Number of linear iterations have changed as well.
Change in partioning has reduced the number of linear iterations.
@maxfirmbach maxfirmbach force-pushed the 927-Move-from-isorropia-to-zoltan2 branch from 6285ebd to 25bdbb6 Compare October 17, 2025 11:03
@maxfirmbach
Copy link
Contributor Author

Here is a list of the remaining failing tests:

570 - elch_3D_tet4_s2i_butlervolmer_AMG_pointbased.4C.yaml-p3 (Failed)
571 - elch_3D_tet4_s2i_butlervolmer_AMG_vectorbased.4C.yaml-p3 (Failed)
1537 - porofluid_pressure_based_elast_scatra_2D_quad4_nodetopoint_artery_airway_coupling_volfrac_closingrelation_lung_mono.4C.yaml-p3 (Failed)
1538 - porofluid_pressure_based_elast_scatra_2D_quad4_nodetopoint_artery_airway_coupling_volfrac_closingrelation_lung_mono.4C.yaml-p3-restart_from_same_input (Not Run)
2112 - ssi_1wc_membrane_active_strain_scatrafromrestart_vol_nonmatching.4C.yaml-p2_from_ssi_1wc_membrane_active_strain_scatra_only.4C.yaml-p2-restart_step_0 (Failed)
1537 - porofluid_pressure_based_elast_scatra_2D_quad4_nodetopoint_artery_airway_coupling_volfrac_closingrelation_lung_mono.4C.yaml-p3 (Failed)
2780 - porofluid_pressure_based_elast_scatra_2D_quad4_nodetopoint_artery_airway_coupling_volfrac_closingrelation_lung_mono-vtk-porofluid.4C.yaml-p3 (Not Run)

@maxfirmbach
Copy link
Contributor Author

maxfirmbach commented Oct 18, 2025

Something in the porofluid tests is going wrong. They run through ... but the results are showing numbers that hint at some out-of-bound memory access or similar.

@maxfirmbach
Copy link
Contributor Author

The ssi test is interesting. The restart fails, because it reads in a file that was written on a different partitioning of the structure part, means maps and local lengths don't fit. This test only ran through before, due to matching partitionings (so I assume that was kind of lucky).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: in progress Issues that are currently being worked on taskforce: tpetra Issues related to the migration from Epetra to Tpetra team: discretization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants