Issue with "Infinity in DO_CLOUD_CONVECTION"

[ast222340]

Your name

PrincDeep

Your affiliation

IIT

What happened? What did you expect to happen?

Dear team,
I want to run a CO2 simulation over India at a resolution of 0.125 × 0.15625 using GEOS-FP meteorological data for the period 2019 to 2021. For this purpose, boundary condition files are needed from a global model run, either at 2×2.5 or 4×5 resolution. Since I am working with a fine resolution, I chose the 2×2.5 global run.
However, when using GEOS-FP, I encountered an error: "Infinity in DO_CLOUD_CONVECTION". To better understand the cause of this issue, I reran the simulation using MERRA-2 meteorological data instead. With MERRA-2, the model ran successfully without any errors, and output files were generated.

What are the steps to reproduce the bug?

This type of error indicates that variables such as T0, T1, T2, T3, or T4 are zero. I recompiled and reran the model with print statements added, but I was unable to identify any solution or reach a conclusion.

IF ( .not. IT_IS_FINITE( Q(K) ) ) THEN
                   WRITE( 6, 200 )
200                FORMAT( 'Infinity in DO_CLOUD_CONVECTION!' )
                   WRITE( 6, 255 ) K, IC, Q(K), TRIM(SpcInfo%Name)
                   WRITE(*,*) T0,T1,T2,T3,T4,DELQ,SDT,TSUM
                   RC = GC_FAILURE
                   RETURN
                ENDIF

Please attach any relevant configuration and log files.

I have attached all the necessary files. The model runs successfully with MERRA-2, but it encounters an error when using GEOS-FP. Could you please assist me with identifying the issue and guide me on the next steps?

GC_GEOSFP.log.txt
geoschem_config.yml.txt
HEMCO_Config.rc.txt
HISTORY.rc.txt
GC_MERRA2.log.txt

What GEOS-Chem version were you using?

14.6.2

What environment were you running GEOS-Chem on?

Local cluster

What compiler and version were you using?

gcc (GCC) 6.5.0 ,ifort (IFORT) 19.1.3.304 20200925

Will you be addressing this bug yourself?

Yes, but I will need some help

In what configuration were you running GEOS-Chem?

GCClassic

What simulation were you running?

CO2

As what resolution were you running GEOS-Chem?

2 x 2.5

What meterology fields did you use?

GEOS-FP

Additional information

I think the error might be related to the GEOS-FP meteorological files, because when I switch to MERRA2, the model runs without any issues. It could possibly be due to the "upward moist convective precipitation flux" (CMFMC), though I'm not entirely sure.
As we already know, there were some known issues starting in June 2020, which led to a change in the meteorological scheme from RAS to GF convection. To the best of my knowledge, those issues were fully resolved.
However, in my case, the model error is appearing for 2020/05/13, which is before the known issue time.

[yantosca]

Thanks for writing @ast222340. The G-F convection fix for GEOS-FP dates after June 1, 2020 was not implemented into 14.6.2, as we had to rush that version to include a fix for the Integrated Methane Inversion version 2.2.0. It is slated to go into version 14.6.3. You can try manually implementing the G-F convection fix from PR #2843 in the meantime in your own code if you wish to still use GEOS-FP meteorology. You could also just stick with MERRA-2 meteorology, which does not have the convection issue.

Also I noticed one of the compilers you mentioned was gcc 6.5.0. The older versions of GCC are known to cause some issues with the current GEOS-Chem version. We recommend using GCC 9.5 or later if possible. Intel 19 should be OK.

[ast222340]

Thank you @yantosca. As per your instructions, I have modified the code to use G-F convection. However, this change is applicable only for simulations after June 1, 2020. When I run the model for dates before that, the same error still persists. I'm not sure what the exact issue is. I have attached the updated log file for your reference.

Sure, we will try to use that version as soon as possible.

GC.log.txt

[yantosca]

Thanks @ast222340. Have you determined if the error always occurs at the same date/time in May 2020? One possibility is that one or more met field files are corrupted.

There's a couple more things we can try.

• Build the executable with -DCMAKE_BUILD_TYPE=Debug, which will turn on some extra debugging flags. This will make the model run more slowly, but the CO2 simulation isn't that computationally expensive. Start from May 1, 2020 and see if it flags any runtime errors like array out-of-bounds.
• Also try turning off operations (transport, mixing, convection) one at a time, to see if you can get past the error. That will tell you in which operation the error is occurring.
• You can also try to add print statements before & after convection to print the sum of CO2. If CO2 is getting removed from the model so that there are negative species concentrations that could be causing the error.

[ast222340]

Thank @yantosca. The error persists at the same date and time. I think the issue is related to the GEOS-FP meteorology files. I observed that the error occurs at a specific date, and when I checked the corresponding meteorological files, I noticed that the file size for that date and for previous date is unusually small (only a few kb).
In particular, most of the GEOS-FP meteorology files after May 16, 2020 appear to have significantly reduced file sizes.

[yantosca]

Thanks for the update @ast222340. Yes, corrupted met fields will definitely result in this type of error.

@yuyao-cyber: Would you be able to take a look at these fields and see if they need to be re-generated?

[yuyao-cyber]

Hi @ast222340, Thank you for flagging the issue. I’ve recently implemented an automation to detect and reprocess corrupted files. I’ll prioritize reprocessing the GEOS-FP data for May 2020 and notify you as soon as the corrected files are available.

[yantosca]

Thanks @yuyao-cyber! Thanks to your automation we should now have many fewer instances of these corrupted met files going forward.

[ast222340]

Thanks @yantosca @yuyao-cyber.
Please notify me when all files are regenerated.