Unexpected result from ModelFinder in IQTREE v 2.3.4

[lsjermiin]

Hi,
I'm upgrading to the latest version of IQTREE 2, and therefore testing whether the code works as expected. In this regard, I ran the following job (on my Mac, using the COVID-edition for MacOS ARM 64-bit built May 11 2024):

iqtree2 -s Norovirus_PRO_einsi.fst -T AUTO --seqtype AA -m MF -cmax 15 --merit BIC --safe

The .log file included the following (included simply to show you some features of the data):

Alignment has 20 sequences with 3083 columns, 2180 distinct patterns
1651 parsimony-informative, 418 singleton sites, 1014 constant sites

The .log file also included the following lines:

No. Model -LnL df AIC AICc BIC
1 LG 56111.512 37 112297.024 112297.947 112520.269
2 LG+I 54824.426 38 109724.853 109725.826 109954.132
3 LG+G4 53563.273 38 107202.545 107203.519 107431.824
4 LG+I+G4 53543.629 39 107165.257 107166.283 107400.570
5 LG+R2 53897.546 39 107873.091 107874.117 108108.404
6 LG+R3 53571.700 41 107225.400 107226.532 107472.780
7 LG+R4 53500.052 43 107086.103 107087.348 107345.550
8 LG+R5 53494.707 45 107079.415 107080.778 107350.930
19 LG+I+R2 53648.279 40 107376.559 107377.637 107617.905
20 LG+I+R3 53509.295 42 107102.590 107103.778 107356.004
21 LG+I+R4 53495.975 44 107079.950 107081.253 107345.431
22 LG+I+R5 53495.012 46 107082.024 107083.448 107359.572
39 LG+F+R4 53304.288 62 106732.576 106735.163 107106.663
40 LG+F+R5 53298.983 64 106725.966 106728.723 107112.120
53 LG+F+I+R4 53299.991 63 106725.983 106728.654 107106.103
71 WAG+R4 53774.458 43 107634.916 107636.161 107894.364
72 WAG+R5 53772.955 45 107635.910 107637.273 107907.424
85 WAG+I+R4 53772.853 44 107633.705 107635.009 107899.186
103 WAG+F+R4 53678.880 62 107481.761 107484.348 107855.848
104 WAG+F+R5 53677.300 64 107482.599 107485.356 107868.753
117 WAG+F+I+R4 53677.008 63 107480.016 107482.687 107860.137
135 JTT+R4 53939.965 43 107965.931 107967.176 108225.378
136 JTT+R5 53935.544 45 107961.087 107962.450 108232.602
149 JTT+I+R4 53936.045 44 107960.090 107961.394 108225.571
167 JTT+F+R4 53938.387 62 108000.773 108003.360 108374.860
168 JTT+F+R5 53933.377 64 107994.755 107997.511 108380.909
181 JTT+F+I+R4 53933.944 63 107993.888 107996.559 108374.008

And now the questions:

1. Why are the following models not considered: LG+F, LG+F+I, LG+F+G4, LG+F+I+G4, LG+F+R2, LG+F+R3, LG+F+R4, LG+F+R5, LG+F+I+R2, LG+F+I+R3? It seems like the model-selection procedure jumps straight from No. 22 to No. 39, which is LG+F+I+R4.
2. Likewise, it appears that the model-selection procedure skips over a lot of models involving WAG and JTT. Is this intended or have I uncovered a bug?

As I recall it, the original implementation of ModelFinder included these other models.

All the best,

Lars

[roblanf]

Hi lars,

Could you post the full log file, and the other output files too?

Rob

[roblanf]

Digging a bit further back, the behaviour has been very similar since v2.1.0. I think I see what is happening:

The original intention was something like:

1. ModelFinder first uses LG (or a user-specified model) to estimate the rate distribution from all combinations

2. It then tests this with and without +F for every other model.

However, in your case it's clearly not quite doing this (and neither does it quite do this on my test case either).

@thomaskf @bqminh and I can meet and get this figured out. Thanks for highlighting it.

[roblanf]

and for the sake of completeness, just confirming that v.1.6.3 does do all combinations. E.g. on my test alignment it starts:

(noting that there are still 'jumps' in the numbers, but this is just it skipping overparameterised +R models)

Create initial parsimony tree by phylogenetic likelihood library (PLL)... 0.023 seconds
NOTE: ModelFinder requires 170 MB RAM!
ModelFinder will test 756 protein models (sample size: 5018) ...
 No. Model         -LnL         df  AIC          AICc         BIC
  1  Dayhoff       33120.221    91  66422.443    66425.842    67015.834
  2  Dayhoff+I     31798.056    92  63780.113    63783.587    64380.025
  3  Dayhoff+G4    31690.884    92  63565.769    63569.243    64165.681
  4  Dayhoff+I+G4  31555.752    93  63297.505    63301.056    63903.938
  5  Dayhoff+R2    31579.939    93  63345.878    63349.429    63952.312
  6  Dayhoff+R3    31472.789    95  63135.578    63139.284    63755.052
  7  Dayhoff+R4    31472.233    97  63138.467    63142.331    63770.983
 19  Dayhoff+F     32878.699    110 65977.399    65982.375    66694.685
 20  Dayhoff+F+I   31553.596    111 63329.191    63334.260    64052.999
 21  Dayhoff+F+G4  31448.224    111 63118.447    63123.515    63842.255
 22  Dayhoff+F+I+G4 31313.577    112 62851.154    62856.315    63581.482
 23  Dayhoff+F+R2  31335.273    112 62894.545    62899.706    63624.874
 24  Dayhoff+F+R3  31232.388    114 62692.776    62698.124    63436.146
 25  Dayhoff+F+R4  31231.968    116 62695.937    62701.475    63452.348
 37  mtMAM         33952.259    91  68086.518    68089.917    68679.909
 38  mtMAM+I       32626.276    92  65436.552    65440.026    66036.464
 39  mtMAM+G4      32359.961    92  64903.922    64907.397    65503.835
 40  mtMAM+I+G4    32226.531    93  64639.062    64642.612    65245.495
 41  mtMAM+R2      32294.768    93  64775.536    64779.087    65381.969
 42  mtMAM+R3      32152.293    95  64494.585    64498.291    65114.060
 43  mtMAM+R4      32151.236    97  64496.472    64500.336    65128.988
 55  mtMAM+F       32769.800    110 65759.600    65764.577    66476.887
 56  mtMAM+F+I     31437.353    111 63096.706    63101.775    63820.514
 57  mtMAM+F+G4    31272.482    111 62766.963    62772.031    63490.770
 58  mtMAM+F+I+G4  31144.100    112 62512.200    62517.360    63242.528
 59  mtMAM+F+R2    31188.354    112 62600.708    62605.869    63331.036
 60  mtMAM+F+R3    31062.464    114 62352.927    62358.275    63096.297
 61  mtMAM+F+R4    31062.342    116 62356.685    62362.223    63113.096

[roblanf]

@lsjermiin - we chatted and I can confirm that what is happening here is a (poorly documented) approach designed to speed things up with minimal loss of rigour, roughly described above.

In the short term, we plan to implement an option -m MF1 to mimic the behaviour of ModelFinder1 for those who want it.

I'll then benchmark the various new speedup options we've designed, so we can choose good defaults to improve the speed/rigour tradeoff, but of course leaving users the option to set things as they wish

[lsjermiin]

Hi again,

Thanks for the replies. I discovered this 'odd' behaviour several years ago and reported it (probably via email rather than through GitHub).

I have attached the log file, as requested.

Norovirus_PRO_einsi.fst.log

You mentioned "skipping overparameterised +R models" in the output above. I don't see any skipping -- I see R2, R3, and R4, with R4 being worse than R3 (which should cause MF to abandon trying R5).

Implementing MF1 is a good intermediate solution.

In the meantime, I'm using the following command to retain control over the model selection.

iqtree2 -s -T AUTO --seqtype AA -m MF -mfreq FU,FO,F -mrate E,I,G,*G,I+G,R,*R,I+R,H,*H -cmax 15 --merit BIC --safe

All the best,

Lars

[roblanf]

re. skipping the overparameterised models, I was referring to IQ-TREE not testing R6-R15 (you set cmax to 15), because R5 is already worse than R4 (R4 is better than R3 according to the BIC on the LG models).

[lsjermiin]

Cool – understood. From: roblanf ***@***.***> Date: Wednesday, 15 May 2024 at 03:18 To: iqtree/iqtree2 ***@***.***> Cc: Jermiin, Lars ***@***.***>, Mention ***@***.***> Subject: Re: [iqtree/iqtree2] Unexpected result from ModelFinder in IQTREE v 2.3.4 (Discussion #193) EXTERNAL EMAIL: This email originated outside the University of Galway. Do not open attachments or click on links unless you believe the content is safe. RÍOMHPHOST SEACHTRACH: Níor tháinig an ríomhphost seo ó Ollscoil na Gaillimhe. Ná hoscail ceangaltáin agus ná cliceáil ar naisc mura gcreideann tú go bhfuil an t-ábhar sábháilte. re. skipping the overparameterised models, I was referring to IQ-TREE not testing R6-R15 (you set cmax to 15), because R5 is already worse than R4 (R4 is better than R3 according to the BIC on the LG models). — Reply to this email directly, view it on GitHub<#193 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AG6JXEHLDLQJJB44RCVBBPDZCLAXLAVCNFSM6AAAAABHSAFV36VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TINBQGA4TK>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lsjermiin]

Hi again,

Regarding making ModelFinder faster, here are some observations that highlight were easy gains can be made:

Norovirus_PRO_einsi.fst.log

Testing

-m MF -mset LG -mfreq FO,FU,F -mrate E,I,G,G,I+G,IG,R,R,I+R,IR,H,H,I+H,IH

Reasons

• To identify unsupported/invalid options of -mrate
• To identify redundant combinations of -mfreq and -mrate

Observations

1. When using -mfreq FO, none of the RHAS models are the same, which is good, and all the RHAS model appear to work, which is even better.
2. When using -mfreq FU, some models are simply labelled as LG. For the sake of clarity and consistency, it might be better to label them LG+FU (like LG+FO and LG+F).
3. When using -mfreq FU, some RHAS models are the same (i.e., +G4 = G4; +I+G4 = +IG4; +R = R; +I+R = +IR; +H = H; +I+H = +IH) because FU is defined by the LG model), so some models need not be tested. In this example, 17 of the 36 models tested need not have been tested.
4. When using -mfreq F, some RHAS models are the same (i.e., +G4 = G4; +I+G4 = +IG4; +R = R; +I+R = +IR; +H = H; +I+H = +IH) because F is 'counted from the data'), so some models need not be tested. In this example, 17 of the 36 models tested need not have been tested.

Note

One type of warning (see below) could be stated differently making it clearer to users.

Question

Given -mfreq FO, the following appears to be acceptable options for -mrate: E,I,G,G,I+G,IG,R,R,I+R,IR,H,H,I+H,IH. Can these now be assumed a being properly tested or should users stay clear of some of them?

[lsjermiin]

Oups -- the formatting of -mrate models is stuffed up (by the use of asterisk). Check the attached file to see what I mean.

All the best,

Lars
ModelFinder_testing.txt