Slow analysis

[jonaskohh]

Hello

I am currently creating a tree with 22996 ASVs (short sequences, 16S) and I am trying to perform 1000 bootstraps. I am also doing them in parallel with 10 different job scripts as I am running them on HPC (according to FAQ in http://www.iqtree.org/doc/Frequently-Asked-Questions). However, because it uses only 1 thread (set to auto), the construction of the tree is taking a very long time. Is there a way where I can speed things up, or use more than 1 thread? I have attached 1 log file as an example, I can upload more if needed.

Best regards,
Jonas
BOOT1.log

[thomaskf]

Hi Jonas, I saw you are running on a machine with 80 CPU cores. You may specify the number of CPUs instead of using “AUTO” by using the option “-nt [number of threads]”. For example, if you want to use 10 CPU threads, then your command will be: iqtree -s /gpfs1/scratch/jonas/tree_mafft_iqtree/mafft_allsamples_alignment.fasta -m SYM+I+R10 --bonly 100 -redo -nt 10 --prefix BOOT1 Thomas From: jonaskohh ***@***.***> Date: Tuesday, 7 May 2024 at 4:37 PM To: iqtree/iqtree2 ***@***.***> Cc: Subscribed ***@***.***> Subject: [iqtree/iqtree2] Slow analysis (Discussion #190) Hello I am currently creating a tree with 22996 ASVs (short sequences, 16S) and I am trying to perform 1000 bootstraps. I am also doing them in parallel with 10 different job scripts as I am running them on HPC (according to FAQ in http://www.iqtree.org/doc/Frequently-Asked-Questions). However, because it uses only 1 thread (set to auto), the construction of the tree is taking a very long time. Is there a way where I can speed things up, or use more than 1 thread? I have attached 1 log file as an example, I can upload more if needed. Best regards, Jonas BOOT1.log<https://github.com/iqtree/iqtree2/files/15231317/BOOT1.log> — Reply to this email directly, view it on GitHub<#190>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AENQCRRSYNJIMDC2YDT3XNLZBBZBFAVCNFSM6AAAAABHKK3GR6VHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWGYZDCNBTHA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jonaskohh]

Previously I did put a thread number but it says it will be too much for short alignment; can I still proceed to do that? Thank you!

[thomaskf]

I see. There only 396 columns in the alignment. Thus, you should use only 1 thread, as too much for short alignment will slow down the process. From: jonaskohh ***@***.***> Date: Tuesday, 7 May 2024 at 5:22 PM To: iqtree/iqtree2 ***@***.***> Cc: Thomas Wong ***@***.***>, Comment ***@***.***> Subject: Re: [iqtree/iqtree2] Slow analysis (Discussion #190) Previously I did put a thread number but it says it will be too much for short alignment; can I still proceed to do that? Thank you! — Reply to this email directly, view it on GitHub<#190 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AENQCRVHHAFSGMI4IWYUBODZBB6LVAVCNFSM6AAAAABHKK3GR6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMZXHEYTM>. You are receiving this because you commented.Message ID: ***@***.***>

[roblanf]

@jonaskohh, I think 22996 tips is likely too much for IQ-TREE to ever be really fast, particularly if you want a lot of bootstraps.

Here are a few things to consider, which might (or might not, depending on the context) help:

1. fasttree2: very good pseudo-maximum likelihood, works well on datasets up to about 100K tips, I suspect this is the best solution for you (and if you like you can always get the trees from fasttree2, then re-optimise branch lengths and model parameters in IQ-TREE2, to get a slightly better idea of the likelihood)
2. RAxML-NG: not sure how well it will scale to 23K tips, but it might be worth a try. I suspect it will be similar to IQ-TREE, but you never know!
3. DecentTree: very fast neighbour joining (https://academic.oup.com/bioinformatics/article/39/9/btad536/7257068)
4. matOptimize: parsimony, exceptionally fast but you should only use it if parsimony would be a good method for what you're doing (e.g. it's great for SARS-CoV-2: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9128781/)
5. MAPLE/CMAPLE: parsimony/likelihood hybrid. Similar to matOptimize in that you should only use it on (very) closely related sequences

Hope some of that helps...

[jonaskohh]

Thank you so much for this! This information will definitely help! :)