Issue reading h5 files

[pimentel]

Some users have reported having issues reading the H5 files.

Here is the error:

> so <- sleuth_prep(s2c, ~ condition)
reading in kallisto results
..Error in H5Fopen(file, "H5F_ACC_RDONLY") : 
  HDF5. File accessability. Unable to open file.

I would like to track this down so if you are having this issue please respond with the following:

• Your sessionInfo() in R
• Your operating system
• Your version of gcc: gcc --version

And any other information you think might be informative.

Thanks,

Harold

[rachelzoeb]

Hi Harold,

I am getting this error as well.

sessionInfo()
R version 3.3.0 (2016-05-03)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.12.5 (unknown)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] bindrcpp_0.1 sleuth_0.29.0 dplyr_0.7.0 ggplot2_2.2.1

loaded via a namespace (and not attached):
[1] Rcpp_0.12.11 tidyr_0.6.3 assertthat_0.2.0 grid_3.3.0 plyr_1.8.4 R6_2.2.1 gtable_0.2.0 magrittr_1.5 scales_0.4.1 zlibbioc_1.16.0
[11] rlang_0.1.1 lazyeval_0.2.0 data.table_1.10.4 tools_3.3.0 glue_1.0.0 munsell_0.4.3 parallel_3.3.0 rhdf5_2.14.0 pkgconfig_2.0.1 colorspace_1.3-2
[21] bindr_0.1 tibble_1.3.3

Operating system: macOS Sierra version 10.12.5
RStudio Version 1.0.136
gcc version Apple LLVM version 8.1.0 (clang-802.0.42)

Hope this helps!

Rachel

[jmcribeiro]

Hi Harold and Rachel,

Same problem here.

so <- sleuth_prep(s2c, full_model = full_design)
reading in kallisto results
dropping unused factor levels
........................................................................
normalizing est_counts
72457 targets passed the filter
normalizing tpm
merging in metadata
Error in H5Fopen(file, "H5F_ACC_RDONLY") :
HDF5. File accessability. Unable to open file.

#########################################################

R version 3.3.2 (2016-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2008 R2 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] splines stats graphics grDevices utils datasets methods base

other attached packages:
[1] sleuth_0.29.0 dplyr_0.5.0 ggplot2_2.2.1 BiocInstaller_1.24.0

loaded via a namespace (and not attached):
[1] Rcpp_0.12.11 magrittr_1.5 zlibbioc_1.20.0 devtools_1.13.2
[5] munsell_0.4.3 colorspace_1.3-2 R6_2.2.1 rlang_0.1.1
[9] httr_1.2.1 plyr_1.8.4 tools_3.3.2 parallel_3.3.2
[13] grid_3.3.2 rhdf5_2.18.0 data.table_1.10.4 gtable_0.2.0
[17] DBI_0.6-1 git2r_0.18.0 withr_1.0.2 lazyeval_0.2.0
[21] digest_0.6.12 assertthat_0.2.0 tibble_1.3.3 tidyr_0.6.3
[25] curl_2.6 memoise_1.1.0 scales_0.4.1

#############################################################
I tried another drive using setwd() withouth success
I also tried

[jmcribeiro]

...continuing...

options(max.print=10000000)

also without success.

Regards,

Jose

[jmcribeiro]

Hi,

I run the same script on a linux machine. This time I got errors/warnings, including:

1: In read_kallisto(path, read_bootstrap = TRUE, max_bootstrap = max_bootstrap) :
You specified to read bootstraps, but we won't do so for plaintext

Indeed I have run kallisto with the --plain-text option

Now I am re-running kallisto withouth the option, and we will see what happens.

Perhaps the R versions of sleuth on Mac and Windows are not reporting the errors/warnings above.

Regards,

Jose

[jmcribeiro]

Hi,

I rerun kallisto without the --plain-text option.

Now the .h5 files were created in the expected subdirectories, it was not there before.

when running the command

so <- sleuth_prep(s2c, full_model = full_design)

on a Windows machine I now get

reading in kallisto results
dropping unused factor levels
........................................................................
normalizing est_counts
72457 targets passed the filter
normalizing tpm
merging in metadata
summarizing bootstraps
Error in parallel::mclapply(x, y, mc.cores = num_cores) :
'mc.cores' > 1 is not supported on Windows
###################################################

On looking the documentation on sleuth_prep at

https://pachterlab.github.io/sleuth/docs/sleuth_prep.html

SUGGESTION 1:
I cannot find an option to limit the script to a single core, so I suggest that that is either included as a switch on sleuth_prep or that a new version of the function takes into account the environment and indicates how many cpu's to use.

SUGGESTION 2:

On

https://pachterlab.github.io/kallisto/manual

The text

Optional arguments:
--bias Perform sequence based bias correction
-b, --bootstrap-samples=INT Number of bootstrap samples (default: 0)
--seed=INT Seed for the bootstrap sampling (default: 42)
--plaintext Output plaintext instead of HDF5
--fusion Search for fusions for Pizzly

could be changed to (ADDED TEXT IN BOLD).

Optional arguments:
--bias Perform sequence based bias correction
-b, --bootstrap-samples=INT Number of bootstrap samples (default: 0)
--seed=INT Seed for the bootstrap sampling (default: 42)
--plaintext Output plaintext instead of HDF5 (NOT COMPATIBLE WITH SLEUTH)
--fusion Search for fusions for Pizzly

On the other hand, running the same script on a Linux machine after rerunning kallisto I got no error messages! Bingo!

so <- sleuth_prep(s2c, full_model = full_design)
reading in kallisto results
........................................................................
normalizing est_counts
72457 targets passed the filter
normalizing tpm
merging in metadata
normalizing bootstrap samples
summarizing bootstraps

Regards,

Jose

[warrenmcg]

@jmcribeiro, it appears the documentation on the website is not up-to-date with the current version, as it is run separately, so it doesn't have the new options. If you go into R and do ?sleuth_prep you'll see the most up-to-date documentation.

The option you want for sleuth_prep is num_cores. So so <- sleuth_prep(s2c, full_model = full_design, num_cores = 1).

[jmcribeiro]

Hi Warren,

Thanks for your comment. your recommendation worked! Thanks.

Please see my recommendation to make sure R in windows flags the plain-text flag as well to avoid other users getting lost.

Regards,

Jose

[warrenmcg]

Hello!

Those are two great suggestions.

For the Windows issue, we can set a quick patch to warn users that Windows does not support mclapply and switch num_cores to 1. Moving forward, we can explore switching to using the future package, which would allow Windows users to operate multiple cores too.

For the text files issue, I wonder if this is the reason most people are having issues? I think it would make sense for sleuth_prep to check for abundance.tsv files if the abundance.h5 is absent, and use the appropriate read method.

What do you think @pimentel of these two options?

[rachelzoeb]

I have rerun kallisto and removed the --plain-text flag which removed the h5 error. However, now I get this error:

.Error in process_bootstrap(i, samp_name, kal_path, num_transcripts, est_counts_sf[[i]], :
File h5 has no bootstraps.Please generate bootstraps using "kallisto quant -b".File ../analysis/data/kallisto/Deer_R1_S22/abundance.h5 has no bootstraps.Please generate bootstraps using "kallisto quant -b".

Any help is greatly appreciated,

Rachel

[warrenmcg]

Hi @rachelzoeb,

What was your full kallisto command, and what version of kallisto did you use?

It seems that you did not use the -b option when running kallisto, which is a requirement to take full advantage of sleuth.

If you did use the -b option with this error, maybe there is a something wrong with how sleuth is interacting with your particular version of kallisto.

If you are using the latest version of kallisto, then it would be helpful if you gave your OS and version of gcc (use gcc --version) as Harold suggested above, and emailed your abundance.h5 file to him or posted it here for me and other users to look at to help you out.

[pimentel]

@warrenmcg thanks so much for fielding these questions.

regarding the windows patch: that sounds like a great idea

Unfortunately the bootstraps are not available via plaintext at all. This is because H5 provides nice compression that is a bit of a pain to get otherwise. Initially, plaintext abundance.tsv was only intended for quick sanity checks. However, we have been discussing changing the format to remove the dependency on H5 which has proven to be an issue for some time now...

More on this soon.

[xindiguo]

Hi,

I ran kallisto with quant --bootstrap-samples=100 --threads=16 and 4 out of 8 of my h5 files had the can-not-open error. My kallisto ran on a linux server and then I downloaded the h5 files to my local machine (mac OS) to run sleuth in R. Do you think there might be an error during the file transfer? Also, I have checked the file size of the error h5 and for 3 out of 4 files have the error, the h5 file size is smaller than the tsv file. Not sure it is related. Thanks in advance for any help!

sessionInfo() in R -

> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rhdf5_2.20.0         BiocInstaller_1.26.0 bindrcpp_0.2         synapseClient_1.15-0 sleuth_0.29.0       
[6] dplyr_0.7.2          ggplot2_2.2.1        biomaRt_2.32.1      

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.12         compiler_3.4.1       plyr_1.8.4           bindr_0.1            zlibbioc_1.22.0     
 [6] bitops_1.0-6         tools_3.4.1          digest_0.6.12        bit_1.1-12           RSQLite_2.0         
[11] memoise_1.1.0        tibble_1.3.4         gtable_0.2.0         pkgconfig_2.0.1      rlang_0.1.2         
[16] DBI_0.7              parallel_3.4.1       stringr_1.2.0        S4Vectors_0.14.3     IRanges_2.10.2      
[21] stats4_3.4.1         bit64_0.9-7          grid_3.4.1           glue_1.1.1           Biobase_2.36.2      
[26] data.table_1.10.4    R6_2.2.2             AnnotationDbi_1.38.2 XML_3.98-1.9         tidyr_0.7.0         
[31] reshape2_1.4.2       blob_1.1.0           magrittr_1.5         matrixStats_0.52.2   scales_0.5.0        
[36] BiocGenerics_0.22.0  assertthat_0.2.0     colorspace_1.3-2     stringi_1.1.5        RCurl_1.95-4.8      
[41] lazyeval_0.2.0       munsell_0.4.3        rjson_0.2.15        
> 

OS -
kallisto was ran on 3.2.0-29-generic GNU/Linux
sleuth was ran in R on macOS Sierra version 10.12.5

gcc -

$ gcc --version
Apple LLVM version 8.1.0 (clang-802.0.42)

[miguelroboso]

I am currently having this issue.

I have a data frame built as it is in the walkthrough, and it looks like this:

    sample condition           path
 1:     P1        ns  expression/P1
 2:     P2        ns  expression/P2
 3:     P3         s  expression/P3
 4:     P4         s  expression/P4
 5:     P5        ns  expression/P5
 6:     P6        ns  expression/P6
 7:     P7         s  expression/P7
 8:     P8         s  expression/P8
 9:     P9        ns  expression/P9
10:    P10        ns expression/P10
11:    P11         s expression/P11
12:    P12         s expression/P12

>sessionInfo() 
R version 3.3.1 (2016-06-21) 
Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS release 6.8 (Final)  locale: 
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8         
[4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8     
[7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C         

attached base packages:
 [1] stats     graphics  grDevices utils     datasets  methods   base       

other attached packages: 
[1] bindrcpp_0.2        sleuth_0.29.0       dplyr_0.7.4         ggplot2_2.2.1       edgeR_3.16.5       
 [6] biomaRt_2.30.0      limma_3.30.13       data.table_1.10.4-3  

loaded via a namespace (and not attached):  
[1] locfit_1.5-9.1       tidyselect_0.2.2     purrr_0.2.4          lattice_0.20-34      
[5] rhdf5_2.18.0         colorspace_1.3-2     htmltools_0.3.6      stats4_3.3.1          
[9] viridisLite_0.2.0    yaml_2.1.14          base64enc_0.1-3      XML_3.98-1.7         [13] plotly_4.7.1         rlang_0.1.2          glue_1.1.1           DBI_0.6-1            
[17] BiocGenerics_0.20.0  bindr_0.1            plyr_1.8.4           stringr_1.2.0        [21] zlibbioc_1.20.0      munsell_0.4.3        gtable_0.2.0         htmlwidgets_0.9     
 [25] memoise_1.1.0        evaluate_0.10        Biobase_2.34.0       knitr_1.15.1         
[29] IRanges_2.8.2        parallel_3.3.1       AnnotationDbi_1.36.2 Rcpp_0.12.13         
[33] scales_0.5.0         backports_1.1.0      S4Vectors_0.12.2     jsonlite_1.5        
 [37] digest_0.6.12        stringi_1.1.5        grid_3.3.1           rprojroot_1.2        [41] tools_3.3.1          bitops_1.0-6         magrittr_1.5         lazyeval_0.2.0      
 [45] RCurl_1.95-4.8       tibble_1.3.4         RSQLite_1.1-2        tidyr_0.7.2          [49] pkgconfig_2.0.1      assertthat_0.1       rmarkdown_1.6        httr_1.2.1           
[53] R6_2.2.2

OS is CENTOS, 2.6.32-696.10.2.el6.x86_64

bash-4.1$ gcc --version
gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6)
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

However, I don't know why it is trying to read H5 files.
In the expression directories I only have tsv files (ran kallisto with --plain-text output)

Lastly, the error is:

It appears that you are running Sleuth from within Rstudio.
Because of concerns with forking processes from a GUI, 'num_cores' is being set to 1.
If you wish to take advantage of multiple cores, please consider running sleuth from the command line.reading in kallisto results
dropping unused factor levels
............
normalizing est_counts
59202 targets passed the filter
normalizing tpm
merging in metadata
Error in H5Fopen(file, "H5F_ACC_RDONLY") : 
  HDF5. File accessability. Unable to open file.