Attempt using plyxp for some cases in tidySummarizedExperiment

[mikelove]

Is there a case to try this, where we can guess from the enquos() arguments how to employ plyxp under the hood?

Maybe a simple mutate call? Are there any cases that can be split out from this logic:

https://github.com/stemangiola/tidySummarizedExperiment/blob/4b8a4e1bdba6230abe00fc2fb0e99eaffeca2532/R/utilities.R#L355

?

[stemangiola]

Showcasing filter is a good start. The guessing logic is already there.

For mutate would be good, but the guessing logic is NOT there at the moment, but it could be easily transferred from filter

[mikelove]

I'll try to come up with some examples so we can do benchmarking and see where any effort is most worthwhile.

[mikelove]

Here's somewhere to start. If we can detect a new assay is being added, and we can see that we need something from rowData or colData, we can prefix these with .rows$ and .cols$ and send into a plyxp pipe and back out

https://gist.github.com/mikelove/5f388e1af24453010dbc619a78854b42

 > system.time({
 +   new_se <- se |>
 +     mutate(newdata = data * x)
 + })
 +    user  system elapsed
   2.940   0.472   3.503

 > system.time({
 +   new_se2 <- se |>
 +     new_plyxp() |>
 +     mutate(newdata = data * .cols$x) |>
 +     se()
 + })
 +    user  system elapsed
   0.030   0.014   0.045

[mikelove]

If you knew all the names in each space (assays, colData, rowData) are unique, it would be easy to simply map using substitute from x to .cols$x. So if we were just handling updates to assays, this wouldn't be too hard.

Haven't figured out yet how the logic would work if user wants to update something other than assays. If we assume nrows != ncols we could look at the dimensions of the return value. Anyway, need to discuss this more.

[stemangiola]

There are routines in tidySE for linking the query to

• colData
• rowData
• assay
• mixed

I could make them more structured.

Basically, link routine + plyxp and it's done.

Of course, as you mention, there are edge cases where column names are duplicated in row and col data.

We could

• add suffixes (.r, .c) for duplicates like tidyverse does for duplicate columns (plus warning/message), or
• add an error and force an explicit declaration (plyxp) in cases we cannot solve, or
• ...

[stemangiola]

I am linking the tidySE issue

tidyomics/tidySummarizedExperiment#106

[stemangiola]

OK, I have a first implementation for mutate, filter, select

Benchmark report: https://rpubs.com/mangiolas/1339985

Single example

> nfeat <- 1e4
> nsamp <- 1e3
> coldata <- data.frame(x=rep(1:2,length=nsamp))
> assay <- matrix(rnorm(nfeat*nsamp), ncol=nsamp)
> rownames(assay) <- paste0("f",1:nfeat)
> colnames(assay) <- paste0("s",1:nsamp)
> 
> se <- SummarizedExperiment(list(data=assay), colData=coldata)
> 
> system.time({
+     new_se <- se |>
+         mutate(newdata = data * x)
+ })
   user  system elapsed 
  0.064   0.071   0.135 
> 
> system.time({
+     new_se2 <- se |>
+         new_plyxp() |>
+         mutate(newdata = data * .cols$x) |>
+         se()
+ })
   user  system elapsed 
  0.047   0.009   0.055 
> 
> all.equal(assay(new_se), assay(new_se2))
[1] TRUE

Now we use plyxp for complex multi-slot operations, while isolating sample-only, feature-only, and assay-only (potentially we could fold back to plyxp for single-slot operations as well).

It will require more checks and generalisations. The key utilities are

1. decompose operation-series (e.g. mutate(..., ..., ...)) into single onea for simple handling

https://github.com/tidyomics/tidySummarizedExperiment/blob/92072d71f9d3b9a82cfc5fdced8e52477c44d80f/R/mutate.R#L146

2. Analyse scope (very overengineered probably need to simplify)

https://github.com/tidyomics/tidySummarizedExperiment/blob/92072d71f9d3b9a82cfc5fdced8e52477c44d80f/R/mutate.R#L149

3. If mixed operations send to plyxp

https://github.com/tidyomics/tidySummarizedExperiment/blob/92072d71f9d3b9a82cfc5fdced8e52477c44d80f/R/mutate.R#L155

[mikelove]

nice! @jtlandis and I can test this out this week.