Warning message reporting more samples that I have

[Valentin-Bio]

Hello developer, I have tested ulrb software on a dataframe extracted from a phyloseq object:

This is the code I run:

ant_ps <- read_rds("antartic_ps.RDS")

ant_mod <- ant_ps %>%
  psmelt()

ant_mod$Sample <- as.factor(ant_mod$Sample)

ant_abs <- ant_mod %>% 
  dplyr::select(OTU, Abundance, Sample)

res <- ulrb::define_rb(ant_abs, samples_col = "Sample", abundance_col = "Abundance")

# 3794 out of 18451 observations with silhouette scores lower than 0.5 
length(which(res$Silhouette_scores < 0.5))

and this is the Warning message I got:

If half the observations within a classification are below 0.5 Silhouette score, we consider that the clustering was 'Bad'.
Check 'Evaluation' collumn for more details.
Warning message:
In ulrb::define_rb(ant_abs, samples_col = "Sample", abundance_col = "Abundance") :
53 samples got a bad Silhouette score. Consider changing the number of classifications.

In context, I have 49 samples in my dataset, so , why the warning message is telling me 53 samples got bad classifications? I even considered transforming the Sample variable into a factor but still getting the same warning. Then I thought the program was counting the number of observations with silhouette scores lower than 0.5 expecting there were 52 observations with bad classifications but this is not the case as I see it with the length(which(res$Silhouette_scores < 0.5)) code. So why is the define_rb() command reporting 53 samples?.

bests,

Valentín.

[pascoalf]

Hi Valentin,

I have to check if this is a bug, so I'll look at this more carefully when I have time and report back.

Can you provide the antartic_ps.RDS file?

Thank you,
Francisco

[pascoalf]

Hi Valentin,

We were able to identify the problem. Briefly, the warning message was counting the number of bad clusters, instead of the number of bad samples overall. However, one sample may have multiple good clusters and one bad cluster, for example.

To solve this issue, we modified the warning message, so that now it specifically counts the number of bad clusters and the number of bad samples. Note that a sample with one weak cluster can have an overall high score, if the remaining clusters are very good. Thus, it will be possible to have a warning message indicating that there are 0 bad samples, but 3 bad clusters.

We made a new branch with the bug fix, which will correspond to version 0.1.9. To install it, do:

devtools::install_github("pascoalf/ulrb@issue_4")
# check you have the right version (0.1.9)
library(ulrb); packageVersion("ulrb")

Below, we exemplify the warning message from the corrected version, using the esophagus dataset from Phyloseq.

library(phyloseq)
library(dplyr)
library(ulrb); packageVersion("ulrb") ## should be 0.1.9
data(esophagus)
esophagus_mod <- esophagus %>% psmelt()
esophagus_mod$Sample <- as.factor(esophagus_mod$Sample)
esophagus_abs <- esophagus_mod %>% dplyr::select(OTU, Abundance, Sample)
res <- ulrb::define_rb(esophagus_abs, 
                 samples_col = "Sample",
                 abundance_col = "Abundance")

In this example, the esophagus dataset has 3 samples, which obtained an overall good score. However, within the good samples, there were 4 different clusters with weak scores. Thus, the warning message was the following:

Within 0 bad samples, there were 4 clusters with bad average Silhouette score.
If half the observations within a classification are below 0.5 Silhouette score, we consider that the clustering was 'Bad'.
Check 'Evaluation' collumn for more details.
Warning message:
In define_rb(esophagus_abs, samples_col = "Sample", abundance_col = "Abundance") :
0 samples got bad average Silhouette score.

To verify if the warning makes sense, we can check results visually:

ulrb::plot_ulrb(res, plot_all = TRUE, taxa_col = "OTU")

Notes:

• This bug fix will be added to a new version of ulrb, 0.1.9.
• This bug fix is currently available in a dedicated branch, issue_4
• We will discuss with colleagues the warning message and do further testing before merging this branch to main and before submitting version 0.1.9 to CRAN.

If possible, check the new warning message on your data.

Thank you for your contribution!
Francisco Pascoal