MARADONER

Goodness of fit

Fraction of variance explained

How much of the data does the model explain? The gof command answers that for each nested part of the model. It runs after predict and needs only the project name:

> maradoner gof MyProject
                           Fraction of variance explained (Total)
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Set   ┃ centering               ┃ + mean motif activity ┃ + group-specific motif activity ┃
┃ Y =   ┃ mu_p 1_s^T + 1_p mu_s^T ┃ + B M X               ┃ + B U                           ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ train │ 0.053025                │ 0.631743              │ 0.951265                        │
└───────┴─────────────────────────┴───────────────────────┴─────────────────────────────────┘

The table is from the Quickstart example. Each column adds a term to the model of the previous one:

  1. Promoter-wise and sample-wise means alone;
  2. The same with the mean motif activities;
  3. The same with the deviations of activities in each group, i.e. the full model.

The second line of the header spells the terms out. There, B M X is the loading matrix times the mean activities, and it reduces to $B μ_m 1_s^T$ when the project has no covariates.

If gof is run before export, the same statistics are exported into the fov folder: for the whole dataset, for each promoter and for each sample.

The statistic and the way it is computed can be changed:

Name Description Default value
--stat-type <str> fov for the fraction of variance explained, corr for the Pearson correlation between the predicted and the observed expression fov
--stat-mode <str> total accumulates the terms as described above. With residual the columns go the other way: the full model first, then the model against the data with the promoter and sample means subtracted, then the deviations alone against the data with the mean activities subtracted as well total
--use-groups The statistic is computed for the expression averaged within each group Disabled
--no-weights Disables the weighting of promoters. Applicable only to fits with --promoter-variance --weights

Testing set

The numbers above are computed on the same promoters the model was fitted to, and they are optimistic. For a fair estimate, some promoters are held out at the fit step and the statistics are computed for them separately. The whole sequence is:

> maradoner fit MyProject --test-chromosomes 3 --test-chromosomes X
> maradoner predict MyProject
> maradoner estimate-promoter-mean MyProject
> maradoner gof MyProject

Here, the promoters of chromosomes 3 and X take no part in the fit. Alternatively, a list of promoter names can be given with --test-promoters-filename. The third command is new, and we get to it in a moment. With 198 promoters of the Quickstart example held out (by a list of names, as synthetic promoters have no chromosomes), the gof table gets a second row:

┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Set   ┃ centering               ┃ + mean motif activity ┃ + group-specific motif activity ┃
┃ Y =   ┃ mu_p 1_s^T + 1_p mu_s^T ┃ + B M X               ┃ + B U                           ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ train │ 0.066498                │ 0.646975              │ 0.952379                        │
│ test  │ 0.052690                │ 0.606242              │ 0.896829                        │
└───────┴─────────────────────────┴───────────────────────┴─────────────────────────────────┘

The full model explains 90% of the variance of the promoters it has not seen, against 95% of those it has.

Why is estimate-promoter-mean needed?

Each promoter has a mean $μ_p$, and it is a parameter estimated by fit. A held-out promoter has none, and the model can't predict its expression without one. The estimate-promoter-mean command fills the gap. Without it, gof stops with an error asking for it. It has to be run again after each predict.

The method is chosen with the --mean-mode option:

Name Description
null (Default) The means of held-out promoters are set to zero.
gls A generalized least squares estimate from the expression of the held-out promoters. It uses the data the model is then tested on, so the resulting statistics are prone to leakage.
knn The mean is borrowed from the nearest promoters of the training set. The neighbours are searched by principal components of the loading matrix and of the predicted effects. Their numbers are set with --pca-b (64 by default) and --pca-z (3), and the number of neighbours with --knn-n (256).

A table of numeric promoter covariates (promoters in rows, covariates in columns) can be passed with the --covariate option. The means are then predicted with the help of a generalized additive model of the covariates. The gls mode ignores the option.

A model of the covariates is trained on the fitted means of the training promoters, so it describes the part of a promoter level that the motifs leave unexplained. The --covariate-model option chooses it:

Name Description
gam (Default) A spline per covariate. It suits a few covariates, such as GC content and CpG ratio.
ridge Ridge regression, with the penalty chosen by cross-validation.
boosting Gradient boosting of trees.
ensemble The average of ridge and boosting.

The last 3 take tables of thousands of columns. Such a table is made from promoter sequences by the sequence-features command: the counts of all k-mers, the GC content and the CpG ratio in bins of positions around the TSS. The sequences go in a FASTA file, in the orientation of the promoter and with the TSS at the same position, for example windows from 1000 bp upstream to 500 bp downstream cut by bedtools getfasta -s -name+:

> maradoner sequence-features promoters.fa seqfeatures.tsv --tss 1000 --bins -1000,-500,-250,-50,50,250,500 --k 5
> maradoner estimate-promoter-mean MyProject --covariate seqfeatures.tsv --covariate-model ensemble
> maradoner gof MyProject

On a muscle atlas of 36,120 CAGE promoters with 4 chromosomes held out, these features raised the FOV of the held-out promoters from 0.21 to 0.33. The gain is in the promoter levels: the motif activities and their tests stay as they are.

On this page