MARADONER

Quickstart

Installation

MARADONER is best installed via the pip package manager:

> pip install maradoner

Alternatively, the latest version can be fetched from the git repository:

> git clone https://github.com/autosome-ru/maradoner
> cd maradoner
> pip install .

MARADONER needs Python 3.11 or newer.

Overview

A user communicates with MARADONER through the command-line interface. After the package is installed, the environment should become aware of the maradoner command. Let's check that it is indeed the case by invoking

> maradoner --help

A list of available commands alongside their short descriptions should appear. Help is available for any listed command as well, e.g.

> maradoner create --help

In theory, help should provide sufficient information to deduce how to do everything. This can be infeasible in practice, though, and the goal of this tutorial is to explain everything in greater detail and in an easier-to-follow fashion.

A MARADONER session always starts with creating a project from the data of interest. That is done with the create command. It takes a project name, a table of promoter expression and a matrix of motif loadings:

> maradoner create MyProject expression.tsv loadings.tsv --sample-groups groups.json

Here, groups.json tells which samples are replicates of the same condition. Data formats and filters are explained in the next section. After MARADONER is done reading the tables, it writes the MyProject.init.raw file. MARADONER stores intermediate results in files whose names start with the project name followed by the step name and a compression method (raw stands for none):

{ProjectName}.{StepName}.{CompressionMethod}

The init file holds the data after filtering, so it can be moved to another machine, and all the other steps run there without the source tables.

Usually, after the project is created, we fit model parameters to the data with the fit command:

> maradoner fit MyProject

This step estimates the noise level of each group of samples, the variance of activity of each motif and the mean activity of each motif. More on that in the Fit section of the tutorial. Successful invocation of the command should result in the MyProject.fit.raw file being created.

Next, we predict motif activities with predict:

> maradoner predict MyProject

The result is saved to MyProject.predict.raw. It holds the deviation of the activity of each motif from its mean in each group, as covered in the Motif activities section.

Finally, we export all our results as tabular data to a folder result_folder:

> maradoner export MyProject result_folder

The other commands are optional. A possible workflow is depicted in the diagram below:

Read datacreate Estimate parametersfit Predict activitiespredict Export tablesexport Goodness of fitgof Group testsanova, contrast Regulatory networkgrn

The commands depend on each other: fit needs create, predict needs fit, and the commands of the lower row need predict. A new run of an earlier step makes the later ones outdated. MARADONER removes their files, so after a new fit we run predict again.

Example

MARADONER arrives with a generator of synthetic data. A synthetic dataset suits a tutorial well, as the true values of all parameters are known and the estimates can be checked against them. First, let's generate one:

> maradoner generate demo

In the current working directory, a folder demo should appear. It holds the 3 input files (expression.tsv, loadings.tsv and groups.json) and the true parameters (activities.tsv, sigma.tsv, motif_means.tsv and others). The dataset has 2000 promoters, 500 motifs and 57 samples in 10 groups. The activity of 100 motifs is zero in all samples, and their names start with inactive_.

Then we run the create command with the --loading-transform none argument. By default, MARADONER transforms each column of the loading matrix to put all motifs on one scale (more on that in Loading transformations). The synthetic expression, however, was generated from the loadings exactly as they are in the file, hence we turn the transformation off:

> maradoner create MyProject demo/expression.tsv demo/loadings.tsv --sample-groups demo/groups.json --loading-transform none

After it is done, MARADONER provides some dataset statistics:

Reading dataset...
Filtering promoters of low expression...
[warning] Low-expression filter: the low-expression component of the fitted mixture (weight 0.0%) is not the more probable component for any promoter, so the data have no clearly separated class of unexpressed promoters. Only the promoters of the lowest expression were removed (57 of 2000); use --filter-lowexp-w 1 to keep them all.
Low-expression filter: 57 of 2000 promoters were dropped, 1943 kept.
Saving project...

Number of promoters: 1943, number of motifs: 500, number of samples: 57, number of groups: 10
✔️ Done!         time: 1.51 s.

The warning is fine and almost self-explanatory. Real data have a class of silent promoters, and MARADONER looks for it to drop them. Synthetic data have none, so only the 57 promoters of the lowest expression are gone. The filter also draws what it has done:

MyProject.filter-plot.pngMyProject.filter-plot.png
MyProject.filter-plot.png

The histogram is the expression of promoters, and the curve is the mixture fitted to it. The colour of the curve is the probability that a promoter of that expression is kept: blue is 0 and green is 1. The filter is discussed in the Reading dataset section.

Now, let's obtain parameter estimates for the model:

> maradoner fit MyProject

After a few seconds, we should see the following:

Transforming data...
Computing low-rank decompositions of the loading matrix...
Estimating error variances...
---------------
Current objective: 149199.4516
Objective at a start: 154963.3193
Objective after a warm-up: 149199.4518
Objective after momentum optimization: 149199.4516
Current gradient norm: 4.2854e-08
---------------
Estimating promoter-wise means...
Estimating variances of motif activities...
[motif variances] 7 iterations, 8 evaluations of the deviance (converged=True)
---------------
Current objective: -118060.6118
Objective at a start: -118011.8096
Objective after Fisher scoring: -118060.6118
Current gradient norm: 3.5116e-05
---------------
Smallest eigenvalue of the Fisher information of the variances (scaled to a unit diagonal): 0.04520382150075708
Estimating motif means...
Estimating sample means...

✔️ Done!         time: 3.64 s.

The 2 blocks between the dashed lines are the 2 numerical optimizations of the fit: the first one is for the error variances, the second one is for the variances of motif activities. The line above the second block counts the steps of its optimization and the evaluations of the deviance. The gradient norm at the end of each block is close to zero, hence both have converged. The eigenvalue of the Fisher information is positive, so every variance parameter is identified by the data and has a standard error. When it is not the case, MARADONER prints a warning with the probable cause.

Next, we predict motif activities and compute the goodness of fit:

> maradoner predict MyProject
> maradoner gof MyProject
                           Fraction of variance explained (Total)
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Set   ┃ centering               ┃ + mean motif activity ┃ + group-specific motif activity ┃
┃ Y =   ┃ mu_p 1_s^T + 1_p mu_s^T ┃ + B M X               ┃ + B U                           ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ train │ 0.053025                │ 0.631743              │ 0.951265                        │
└───────┴─────────────────────────┴───────────────────────┴─────────────────────────────────┘

The columns are 3 nested models. Promoter and sample means alone explain 5% of the variance of the expression. With the mean motif activities added it is 63%, and with the activities of each group it is 95%. More details are in the Goodness of fit section.

Finally, we export the results in a tabular form:

> maradoner export MyProject results

The results folder now has the tables and a README.md file with a description of every table and every column in it. Let's take a look at some of them, starting with the variances of groups:

> head -4 results/params/group_variances.tsv
	sigma	sigma_std	nu	nu_std
group_1	1.1980252679782921	0.018713762295625304	0.011435774750403752	0.0006575606857800102
group_10	1.0415625112587141	0.017899609814909023	0.010299003215052907	0.0006184461062784683
group_2	1.4550496356385987	0.022626370390779395	0.014874855285232802	0.0008485312731564691

Here, sigma is the error variance of a group. Its true values are in demo/sigma.tsv: 1.2032 for group_1, 1.0563 for group_10 and 1.4547 for group_2. Over the 10 groups the estimates are off by 0.9% on average. The estimated mean activities of motifs (results/params/motif_means.tsv) have a correlation of 0.994 with the true ones (demo/motif_means.tsv).

The next table tests whether the activity of a motif varies across samples at all:

> cut -f1-4 results/tests/asymptotics_based/anova.tsv | head -5
	stat	p-value	FDR
motif_1	200.3568137989603	8.728457136521635e-46	2.3463594453015148e-45
motif_2	4.667415152383476	0.015370080121629697	0.021288199614445565
inactive_motif_3	0.3552537909882342	0.2755770239378897	0.33362835827831683
motif_4	173.89705895563748	5.212365813227326e-40	1.3573869305279495e-39

The table has 440 motifs out of 500. Where are the other 60? Their variance was estimated as zero, so predict had nothing to predict for them and left them out. They are listed in results/params/filtered_motif_variances.tsv, and 51 of them are inactive motifs. The other 9 are active motifs with a small true variance: 0.054 at most, against a median of 0.34. Of the 440 motifs in the table, 366 pass the FDR threshold of 0.05, and 1 of those is an inactive one. That's fine at this threshold.

The activities are in the results/activities folder:

> cut -f1-4 results/activities/activity.tsv | head -5
	group_1	group_10	group_2
motif_1	4.133243140589579	-3.5091015825203997	4.338709020333753
motif_2	0.05777635556433165	-0.12651655761007372	-0.41085475188123327
inactive_motif_3	0.15207927482971603	-0.8116466142524963	0.9733592975885609
motif_4	0.3947415107085393	-0.39390905635815227	0.22364751235347713

Each number is a z-score: the deviation of the activity of a motif from its mean in a group, divided by the posterior standard deviation. So motif_1 is more active than on average in group_1 and group_2, and less active in group_10.

Likewise, we can compare 2 groups directly:

> maradoner contrast MyProject "group_2 - group_1" results

This adds the results/contrast.tsv table with a z-test for each motif. Here, 69 motifs differ between the 2 groups at an FDR of 0.05. Contrasts and the other tests are covered in the Tests section.

On this page