Quickstart
Installation
MARADONER is best installed via the pip package manager:
> pip install maradoner
Alternatively, the latest version can be fetched from the git repository:
> git clone https://github.com/autosome-ru/maradoner > cd maradoner > pip install .
MARADONER needs Python 3.11 or newer.
Overview
A user communicates with MARADONER through the command-line interface. After the package is installed, the environment should become aware of the maradoner command. Let's check that it is indeed the case by invoking
> maradoner --help
A list of available commands alongside their short descriptions should appear. Help is available for any listed command as well, e.g.
> maradoner create --help
In theory, help should provide sufficient information to deduce how to do everything. This can be infeasible in practice, though, and the goal of this tutorial is to explain everything in greater detail and in an easier-to-follow fashion.
A MARADONER session always starts with creating a project from the data of interest. That is done with the create command. It takes a project name, a table of promoter expression and a matrix of motif loadings:
> maradoner create MyProject expression.tsv loadings.tsv --sample-groups groups.json
Here, groups.json tells which samples are replicates of the same condition. Data formats and filters are explained in the next section. After MARADONER is done reading the tables, it writes the MyProject.init.raw file. MARADONER stores intermediate results in files whose names start with the project name followed by the step name and a compression method (raw stands for none):
{ProjectName}.{StepName}.{CompressionMethod}
The init file holds the data after filtering, so it can be moved to another machine, and all the other steps run there without the source tables.
Usually, after the project is created, we fit model parameters to the data with the fit command:
> maradoner fit MyProject
This step estimates the noise level of each group of samples, the variance of activity of each motif and the mean activity of each motif. More on that in the Fit section of the tutorial. Successful invocation of the command should result in the MyProject.fit.raw file being created.
Next, we predict motif activities with predict:
> maradoner predict MyProject
The result is saved to MyProject.predict.raw. It holds the deviation of the activity of each motif from its mean in each group, as covered in the Motif activities section.
Finally, we export all our results as tabular data to a folder result_folder:
> maradoner export MyProject result_folder
The other commands are optional. A possible workflow is depicted in the diagram below:
The commands depend on each other: fit needs create, predict needs fit, and the commands of the lower row need predict. A new run of an earlier step makes the later ones outdated. MARADONER removes their files, so after a new fit we run predict again.
Example
MARADONER arrives with a generator of synthetic data. A synthetic dataset suits a tutorial well, as the true values of all parameters are known and the estimates can be checked against them. First, let's generate one:
> maradoner generate demo
In the current working directory, a folder demo should appear. It holds the 3 input files (expression.tsv, loadings.tsv and groups.json) and the true parameters (activities.tsv, sigma.tsv, motif_means.tsv and others). The dataset has 2000 promoters, 500 motifs and 57 samples in 10 groups. The activity of 100 motifs is zero in all samples, and their names start with inactive_.
Then we run the create command with the --loading-transform none argument. By default, MARADONER transforms each column of the loading matrix to put all motifs on one scale (more on that in Loading transformations). The synthetic expression, however, was generated from the loadings exactly as they are in the file, hence we turn the transformation off:
> maradoner create MyProject demo/expression.tsv demo/loadings.tsv --sample-groups demo/groups.json --loading-transform none
After it is done, MARADONER provides some dataset statistics:
Reading dataset... Filtering promoters of low expression... [warning] Low-expression filter: the low-expression component of the fitted mixture (weight 0.0%) is not the more probable component for any promoter, so the data have no clearly separated class of unexpressed promoters. Only the promoters of the lowest expression were removed (57 of 2000); use --filter-lowexp-w 1 to keep them all. Low-expression filter: 57 of 2000 promoters were dropped, 1943 kept. Saving project... Number of promoters: 1943, number of motifs: 500, number of samples: 57, number of groups: 10 ✔️ Done! time: 1.51 s.
The warning is fine and almost self-explanatory. Real data have a class of silent promoters, and MARADONER looks for it to drop them. Synthetic data have none, so only the 57 promoters of the lowest expression are gone. The filter also draws what it has done:
The histogram is the expression of promoters, and the curve is the mixture fitted to it. The colour of the curve is the probability that a promoter of that expression is kept: blue is 0 and green is 1. The filter is discussed in the Reading dataset section.
Now, let's obtain parameter estimates for the model:
> maradoner fit MyProject
After a few seconds, we should see the following:
Transforming data... Computing low-rank decompositions of the loading matrix... Estimating error variances... --------------- Current objective: 149199.4516 Objective at a start: 154963.3193 Objective after a warm-up: 149199.4518 Objective after momentum optimization: 149199.4516 Current gradient norm: 4.2854e-08 --------------- Estimating promoter-wise means... Estimating variances of motif activities... [motif variances] 7 iterations, 8 evaluations of the deviance (converged=True) --------------- Current objective: -118060.6118 Objective at a start: -118011.8096 Objective after Fisher scoring: -118060.6118 Current gradient norm: 3.5116e-05 --------------- Smallest eigenvalue of the Fisher information of the variances (scaled to a unit diagonal): 0.04520382150075708 Estimating motif means... Estimating sample means... ✔️ Done! time: 3.64 s.
The 2 blocks between the dashed lines are the 2 numerical optimizations of the fit: the first one is for the error variances, the second one is for the variances of motif activities. The line above the second block counts the steps of its optimization and the evaluations of the deviance. The gradient norm at the end of each block is close to zero, hence both have converged. The eigenvalue of the Fisher information is positive, so every variance parameter is identified by the data and has a standard error. When it is not the case, MARADONER prints a warning with the probable cause.
Next, we predict motif activities and compute the goodness of fit:
> maradoner predict MyProject > maradoner gof MyProject
Fraction of variance explained (Total)
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Set ┃ centering ┃ + mean motif activity ┃ + group-specific motif activity ┃
┃ Y = ┃ mu_p 1_s^T + 1_p mu_s^T ┃ + B M X ┃ + B U ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ train │ 0.053025 │ 0.631743 │ 0.951265 │
└───────┴─────────────────────────┴───────────────────────┴─────────────────────────────────┘
The columns are 3 nested models. Promoter and sample means alone explain 5% of the variance of the expression. With the mean motif activities added it is 63%, and with the activities of each group it is 95%. More details are in the Goodness of fit section.
Finally, we export the results in a tabular form:
> maradoner export MyProject results
The results folder now has the tables and a README.md file with a description of every table and every column in it. Let's take a look at some of them, starting with the variances of groups:
> head -4 results/params/group_variances.tsv
sigma sigma_std nu nu_std group_1 1.1980252679782921 0.018713762295625304 0.011435774750403752 0.0006575606857800102 group_10 1.0415625112587141 0.017899609814909023 0.010299003215052907 0.0006184461062784683 group_2 1.4550496356385987 0.022626370390779395 0.014874855285232802 0.0008485312731564691
Here, sigma is the error variance of a group. Its true values are in demo/sigma.tsv: 1.2032 for group_1, 1.0563 for group_10 and 1.4547 for group_2. Over the 10 groups the estimates are off by 0.9% on average. The estimated mean activities of motifs (results/params/motif_means.tsv) have a correlation of 0.994 with the true ones (demo/motif_means.tsv).
The next table tests whether the activity of a motif varies across samples at all:
> cut -f1-4 results/tests/asymptotics_based/anova.tsv | head -5
stat p-value FDR motif_1 200.3568137989603 8.728457136521635e-46 2.3463594453015148e-45 motif_2 4.667415152383476 0.015370080121629697 0.021288199614445565 inactive_motif_3 0.3552537909882342 0.2755770239378897 0.33362835827831683 motif_4 173.89705895563748 5.212365813227326e-40 1.3573869305279495e-39
The table has 440 motifs out of 500. Where are the other 60? Their variance was estimated as zero, so predict had nothing to predict for them and left them out. They are listed in results/params/filtered_motif_variances.tsv, and 51 of them are inactive motifs. The other 9 are active motifs with a small true variance: 0.054 at most, against a median of 0.34. Of the 440 motifs in the table, 366 pass the FDR threshold of 0.05, and 1 of those is an inactive one. That's fine at this threshold.
The activities are in the results/activities folder:
> cut -f1-4 results/activities/activity.tsv | head -5
group_1 group_10 group_2 motif_1 4.133243140589579 -3.5091015825203997 4.338709020333753 motif_2 0.05777635556433165 -0.12651655761007372 -0.41085475188123327 inactive_motif_3 0.15207927482971603 -0.8116466142524963 0.9733592975885609 motif_4 0.3947415107085393 -0.39390905635815227 0.22364751235347713
Each number is a z-score: the deviation of the activity of a motif from its mean in a group, divided by the posterior standard deviation. So motif_1 is more active than on average in group_1 and group_2, and less active in group_10.
Likewise, we can compare 2 groups directly:
> maradoner contrast MyProject "group_2 - group_1" results
This adds the results/contrast.tsv table with a z-test for each motif. Here, 69 motifs differ between the 2 groups at an FDR of 0.05. Contrasts and the other tests are covered in the Tests section.


