# .readthedocs.yml
Source: https://demo.agenticlabs.com/detailed-code-docs/.readthedocs.yml
## High-level description
This file is a configuration file for Read the Docs, a documentation hosting platform. It specifies the build environment, Python version, Sphinx configuration, and installation requirements for building the project's documentation.
## Symbols
### `version`
#### Description
Specifies the version of the Read the Docs configuration format being used.
#### Inputs
| Name | Type | Description |
| :------ | :------ | :--------------------------------------------- |
| version | integer | The version number of the configuration format |
### `build`
#### Description
Defines the build environment settings for the documentation.
#### Inputs
| Name | Type | Description |
| :---- | :----- | :--------------------------------------------------- |
| os | string | The operating system to use for building |
| tools | object | Specifies the tools and their versions for the build |
### `sphinx`
#### Description
Configures the Sphinx documentation generator.
#### Inputs
| Name | Type | Description |
| :------------ | :----- | :------------------------------------ |
| configuration | string | Path to the Sphinx configuration file |
### `python`
#### Description
Specifies Python-related settings for the documentation build.
#### Inputs
| Name | Type | Description |
| :------ | :---- | :---------------------------------------------------- |
| install | array | List of installation methods and their configurations |
## Dependencies
| Dependency | Purpose |
| :------------ | :---------------------------------------------------------- |
| Read the Docs | Documentation hosting platform |
| Sphinx | Documentation generator |
| Python | Programming language used for the project and documentation |
## Configuration
| Option | Type | Default | Description |
| :------------------- | :------ | :------------- | :---------------------------------------------------------- |
| version | integer | 2 | Read the Docs configuration format version |
| build.os | string | "ubuntu-22.04" | Operating system for the build environment |
| build.tools.python | string | "3.9" | Python version to use for the build |
| sphinx.configuration | string | "docs/conf.py" | Path to the Sphinx configuration file |
| python.install | array | - | Installation methods and configurations for Python packages |
## API/Interface Reference
| Endpoint | Method | Request | Response | Description |
| :------- | :----- | :------ | :------- | :----------------------------------------------------------------- |
| N/A | N/A | N/A | N/A | This configuration file does not define an API or public interface |
This configuration file sets up the build environment for Read the Docs to generate and host the project's documentation. It specifies Ubuntu 22.04 as the operating system and Python 3.9 as the Python version to use. The Sphinx configuration file is located at `docs/conf.py`.
The `python.install` section defines the installation method for the project and its dependencies. It uses pip to install the current project (`.`) along with extra requirements for documentation (`docs`) and spatial features (`spatial`).
This configuration ensures that the documentation can be built consistently and with the necessary dependencies on the Read the Docs platform.
# README.md
Source: https://demo.agenticlabs.com/detailed-code-docs/README.md
\)
[](https://github.com/YosefLab/cassiopeia/stargazers)
[](https://cassiopeia.readthedocs.io/en/stable/?badge=stable)
[](https://codecov.io/gh/YosefLab/cassiopeia)
# Cassiopeia: A pipeline for single-cell lineage tracing data
Cassiopeia is an end-to-end pipeline for single-cell lineage tracing experiments.
The software contained here comes equipped with modules for processing sequencing reads,
reconstructing & plotting trees, analyzing these trees, and benchmarking new algorithms.
You can find all of our [documentation here](https://cassiopeia-lineage.readthedocs.io/en/latest/index.html).
We also have provided tutorials for three modules:
* [processing fastqs](https://github.com/YosefLab/Cassiopeia/blob/master/notebooks/preprocess.ipynb)
* [reconstructing trees](https://github.com/YosefLab/Cassiopeia/blob/master/notebooks/reconstruct.ipynb)
* [simulating trees and benchmarking](https://github.com/YosefLab/Cassiopeia/blob/master/notebooks/benchmark.ipynb)
* [plotting trees with our local library](https://github.com/YosefLab/Cassiopeia/blob/master/notebooks/local_plotting.ipynb)
You can also find our originally describing Cassiopeia published in [Genome Biology](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-02000-8).
Free Software: MIT License
## Installation
For users:
```
pip install git+https://github.com/YosefLab/Cassiopeia@master#egg=cassiopeia-lineage
```
For developers:
1. Clone the package as so: `git clone https://github.com/YosefLab/Cassiopeia.git`
2. Ensure that you have Python >= 3.8 installed. (Due to dependencies no longer supporting Python 3.7, we have now stopped supporting Python \<= 3.7) We prefer using [miniconda](https://docs.conda.io/en/latest/miniconda.html).
3. \[Optional] If you intend to use the CassiopeiaILP solver, please be sure that Gurobi is installed. You can follow the instructions listed [here](http://www.gurobi.com/academia/for-universities). To verify that it's working correctly, use the following tests:
* Run the command `gurobi.sh` from a terminal window
* From the Gurobi installation directory (where there is a setup.py file), use `python setup.py install --user`
4. \[Optional] To use fast versions of Neighbor-Joining and UPGMA, install [CCPhylo](https://bitbucket.org/genomicepidemiology/ccphylo/src/master/). Then copy the file `./data/ccphylo_config.ini` to your `./cassiopeia` directory, rename it `config.ini` and set the Path variable to point to your CCPhylo installation.
5. Install Cassiopeia by first changing into the Cassiopeia directory and then `pip3 install .` or `make install`. To install dev and docs requirements, you can run `pip3 install .[dev,docs]`.
6. \[Optional] To use tools built for the analysis of spatial lineage tracing datasets, you can install Cassiopeia with `pip install .[spatial]`. Please note that we recommend using Python >= 3.9 for these analyses as some features might not be available otherwise, due to package dependencies (especially 3D visualization).
To verify that it installed correctly, install `pytest` (`pip install pytest`) and try running our tests with `make test`.
## Reference
If you've found Cassiopeia useful for your research, please consider citing our paper published in Genome Biology:
Matthew G Jones\*, Alex Khodaverdian\*, Jeffrey J Quinn\*, Michelle M Chan, Jeffrey A Hussmann, Robert Wang, Chenling Xu, Jonathan S Weissman, Nir Yosef. (2020), [*Inference of single-cell phylogenies from lineage tracing data using Cassiopeia*](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-02000-8), Genome Biology
# build.py
Source: https://demo.agenticlabs.com/detailed-code-docs/build.py
## High-level description
This Python script, `build.py`, is responsible for building and compiling Cython extensions for the Cassiopeia project. It defines three extensions, compiles them using Cython, and copies the built extensions back to the project directory.
## Code Structure
The script defines a `build()` function that handles the entire build process. It creates Extension objects, uses Cython to compile them, and then copies the compiled files to the appropriate locations. The main execution is controlled by the `if __name__ == "__main__":` block at the end.
## Symbols
### build()
#### Description
This function manages the entire process of building Cython extensions for the Cassiopeia project.
#### Internal Logic
1. Define a list of `Extension` objects for the Cython modules to be compiled.
2. Use `cythonize()` to compile the extensions.
3. Create a `Distribution` object with the compiled extensions.
4. Initialize and run a `build_ext` command to build the extensions.
5. Copy the built extensions back to the project directory and set appropriate file permissions.
### Extension objects
#### Description
Three `Extension` objects are defined, each representing a Cython module to be compiled:
1. `"cassiopeia.preprocess.collapse_cython"`
2. `"cassiopeia.solver.ilp_solver_utilities"`
3. `"cassiopeia.tools.branch_length_estimator._iid_exponential_bayesian"`
Each extension specifies its source files and, where necessary, include directories and compile arguments.
## Dependencies
| Dependency | Purpose |
| :--------------------------- | :------------------------------------------------------ |
| os | File and directory operations |
| shutil | High-level file operations |
| distutils.command.build\_ext | Provides `build_ext` command for building extensions |
| distutils.core | Provides `Distribution` and `Extension` classes |
| numpy | Used for include directories in one of the extensions |
| Cython.Build | Provides `cythonize` function for compiling Cython code |
## Performance Considerations
The third extension, `"cassiopeia.tools.branch_length_estimator._iid_exponential_bayesian"`, includes C++ source files and uses aggressive optimization flags (`-O3`), indicating that this module is likely performance-critical.
## Error Handling
This script does not implement explicit error handling. It relies on the default error handling provided by the libraries and functions it uses.
## Side Effects
The script modifies the file system by creating compiled extension files and copying them to the project directory. It also changes the file permissions of the copied extensions.
This documentation provides a comprehensive overview of the `build.py` script, explaining its purpose, structure, and key components. It should help engineers and technical PMs understand how the Cython extensions for the Cassiopeia project are built and integrated.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/__init__.py
Here's a documentation for the `cassiopeia/__init__.py` file:
## High-level description
This file serves as the top-level initialization for the Cassiopeia package. It imports and exposes key submodules, sets up version information, and configures the package namespace.
## Code Structure
The file imports various submodules of Cassiopeia and makes them available at the top level of the package. It also sets up version information using importlib.metadata and configures the package namespace.
## Symbols
### Imported submodules
#### Description
The following submodules are imported and made available at the top level of the Cassiopeia package:
* `preprocess` (aliased as `pp`)
* `solver`
* `plotting` (aliased as `pl`)
* `data`
* `critique`
* `simulator` (aliased as `sim`)
* `tools` (aliased as `tl`)
These imports allow users to access these submodules directly from the Cassiopeia package.
### `__version__`
#### Description
Stores the version number of the Cassiopeia package.
#### Internal Logic
The version is retrieved using `importlib.metadata.version()` with the package name "cassiopeia-lineage". This allows for dynamic version retrieval based on the installed package metadata.
### Package namespace configuration
#### Description
Configures the package namespace to allow direct access to certain submodules.
#### Internal Logic
The `sys.modules` dictionary is updated to allow direct access to the `tl`, `pp`, `pl`, and `sim` submodules from the Cassiopeia package namespace. This means users can access these submodules directly, e.g., `cassiopeia.tl` instead of `cassiopeia.tools`.
## Dependencies
| Dependency | Purpose |
| :------------------ | :-------------------------------------------------------------- |
| importlib.metadata | Used for retrieving the package version |
| importlib\_metadata | Fallback for Python versions that don't have importlib.metadata |
## Error Handling
The code includes a try-except block to handle potential `ModuleNotFoundError` when importing `importlib.metadata`. If this import fails, it falls back to using `importlib_metadata`.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/critique/__init__.py
## High-level description
This file serves as the top-level module for the cassiopeia critique package. It imports and exposes two key functions, `robinson_foulds` and `triplets_correct`, from the `compare` module, which are used for comparing phylogenetic trees.
## Symbols
### `robinson_foulds`
#### Description
This function computes the Robinson-Foulds distance between two phylogenetic trees. It is imported from the `compare` module.
#### Inputs
| Name | Type | Description |
| :---- | :------------- | :------------------------- |
| tree1 | CassiopeiaTree | The first tree to compare |
| tree2 | CassiopeiaTree | The second tree to compare |
#### Outputs
| Name | Type | Description |
| :------ | :---- | :-------------------------------------------------------------- |
| rf | float | The Robinson-Foulds distance between the two trees |
| rf\_max | float | The maximum possible Robinson-Foulds distance for normalization |
### `triplets_correct`
#### Description
This function calculates the triplets correct accuracy between two phylogenetic trees. It samples triplets at different depths and compares their structure between the two trees.
#### Inputs
| Name | Type | Description |
| :----------------------- | :------------- | :---------------------------------------------------------------------- |
| tree1 | CassiopeiaTree | The first tree to compare |
| tree2 | CassiopeiaTree | The second tree to compare |
| number\_of\_trials | int | Number of triplets to sample at each depth (default: 1000) |
| min\_triplets\_at\_depth | int | Minimum number of triplets needed at a depth for inclusion (default: 1) |
#### Outputs
| Name | Type | Description |
| :---------------------------- | :---------------- | :------------------------------------------------------- |
| all\_triplets\_correct | Dict\[int, float] | Total triplets correct at each depth |
| resolvable\_triplets\_correct | Dict\[int, float] | Triplets correct for resolvable triplets at each depth |
| unresolved\_triplets\_correct | Dict\[int, float] | Triplets correct for unresolvable triplets at each depth |
| proportion\_resolvable | Dict\[int, float] | Proportion of unresolvable triplets per depth |
## Dependencies
| Dependency | Purpose |
| :-------------------------- | :--------------------------------------------------- |
| cassiopeia.critique.compare | Source of the imported functions for tree comparison |
This file acts as an interface to the tree comparison functionality in the cassiopeia critique module, making it easier for users to access these functions without having to import from deeper within the package structure.
# compare.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/critique/compare.py
Here's a comprehensive documentation for the `compare.py` file in the `cassiopeia/critique` module:
## High-level description
This module provides functions for comparing two phylogenetic trees. It includes implementations for calculating the triplets correct accuracy and the Robinson-Foulds distance between two trees.
## Code Structure
The module contains two main functions: `triplets_correct` and `robinson_foulds`. These functions operate on `CassiopeiaTree` objects, which represent phylogenetic trees. The `triplets_correct` function is more complex, involving sampling and comparison of triplets at different depths of the trees, while the `robinson_foulds` function is a wrapper around the Ete3 library's implementation of the Robinson-Foulds distance.
## Symbols
### `triplets_correct`
#### Description
This function calculates the triplets correct accuracy between two trees. It samples triplets (sets of three leaves) at each depth of the tree and compares their topology between the two input trees.
#### Inputs
| Name | Type | Description |
| :----------------------- | :------------- | :---------------------------------------------------------------------- |
| tree1 | CassiopeiaTree | The first input tree |
| tree2 | CassiopeiaTree | The second input tree to be compared |
| number\_of\_trials | int | Number of triplets to sample at each depth (default: 1000) |
| min\_triplets\_at\_depth | int | Minimum number of triplets needed at a depth for inclusion (default: 1) |
#### Outputs
| Name | Type | Description |
| :---------------------------- | :---------------- | :------------------------------------------------------------ |
| all\_triplets\_correct | Dict\[int, float] | The proportion of all triplets correct at each depth |
| resolvable\_triplets\_correct | Dict\[int, float] | The proportion of resolvable triplets correct at each depth |
| unresolved\_triplets\_correct | Dict\[int, float] | The proportion of unresolvable triplets correct at each depth |
| proportion\_unresolvable | Dict\[int, float] | The proportion of unresolvable triplets at each depth |
#### Internal Logic
1. Initialize dictionaries to store results
2. Create copies of input trees and collapse unifurcations
3. Annotate tree depths and compute number of triplets at each depth
4. For each depth:
* Sample triplets and compare their topology between the two trees
* Calculate various statistics (all, resolvable, unresolved triplets correct)
5. Return the computed statistics
### `robinson_foulds`
#### Description
This function computes the Robinson-Foulds distance between two trees. It's a wrapper around the Ete3 library's implementation.
#### Inputs
| Name | Type | Description |
| :---- | :------------- | :----------------------------------- |
| tree1 | CassiopeiaTree | The first input tree |
| tree2 | CassiopeiaTree | The second input tree to be compared |
#### Outputs
| Name | Type | Description |
| :------ | :---- | :------------------------------------------------------------ |
| rf | float | The Robinson-Foulds distance between the two trees |
| rf\_max | float | The maximum possible Robinson-Foulds distance for these trees |
#### Internal Logic
1. Create copies of input trees and collapse unifurcations
2. Convert trees to Ete3 format
3. Compute Robinson-Foulds distance using Ete3's implementation
4. Return the distance and maximum possible distance
## Dependencies
| Dependency | Purpose |
| :-------------------------------------- | :---------------------------------------------------------------- |
| collections | Used for defaultdict |
| copy | Used for deep copying trees |
| ete3 | Used for Robinson-Foulds distance calculation |
| networkx | Not directly used in this file, but likely used in CassiopeiaTree |
| numpy | Used for numerical operations |
| typing | Used for type hinting |
| cassiopeia.critique.critique\_utilities | Used for utility functions |
| cassiopeia.data.CassiopeiaTree | Used as the main data structure for trees |
## Error Handling
The code does not implement explicit error handling. It relies on the underlying libraries and data structures to raise appropriate exceptions.
This module provides essential functionality for comparing phylogenetic trees in the Cassiopeia project. The `triplets_correct` function offers a detailed comparison of tree topologies at different depths, while the `robinson_foulds` function provides a standard distance metric for tree comparison. These functions are likely used in various analysis and validation tasks within the larger Cassiopeia framework.
# critique_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/critique/critique_utilities.py
## High-level description
This file, `critique_utilities.py`, provides utility functions for the `critique` module in the Cassiopeia library. These functions are primarily used for analyzing and comparing phylogenetic trees, particularly in the context of lineage tracing experiments. They include functions for calculating combinatorial terms, annotating tree depths, identifying outgroups in triplets, and sampling triplets from a tree.
## Code Structure
The code consists of several independent functions that operate on `CassiopeiaTree` objects. These functions are used for tasks such as annotating tree depths, sampling triplets, and identifying outgroups.
## Symbols
### `nCr`
#### Description
This function calculates the binomial coefficient, which represents the number of ways to choose `r` items from a set of `n` items without replacement.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :----------------------------- |
| n | int | The total number of items. |
| r | int | The number of items to choose. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :----------------------------------------------------------------- |
| nCr | float | The binomial coefficient, representing the number of combinations. |
#### Internal Logic
The function first checks for invalid input values (r > n or n \< 0 or r \< 0) and returns 0 if any are found. Otherwise, it calculates the factorial of n, r, and (n-r) using the `math.factorial` function and returns their quotient as the binomial coefficient.
### `annotate_tree_depths`
#### Description
This function annotates each node in a `CassiopeiaTree` with its depth from the root and the number of triplets rooted at that node.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :-------------------------- |
| tree | CassiopeiaTree | The input tree to annotate. |
#### Outputs
| Name | Type | Description |
| :--------------- | :---------------- | :---------------------------------------------------------------- |
| depth\_to\_nodes | defaultdict(list) | A dictionary mapping each depth to a list of nodes at that depth. |
#### Internal Logic
The function performs a depth-first traversal of the tree. For each node, it calculates its depth based on its parent's depth and stores it as a node attribute named "depth". It also calculates the number of triplets rooted at that node by subtracting the number of triplets rooted at its children from the total number of triplets possible with the leaves under the current node. This information is stored as a node attribute named "number\_of\_triplets".
### `get_outgroup`
#### Description
This function infers the outgroup of a given triplet of leaves in a `CassiopeiaTree`.
#### Inputs
| Name | Type | Description |
| :------ | :-------------------- | :-------------------------------------------------------------------- |
| tree | CassiopeiaTree | The input tree. |
| triplet | Tuple\[str, str, str] | A tuple containing the names of the three leaves forming the triplet. |
#### Outputs
| Name | Type | Description |
| :--------- | :--- | :--------------------------------------------------------------------- |
| out\_group | str | The name of the outgroup leaf, or "None" if the triplet is unresolved. |
#### Internal Logic
The function determines the outgroup by finding the pair of leaves within the triplet that share the most recent common ancestor (MRCA). The leaf that is not part of this pair is considered the outgroup. The MRCA is inferred based on the number of shared ancestors between each pair of leaves.
### `sample_triplet_at_depth`
#### Description
This function samples a triplet of leaves from a `CassiopeiaTree` such that the most recent common ancestor (MRCA) of the triplet is at a specified depth.
#### Inputs
| Name | Type | Description |
| :--------------- | :-------------------------------- | :-------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The input tree. |
| depth | int | The desired depth for the MRCA of the sampled triplet. |
| depth\_to\_nodes | Optional\[Dict\[int, List\[str]]] | An optional dictionary mapping depths to lists of nodes at that depth. Providing this dictionary can speed up the function. |
#### Outputs
| Name | Type | Description |
| :-------------------- | :--------------------------------- | :--------------------------------------------------------------------------------------------------- |
| (triplet, out\_group) | Tuple\[Tuple\[str, str, str], str] | A tuple containing the sampled triplet (as a tuple of leaf names) and the name of the outgroup leaf. |
#### Internal Logic
The function first identifies the nodes at the specified depth. It then samples a node from these candidates with probability proportional to the number of triplets rooted at each node. Once a node is chosen, the function samples three of its descendant clades, ensuring that at least two clades are distinct. Finally, it randomly selects one leaf from each of the chosen clades to form the triplet and identifies the outgroup based on the chosen clades.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/critique/overview
## High-level description
The `cassiopeia/critique` directory contains modules for comparing and analyzing phylogenetic trees, particularly in the context of lineage tracing experiments. It provides implementations for calculating tree similarity metrics, such as the Robinson-Foulds distance and triplets correct accuracy, as well as utility functions for tree manipulation and analysis.
## What does it do?
This directory provides tools for researchers and developers working with phylogenetic trees to:
1. Compare two phylogenetic trees using different metrics:
* Robinson-Foulds distance: A standard metric for measuring the structural difference between two trees.
* Triplets correct accuracy: A more detailed comparison that examines the topology of sampled triplets (sets of three leaves) at different depths of the trees.
2. Perform various tree analysis tasks:
* Annotate tree depths and calculate the number of triplets rooted at each node.
* Identify outgroups in triplets of leaves.
* Sample triplets from specific depths in a tree.
3. Provide utility functions for combinatorial calculations and tree manipulations.
These tools are essential for validating and comparing reconstructed phylogenetic trees, which is crucial in lineage tracing experiments and other applications involving evolutionary tree analysis.
## Entry points
The main entry point for this directory is the `__init__.py` file, which exposes two key functions:
1. `robinson_foulds`: Computes the Robinson-Foulds distance between two phylogenetic trees.
2. `triplets_correct`: Calculates the triplets correct accuracy between two phylogenetic trees.
These functions are imported from the `compare.py` module, which contains their implementations. The `critique_utilities.py` file provides supporting functions used by the main comparison algorithms.
The typical workflow would involve:
1. Importing the desired comparison function from `cassiopeia.critique`.
2. Creating or loading two `CassiopeiaTree` objects to compare.
3. Calling the comparison function with the two trees as arguments.
4. Analyzing the results to determine the similarity or differences between the trees.
## Key Files
1. `compare.py`: This file contains the main implementations of the tree comparison algorithms:
* `triplets_correct`: A detailed comparison function that samples triplets at different depths and compares their topology between two trees.
* `robinson_foulds`: A wrapper around the Ete3 library's implementation of the Robinson-Foulds distance calculation.
2. `critique_utilities.py`: This file provides utility functions used in tree analysis and comparison:
* `nCr`: Calculates binomial coefficients.
* `annotate_tree_depths`: Annotates each node in a tree with its depth and the number of triplets rooted at that node.
* `get_outgroup`: Infers the outgroup of a given triplet of leaves in a tree.
* `sample_triplet_at_depth`: Samples a triplet of leaves from a tree with a specified most recent common ancestor depth.
These files work together to provide a comprehensive set of tools for tree comparison and analysis. The `compare.py` file relies on the utility functions in `critique_utilities.py` to perform its calculations efficiently.
## Dependencies
The `critique` module relies on several external libraries and internal Cassiopeia components:
1. External libraries:
* `collections`: Used for `defaultdict` data structure.
* `copy`: Used for deep copying trees.
* `ete3`: Used for Robinson-Foulds distance calculation.
* `networkx`: Likely used in the `CassiopeiaTree` implementation.
* `numpy`: Used for numerical operations.
* `typing`: Used for type hinting.
* `math`: Used for factorial calculations in combinatorial functions.
2. Internal dependencies:
* `cassiopeia.data.CassiopeiaTree`: The main data structure representing phylogenetic trees.
These dependencies are chosen to provide efficient data structures (e.g., `defaultdict`), numerical operations (numpy), and specialized tree operations (ete3). The use of type hinting (typing) suggests a focus on code clarity and potential use of static type checking tools.
In conclusion, the `cassiopeia/critique` directory provides a set of powerful tools for comparing and analyzing phylogenetic trees, with a focus on efficiency and accuracy. It leverages both standard Python libraries and specialized scientific computing packages to deliver robust tree comparison functionality, which is crucial for validating and interpreting results in lineage tracing experiments and other phylogenetic studies.
# CassiopeiaTree.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/data/CassiopeiaTree.py
## High-level description
The `CassiopeiaTree` class is the fundamental data structure in Cassiopeia, representing a phylogenetic tree of a clonal population. It stores a tree topology, a character matrix with mutation states for each cell, and metadata associated with cells and characters. The class provides methods for manipulating the tree, reconstructing ancestral characters, computing dissimilarities, and accessing various tree properties.
## Code Structure
The `CassiopeiaTree` class is the main symbol in the code. It holds references to other data structures like `character_matrix`, `cell_meta`, `character_meta`, `priors`, and `dissimilarity_map`. The class methods operate on these data structures and the underlying tree topology represented as a Networkx DiGraph.
## References
The code references several utility functions from `cassiopeia.data.utilities` for tasks like converting between tree formats, computing LCA characters, and calculating dissimilarity maps. It also uses the `Layers` class from `cassiopeia.data.Layers` to manage multiple versions of the character matrix.
## Symbols
### `CassiopeiaTree`
#### Description
This class represents a phylogenetic tree of a clonal population. It stores the tree topology, character matrix, metadata, and provides methods for manipulating the tree and accessing its properties.
#### Inputs
| Name | Type | Description |
| :------------------------ | :-------------------------------------------- | :----------------------------------------------------------------------------------------- |
| character\_matrix | Optional\[pd.DataFrame] | Character matrix of mutation observations. |
| missing\_state\_indicator | int | An indicator for missing states in the character matrix. Defaults to -1. |
| cell\_meta | Optional\[pd.DataFrame] | Per-cell metadata. |
| character\_meta | Optional\[pd.DataFrame] | Per-character metadata. |
| priors | Optional\[Dict\[int, Dict\[int, float]]] | A dictionary storing the probability of each character mutating to a particular state. |
| tree | Optional\[Union\[str, ete3.Tree, nx.DiGraph]] | A tree for the lineage, specified as a Networkx DiGraph, a newick string, or an ete3 Tree. |
| dissimilarity\_map | Optional\[pd.DataFrame] | An NxN dataframe storing the pairwise dissimilarities between samples. |
| parameters | Optional\[Dict\[str, Any]] | A dictionary storing parameters related to the tree. |
| root\_sample\_name | Optional\[str] | The name of the sample to treat as the root. |
#### Outputs
The class itself is the output, representing a populated CassiopeiaTree object.
#### Internal Logic
The class initializes its attributes and optionally populates the tree if provided. It also sets up a cache for storing computed values. The class methods provide various functionalities for manipulating the tree, reconstructing ancestral characters, computing dissimilarities, and accessing tree properties.
## Side Effects
The class methods can modify the internal state of the `CassiopeiaTree` object, including the tree topology, character matrix, metadata, and dissimilarity map.
## Dependencies
| Dependency | Purpose |
| :---------------------------------- | :-------------------------------------------------- |
| pandas | Data manipulation and storage |
| numpy | Numerical operations |
| networkx | Tree representation and manipulation |
| ete3 | Tree parsing and manipulation |
| scipy | Scientific computing |
| collections | Data structures |
| copy | Deep copying |
| warnings | Warning handling |
| typing | Type hinting |
| cassiopeia.data.utilities | Utility functions for data manipulation |
| cassiopeia.data.Layers | Layers class for managing character matrix versions |
| cassiopeia.mixins | Mixins for error and warning handling |
| cassiopeia.solver.solver\_utilities | Utility functions for solvers |
## Error Handling
The code raises `CassiopeiaTreeError` for various invalid inputs or operations, such as an uninitialized tree, mismatched character matrix and leaves, or negative branch lengths.
## TODOs
* Add check upon initialization that input tree is valid tree.
* Add experimental meta data as arguments.
* Add utility methods to compute the colless index and the cophenetic correlation wrt to some cell meta item
* Add bulk set\_states method.
* Add boolean to `get_tree_topology` which will include all attributes (e.g., node times)
# Layers.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/data/Layers.py
Here's a detailed documentation for the `Layers` class in the `cassiopeia/data/Layers.py` file:
## High-level description
The `Layers` class implements a data structure for storing multiple versions of mutation data for each cell in a CassiopeiaTree object. It behaves like a dictionary, allowing users to store, retrieve, and delete entries using standard dictionary operations.
## Code Structure
The `Layers` class is a subclass of `dict` and is closely tied to the `CassiopeiaTree` class. It maintains a reference to its parent `CassiopeiaTree` object and stores layer data in an internal dictionary.
## Symbols
### Layers
#### Description
A class that represents a collection of character matrices, each storing a version of mutation data for cells in a CassiopeiaTree.
#### Inputs
| Name | Type | Description |
| :----- | :----------------- | :------------------------------------ |
| parent | CassiopeiaTree | The parent CassiopeiaTree object |
| layers | Optional\[Mapping] | Initial layers to populate the object |
#### Internal Logic
* Initializes with a reference to the parent CassiopeiaTree and an empty internal dictionary.
* If initial layers are provided, they are added to the internal dictionary.
* Implements dictionary-like methods for accessing, setting, and deleting layers.
* Provides custom string representation and iPython key completion.
### **repr**
#### Description
Returns a string representation of the Layers object.
#### Outputs
| Name | Type | Description |
| :----- | :--- | :----------------------------------------- |
| return | str | String representation of the Layers object |
### *ipython\_key\_completions*
#### Description
Returns a list of keys for iPython autocompletion.
#### Outputs
| Name | Type | Description |
| :----- | :--------- | :----------------- |
| return | List\[str] | List of layer keys |
### copy
#### Description
Creates a shallow copy of the Layers object.
#### Outputs
| Name | Type | Description |
| :----- | :----- | :----------------------------------- |
| return | Layers | A new Layers object with copied data |
### **getitem**
#### Description
Retrieves a layer by key.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------------- |
| key | str | The key of the layer to retrieve |
#### Outputs
| Name | Type | Description |
| :----- | :----------- | :----------------------- |
| return | pd.DataFrame | The requested layer data |
### **setitem**
#### Description
Sets a layer with the given key and value.
#### Inputs
| Name | Type | Description |
| :---- | :----------- | :-------------------- |
| key | str | The key for the layer |
| value | pd.DataFrame | The layer data to set |
#### Internal Logic
* Validates the input value using the `_validate_value` method.
* Stores the validated value in the internal dictionary.
### **delitem**
#### Description
Deletes a layer with the given key.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :----------------------------- |
| key | str | The key of the layer to delete |
### **contains**
#### Description
Checks if a layer with the given key exists.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------- |
| key | str | The key to check for |
#### Outputs
| Name | Type | Description |
| :----- | :--- | :-------------------------------------- |
| return | bool | True if the key exists, False otherwise |
### **iter**
#### Description
Returns an iterator over the layer keys.
#### Outputs
| Name | Type | Description |
| :----- | :------------- | :----------------------- |
| return | Iterator\[str] | Iterator over layer keys |
### **len**
#### Description
Returns the number of layers.
#### Outputs
| Name | Type | Description |
| :----- | :--- | :--------------- |
| return | int | Number of layers |
### \_validate\_value
#### Description
Validates the input value for a layer.
#### Inputs
| Name | Type | Description |
| :--- | :----------- | :-------------------------------- |
| val | pd.DataFrame | The value to validate |
| key | str | The key associated with the value |
#### Outputs
| Name | Type | Description |
| :----- | :----------- | :------------------ |
| return | pd.DataFrame | The validated value |
#### Internal Logic
* Checks if the input value has the correct number of samples (rows) compared to the parent CassiopeiaTree.
* Raises a ValueError if the shape is incorrect.
## Dependencies
| Dependency | Purpose |
| :----------------------------- | :--------------------- |
| typing | Type hinting |
| pandas | DataFrame operations |
| cassiopeia.data.CassiopeiaTree | Parent class reference |
This `Layers` class provides a flexible way to store and manage multiple versions of mutation data for cells in a CassiopeiaTree object, allowing for easy experimentation with different data representations or imputations.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/data/__init__.py
Here's a documentation for the target file `cassiopeia/data/__init__.py`:
## High-level description
This file serves as the top-level initialization for the `cassiopeia.data` module. It imports and exposes key classes and utility functions from the module, making them easily accessible to users of the Cassiopeia package.
## Symbols
### `CassiopeiaTree`
#### Description
Imports the `CassiopeiaTree` class from the `CassiopeiaTree` module. This class is the core data structure in Cassiopeia, representing a phylogenetic tree with associated character matrices and metadata.
### Utility Functions
#### Description
Imports several utility functions from the `utilities` module. These functions provide various operations and computations related to phylogenetic trees and character matrices.
#### Imported Functions:
1. `compute_dissimilarity_map`: Computes pairwise dissimilarities between samples in a character matrix.
2. `compute_inter_cluster_distances`: Calculates mean distances between categories in a categorical variable.
3. `compute_phylogenetic_weight_matrix`: Computes distances or proximities between all leaves in a tree.
4. `get_lca_characters`: Builds the character vector of the lowest common ancestor (LCA) of a list of character vectors.
5. `net_relatedness_index`: Computes the net relatedness index between two groups of indices.
6. `sample_bootstrap_allele_tables`: Generates bootstrap character matrices from an allele table.
7. `sample_bootstrap_character_matrices`: Creates bootstrapped character matrices from a given character matrix.
8. `to_newick`: Converts a networkx graph to a Newick string representation.
## Dependencies
| Dependency | Purpose |
| :------------- | :------------------------------------------ |
| CassiopeiaTree | Core data structure for phylogenetic trees |
| utilities | Module containing various utility functions |
Note: This file acts as a convenient entry point for users of the Cassiopeia package, allowing them to import commonly used classes and functions directly from `cassiopeia.data` without needing to know the exact submodule structure.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/data/overview
## High-level description
The `cassiopeia/data` directory houses the core data structures and utility functions for representing and manipulating lineage tracing data in the Cassiopeia framework. This includes the fundamental `CassiopeiaTree` class, which encapsulates a phylogenetic tree with associated character matrices, metadata, and methods for tree manipulation and analysis. Additionally, this directory provides utility functions for tasks like data conversion, dissimilarity computation, bootstrapping, and tree traversal.
## What does it do?
This directory provides the building blocks for working with lineage tracing data in Cassiopeia. It allows researchers to:
1. **Represent phylogenetic trees:** The `CassiopeiaTree` class stores the tree topology, character matrix (mutations observed in cells), and metadata associated with cells and characters.
2. **Manipulate tree data:** The `CassiopeiaTree` class offers methods for modifying the tree structure, reconstructing ancestral states, and accessing tree properties.
3. **Compute dissimilarity:** Utility functions calculate pairwise dissimilarities between cells based on their character states, enabling distance-based analyses.
4. **Bootstrap data:** Functions like `sample_bootstrap_character_matrices` generate bootstrapped versions of character matrices for statistical analysis.
5. **Convert data formats:** Utilities handle the conversion between different data representations, such as from allele tables to character matrices.
## Entry points
The main entry point for users is the `CassiopeiaTree` class (`cassiopeia/data/CassiopeiaTree.py`), which serves as the primary interface for interacting with lineage tracing data. The utility functions in `cassiopeia/data/utilities.py` are typically called by the `CassiopeiaTree` methods or used directly for specific data manipulation tasks.
## Key Files
### cassiopeia/data/CassiopeiaTree.py
This file defines the `CassiopeiaTree` class, the fundamental data structure in Cassiopeia. It stores and manages:
* Tree topology (as a Networkx DiGraph)
* Character matrix (mutations observed in each cell)
* Cell and character metadata
* Dissimilarity map (pairwise distances between cells)
The class provides methods for:
* Accessing and modifying tree components
* Reconstructing ancestral character states
* Computing dissimilarity matrices
* Exporting the tree in various formats
### cassiopeia/data/utilities.py
This file contains a collection of utility functions for:
* Converting between different data formats (e.g., allele tables to character matrices)
* Computing dissimilarity maps using various distance metrics
* Bootstrapping character matrices
* Performing tree traversals and manipulations
* Calculating phylogenetic distances and weights
These functions are used internally by `CassiopeiaTree` methods and can also be called directly for specific data manipulation tasks.
### cassiopeia/data/Layers.py
This file defines the `Layers` class, which allows storing and managing multiple versions of character matrices within a `CassiopeiaTree` object. This is useful for:
* Simulations involving imputation of missing data
* Storing different representations or versions of the character matrix
The `Layers` class acts as a dictionary, where keys represent different layers and values are the corresponding character matrices.
## Dependencies
The `cassiopeia/data` directory depends on several external libraries:
* **pandas:** For data manipulation and storage of character matrices and metadata.
* **numpy:** For numerical operations, particularly in dissimilarity calculations.
* **networkx:** For representing and manipulating the tree topology.
* **ete3:** For parsing and manipulating trees in Newick format.
* **scipy:** For scientific computing, including distance calculations and phylogenetic analysis.
It also relies on other modules within the Cassiopeia package, such as `cassiopeia.mixins` for error handling and `cassiopeia.solver.solver_utilities` for solver-related utilities.
# utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/data/utilities.py
Here's a high-level description of the `utilities.py` file in the `cassiopeia/data` directory:
This file contains utility functions for working with Cassiopeia datasets, particularly for manipulating and analyzing character matrices, allele tables, and phylogenetic trees. It includes functions for bootstrapping, converting between different data formats, computing dissimilarity maps, and various other operations on tree structures and genetic data.
## Code Structure
The file consists of standalone functions that can be grouped into several categories:
1. Bootstrapping functions
2. Data conversion functions
3. Tree manipulation functions
4. Dissimilarity and distance computation functions
5. Utility functions for working with character states and indels
These functions are used throughout the Cassiopeia package to support various data processing and analysis tasks.
## Symbols
Here are some of the key functions in this file:
### `sample_bootstrap_character_matrices`
#### Description
Creates bootstrap samples of character matrices by sampling characters with replacement.
#### Inputs
* `character_matrix`: The original character matrix
* `prior_probabilities`: Optional prior probabilities for characters
* `num_bootstraps`: Number of bootstrap samples to create
* `random_state`: Optional random state for reproducibility
#### Outputs
A list of tuples, each containing a bootstrapped character matrix and corresponding priors.
### `convert_alleletable_to_character_matrix`
#### Description
Converts an allele table to a character matrix format.
#### Inputs
* `alleletable`: The input allele table
* Various optional parameters for filtering and processing
#### Outputs
A tuple containing the character matrix, prior probabilities, and a mapping of states to indels.
### `compute_phylogenetic_weight_matrix`
#### Description
Computes a phylogenetic weight matrix based on the distances between leaves in a tree.
#### Inputs
* `tree`: A CassiopeiaTree object
* `inverse`: Whether to compute inverse weights
* `inverse_fn`: Function to use for inverse computation
#### Outputs
A pandas DataFrame representing the phylogenetic weight matrix.
### `net_relatedness_index`
#### Description
Computes the net relatedness index between two groups of indices in a dissimilarity map.
#### Inputs
* `dissimilarity_map`: A numpy array of dissimilarities
* `indices_1`: First group of indices
* `indices_2`: Second group of indices
#### Outputs
The computed net relatedness index as a float.
## Dependencies
The file relies on several external libraries, including:
* numpy
* pandas
* networkx
* scipy
* ete3
It also imports from other parts of the Cassiopeia package, particularly from the `mixins` and `preprocess` modules.
This file is central to many data processing tasks in Cassiopeia, providing essential utilities for working with genetic data and phylogenetic trees.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/__init__.py
## High-level description
This file serves as a central hub for importing various mixins, utilities, and custom exceptions used within the Cassiopeia library. It simplifies the process of accessing these components from other parts of the codebase.
## Code Structure
This file acts as a simple aggregator of symbols from other modules. It doesn't define any complex interconnections between symbols.
## References
This file imports symbols from the following files:
* `cassiopeia/mixins/errors.py`
* `cassiopeia/mixins/logging.py`
* `cassiopeia/mixins/utilities.py`
* `cassiopeia/mixins/warnings.py`
## Symbols
### Symbol Name: `*` from `cassiopeia/mixins/errors.py`
#### Description:
Imports all custom exception classes defined in `errors.py`. These exceptions are likely used for specific error conditions within the Cassiopeia library.
### Symbol Name: `logger` from `cassiopeia/mixins/logging.py`
#### Description:
Imports the `logger` object, which is presumably a configured logging instance used for recording events and messages within the Cassiopeia library.
### Symbol Name: `*` from `cassiopeia/mixins/utilities.py`
#### Description:
Imports all utility functions defined in `utilities.py`. These functions likely provide helper functionality used across different parts of the Cassiopeia library.
### Symbol Name: `*` from `cassiopeia/mixins/warnings.py`
#### Description:
Imports all custom warning classes defined in `warnings.py`. These warnings are likely used to signal potential issues or non-critical errors within the Cassiopeia library.
# errors.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/errors.py
## High-level description
This file defines a set of custom exception classes for the Cassiopeia software. These exceptions are used to handle various error conditions that may arise during the execution of the Cassiopeia pipeline, such as autocorrelation errors, tree errors, data simulator errors, distance solver errors, and more.
## Code Structure
The code consists of a series of custom exception classes, each inheriting from the base `Exception` class. These classes are named according to the specific error condition they represent, such as `AutocorrelationError`, `CassiopeiaTreeError`, `DataSimulatorError`, etc. Each class has a simple `pass` statement, indicating that they do not have any additional logic or attributes beyond the basic exception handling.
## Symbols
### `AutocorrelationError`
#### Description
An exception class for errors encountered in the `tools.autocorrelation` module.
### `CassiopeiaError`
#### Description
A general exception class for the Cassiopeia software.
### `CassiopeiaTreeError`
#### Description
An exception class for errors encountered in the `CassiopeiaTree` class. This class is used to handle errors related to tree manipulation, character state assignments, and other tree-related operations.
### `DataSimulatorError`
#### Description
A generic exception class for errors encountered in the `DataSimulator` subclasses. This class is used to handle errors related to data simulation, such as invalid parameters or data inconsistencies.
### `DistanceSolverError`
#### Description
An exception class for errors encountered in the `DistanceSolver` subclasses. This class is used to handle errors related to distance-based tree solving algorithms, such as Neighbor-Joining and UPGMA.
### `ecDNABirthDeathSimulatorError`
#### Description
An exception class for errors encountered in the `ecDNABirthDeathSimulator` class. This class is used to handle errors specific to the simulation of trees with extrachromosomal DNA (ecDNA).
### `FitchCountError`
#### Description
An exception class for errors encountered in the `FitchCount` algorithm. This algorithm is used to infer the number of transitions between categorical variables (e.g., tissues) in a tree.
### `GreedySolverError`
#### Description
An exception class for errors encountered in the `GreedySolver` class. This class represents the structure of top-down algorithms that build the reconstructed tree by recursively splitting the set of samples based on some split criterion.
### `HybridSolverError`
#### Description
An exception class for errors encountered in the `HybridSolver` subclasses. This class represents a hybrid approach to tree solving, combining a greedy algorithm with a more complex algorithm for subproblems.
### `ILPSolverError`
#### Description
An exception class for errors encountered in the `ILPSolver` subclasses. This class represents an integer linear programming (ILP) approach to tree solving.
### `iTOLError`
#### Description
An exception class for errors encountered when interacting with the iTOL (Interactive Tree Of Life) software.
### `LeafSubsamplerError`
#### Description
An exception class for errors encountered in the `LeafSubsampler` class. This class is used to handle errors related to subsampling leaves from a tree.
### `PreprocessError`
#### Description
An exception class for errors encountered during the preprocessing stage of the Cassiopeia pipeline.
### `PriorTransformationError`
#### Description
An exception class for errors encountered when transforming prior probabilities into weights.
### `SharedMutationJoiningSolverError`
#### Description
An exception class for errors encountered in the `SharedMutationJoiningSolver` class. This class represents an agglomerative clustering procedure that joins samples based on shared mutations.
### `TreeSimulatorError`
#### Description
An exception class for errors encountered in the `TreeSimulator` subclasses. This class is used to handle errors related to tree simulation, such as invalid parameters or stopping conditions.
### `UnknownCigarStringError`
#### Description
An exception class for errors encountered when parsing an unknown CIGAR string. A CIGAR string is a compact representation of an alignment between two sequences.
### `UnspecifiedConfigParameterError`
#### Description
An exception class for errors encountered when a required configuration parameter is not specified.
### `BranchLengthEstimatorError`
#### Description
An exception class for errors encountered in the `BranchLengthEstimator` class. This class is used to estimate branch lengths on a tree.
### `IIDExponentialMLEError`
#### Description
An exception class for errors encountered in the `IIDExponentialMLE` class, a subclass of `BranchLengthEstimator`. This class estimates branch lengths under a model of independent and identically distributed (IID) exponential waiting times for mutations.
### `TreeMetricError`
#### Description
An exception class for errors encountered when calculating tree metrics.
### `ParameterEstimateError`
#### Description
An exception class for errors encountered when estimating tree parameters.
### `PlottingError`
#### Description
An exception class for errors encountered when plotting trees.
# logging.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/logging.py
## High-level description
This code sets up a logger for the `cassiopeia.mixins` module using the `ngs_tools` library. It defines a logger object, sets its name to the module's name, and sets the logging level to INFO.
## References
* `ngs_tools`: This code directly uses the `ngs_tools` library for logging.
## Symbols
### `logger`
#### Description
This symbol is a logger object created using the `ngs.logging.Logger` class from the `ngs_tools` library. It is used for logging messages during the execution of the code within the `cassiopeia.mixins` module.
#### Inputs
The `logger` object itself doesn't take any direct inputs in this code snippet. However, it implicitly receives log messages from other parts of the `cassiopeia.mixins` module.
#### Outputs
The `logger` object doesn't have any direct outputs. Instead, it sends log messages to the configured output handlers, which are responsible for writing the messages to the appropriate destinations (e.g., console, file).
#### Internal Logic
The `logger` object's internal logic is handled by the `ngs_tools` library. It receives log messages from the `cassiopeia.mixins` module and, based on its configuration (name and logging level), determines whether to pass them on to the output handlers.
#### Side Effects
* **Logging**: The `logger` object, when invoked with log messages, will write those messages to the configured output handlers. This is the primary side effect of this symbol.
## Dependencies
* `ngs_tools`: This library provides the logging functionality used in the code.
## Logging
This code implements logging using the `ngs_tools` library. The `logger` object is configured to log messages at the INFO level, meaning that messages with severity INFO or higher (WARNING, ERROR, CRITICAL) will be logged. The specific output destinations for the log messages are not defined in this code snippet and are likely configured elsewhere in the `ngs_tools` library or in the code that utilizes this library.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/overview
## High-level description
The `cassiopeia/mixins` directory contains utility modules that provide essential functionality for the Cassiopeia library. These modules include custom exceptions, logging setup, utility functions, and warning classes. The directory serves as a central location for shared components that are used across different parts of the Cassiopeia codebase.
## What does it do?
The `mixins` directory provides several key functionalities:
1. Error Handling: It defines a comprehensive set of custom exception classes tailored to different components of Cassiopeia, allowing for more specific and informative error reporting.
2. Logging: It sets up a logging system using the `ngs_tools` library, enabling consistent and configurable logging throughout the Cassiopeia library.
3. Utility Functions: It offers a collection of helper functions for working with character matrices and trees, including operations like handling ambiguous states and finding duplicate groups in data.
4. Warning System: It defines custom warning classes for various components of Cassiopeia, allowing for more specific and contextual warnings to be raised during execution.
These components work together to enhance the robustness, debuggability, and user-friendliness of the Cassiopeia library. They provide a foundation for error handling, logging, and common operations that are used across different modules of the library.
## Key Files
1. `__init__.py`: This file serves as the entry point for the `mixins` package. It imports and exposes symbols from the other modules in the directory, making them easily accessible to other parts of the Cassiopeia library.
2. `errors.py`: This file defines a comprehensive set of custom exception classes. These exceptions are tailored to specific error conditions that may occur in different components of Cassiopeia, such as tree operations, data simulation, and various solver algorithms.
3. `logging.py`: This file sets up the logging system for Cassiopeia using the `ngs_tools` library. It creates a logger object configured to log messages at the INFO level and above.
4. `utilities.py`: This file contains a collection of utility functions that are used across the Cassiopeia library. These functions include operations for handling ambiguous states in character matrices, importing modules safely, and finding duplicate groups in data.
5. `warnings.py`: This file defines custom warning classes for different components of Cassiopeia. These warnings allow for more specific and contextual warning messages to be raised during the execution of various library functions.
## Dependencies
The `mixins` directory relies on the following external libraries:
1. `ngs_tools`: Used for setting up the logging system.
2. `functools`: Used in utility functions for operations like reducing lists.
3. `importlib`: Used for dynamically importing modules in utility functions.
4. `numpy`: Used in utility functions for array operations.
5. `pandas`: Used in utility functions for data manipulation.
These dependencies are used to provide efficient and robust implementations of various utility functions and logging capabilities.
## Configuration
The `mixins` directory does not directly use configuration files or environment variables. However, the logging setup in `logging.py` configures the logger to use the INFO level. This configuration might be further customized in other parts of the Cassiopeia library that use this logger.
In summary, the `cassiopeia/mixins` directory provides a set of essential tools and utilities that form the backbone of error handling, logging, and common operations in the Cassiopeia library. These components enhance the overall functionality, reliability, and maintainability of the library by providing consistent mechanisms for handling errors, logging events, and performing common data operations.
# utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/utilities.py
## High-level description
This file contains a set of utility functions for working with character matrices and trees in Cassiopeia. These functions include importing modules, checking for ambiguous states, unraveling ambiguous states, finding duplicate groups in a character matrix, and other helper functions.
## Code Structure
The code consists of a set of independent utility functions that can be used in various parts of the Cassiopeia codebase. Some functions are used by other functions, such as `is_ambiguous_state` being used by `unravel_ambiguous_states` and `find_duplicate_groups`.
## References
This file references the `ngs_tools` and `functools` libraries. It also references the `is_ambiguous_state` function from the `cassiopeia.mixins` module.
## Symbols
### `is_ambiguous_state`
#### Description
Determines if a given state is ambiguous. An ambiguous state is represented as a tuple of integers.
#### Inputs
| Name | Type | Description |
| :---- | :---------------------------- | :------------------------------------------------------------------------ |
| state | Union\[int, Tuple\[int, ...]] | A single character state, which can be an integer or a tuple of integers. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :--------------------------------------------------------------- |
| | bool | True if the state is ambiguous (i.e., a tuple), False otherwise. |
#### Internal Logic
The function simply checks if the input `state` is an instance of a tuple. If it is, it returns True, indicating an ambiguous state. Otherwise, it returns False.
### `try_import`
#### Description
Attempts to import a module and returns it if successful, otherwise returns None. This is useful for handling optional dependencies.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :-------------------------------- |
| module | str | The name of the module to import. |
#### Outputs
| Name | Type | Description |
| :--- | :-------------------- | :------------------------------------------------- |
| | Optional\[ModuleType] | The imported module if successful, otherwise None. |
#### Internal Logic
The function uses a try-except block to attempt importing the module using `importlib.import_module`. If a `ModuleNotFoundError` is raised, it returns None. Otherwise, it returns the imported module.
### `unravel_ambiguous_states`
#### Description
Unravels a list of potentially ambiguous states into a list of unique states.
#### Inputs
| Name | Type | Description |
| :----------- | :----------------------------------- | :----------------------------------------------------------------------------------------- |
| state\_array | List\[Union\[int, Tuple\[int, ...]]] | A list of character states, potentially containing ambiguous states represented as tuples. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :------------------------------------------------------------------ |
| | List\[int] | A list of unique integer states present in the input `state_array`. |
#### Internal Logic
The function iterates through the `state_array`. For each state, if it's ambiguous (a tuple), it converts it to a list. Otherwise, it creates a list containing the single state. Finally, it uses `functools.reduce` to concatenate all the lists into a single list of unique states.
### `find_duplicate_groups`
#### Description
Identifies groups of samples in a character matrix that have identical character states and maps them to a dictionary.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------- | :---------------------------------------------------------- |
| character\_matrix | pd.DataFrame | A character matrix potentially containing ambiguous states. |
#### Outputs
| Name | Type | Description |
| :--- | :--------------------------- | :--------------------------------------------------------------------------------------------------------- |
| | Dict\[str, Tuple\[str, ...]] | A dictionary mapping a single sample name to a tuple of sample names that share the same character states. |
#### Internal Logic
1. Sets the index name of the character matrix to "index".
2. Creates a copy of the character matrix and converts each element to a set to handle ambiguous states.
3. Identifies duplicated rows in the converted matrix using `pd.DataFrame.duplicated`.
4. Finds unique duplicated states.
5. Groups sample names based on unique duplicated states.
6. Creates a dictionary mapping the first sample name in each group to a tuple of all sample names in that group.
## Dependencies
This file depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :---------------------------------------------------------------------------------- |
| functools | Used for reducing a list of lists into a single list in `unravel_ambiguous_states`. |
| importlib | Used for importing modules in `try_import`. |
| numpy | Used for array operations in `find_duplicate_groups`. |
| pandas | Used for data manipulation in `find_duplicate_groups`. |
# warnings.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/mixins/warnings.py
## High-level description
This file defines custom warning classes for various components of the Cassiopeia library. These warning classes inherit from the built-in `UserWarning` class and are used to provide specific warnings related to different aspects of the library's functionality.
## Code Structure
The code consists of several custom warning classes, each inheriting from `UserWarning`. These classes are defined to categorize warnings related to specific parts of the Cassiopeia library.
## Symbols
### `CassiopeiaTreeWarning`
#### Description
A general warning class for the `CassiopeiaTree` class, used to signal potential issues or unexpected behavior within the tree structure or operations.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `DataSimulatorWarning`
#### Description
A warning class for the data simulator component of the library, used to signal potential issues or unexpected behavior during data simulation.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `PreprocessWarning`
#### Description
A warning class for the preprocessing component of the library, used to signal potential issues or unexpected behavior during data preprocessing.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `SharedMutationJoiningSolverWarning`
#### Description
A warning class for the `SharedMutationJoiningSolver` class, used to signal potential issues or unexpected behavior during tree reconstruction using this solver.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `ParameterEstimateWarning`
#### Description
A warning class for the parameter estimation component of the library, used to signal potential issues or unexpected behavior during parameter estimation.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `PlottingWarning`
#### Description
A warning class for the plotting component of the library, used to signal potential issues or unexpected behavior during plotting.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
### `LeafSubsamplerWarning`
#### Description
A warning class for the `LeafSubsampler` class, used to signal potential issues or unexpected behavior during leaf subsampling.
#### Inputs
None
#### Outputs
None
#### Internal Logic
This class simply inherits from `UserWarning` and does not define any additional logic.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/overview
## High-level description
The `cassiopeia` directory contains a comprehensive suite of tools and modules for phylogenetic analysis, particularly focused on single-cell lineage tracing experiments. It provides functionality for data preprocessing, tree reconstruction, simulation, visualization, and various analytical tools for working with phylogenetic trees.
## What does it do?
The Cassiopeia package offers a wide range of functionalities:
1. Data Preprocessing: Converts raw sequencing data to formats suitable for phylogenetic analysis, including filtering, error correction, and allele calling.
2. Tree Reconstruction: Implements various algorithms for reconstructing phylogenetic trees from mutation data, including greedy, distance-based, spectral, and integer linear programming methods.
3. Simulation: Provides tools to generate synthetic phylogenetic trees and associated data, allowing researchers to test and validate analysis methods.
4. Visualization: Offers both local and cloud-based options for visualizing phylogenetic trees and associated data.
5. Analysis Tools: Includes utilities for parameter estimation, fitness calculation, evolutionary coupling analysis, and various tree metrics.
6. Tree Comparison: Implements methods for comparing phylogenetic trees, such as Robinson-Foulds distance and triplets correct accuracy.
These functionalities enable researchers to perform end-to-end analysis of single-cell lineage tracing data, from raw sequencing results to detailed phylogenetic insights.
## Entry points
The main entry points for the Cassiopeia package are:
1. `cassiopeia.preprocess`: For preprocessing raw sequencing data.
2. `cassiopeia.solver`: For reconstructing phylogenetic trees from processed data.
3. `cassiopeia.simulator`: For generating synthetic phylogenetic data.
4. `cassiopeia.plotting`: For visualizing phylogenetic trees and associated data.
5. `cassiopeia.tools`: For various analytical tools and utilities.
6. `cassiopeia.critique`: For comparing and analyzing phylogenetic trees.
The `__init__.py` file in the root directory serves as the main interface, importing and exposing key functionalities from these submodules.
## Key Files
1. `data/CassiopeiaTree.py`: Defines the core data structure for representing phylogenetic trees with associated mutation data.
2. `solver/CassiopeiaSolver.py`: Provides the base class for tree reconstruction algorithms.
3. `simulator/TreeSimulator.py`: Offers the base class for tree simulation models.
4. `plotting/local.py` and `plotting/itol_utilities.py`: Implement local and cloud-based tree visualization.
5. `tools/parameter_estimators.py`: Contains functions for estimating key parameters from tree data.
6. `critique/compare.py`: Implements tree comparison algorithms.
## Dependencies
Cassiopeia relies on several external libraries:
1. numpy and pandas: For data manipulation and numerical computations.
2. networkx: For graph operations and tree manipulations.
3. scipy: For various scientific computing tasks.
4. ete3: For phylogenetic tree manipulation and visualization.
5. matplotlib and plotly: For local plotting.
6. pysam: For handling sequencing data formats.
7. cvxpy and gurobipy: For optimization problems in some solvers.
## Configuration
Cassiopeia uses configuration files and function parameters for customization:
1. Preprocessing pipeline: Configured using an INI-format file specifying parameters for each preprocessing step.
2. Solvers: Customizable through parameters passed during initialization.
3. Simulators: Configurable via parameters set during object creation.
4. Visualization: Customizable through function parameters for both local and cloud-based plotting.
Users can adjust these configurations to adapt Cassiopeia to their specific experimental setups and analysis requirements.
In summary, Cassiopeia provides a comprehensive framework for analyzing single-cell lineage tracing data, offering tools for every step from raw data processing to advanced phylogenetic analysis and visualization.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/__init__.py
## High-level description
This code defines the `__init__.py` file for the `cassiopeia.plotting` package, which provides utilities for visualizing phylogenetic trees. It imports and re-exports specific functions and classes from submodules to make them directly accessible at the package level.
## References
This file references symbols from the following files:
* `cassiopeia/plotting/itol_utilities.py`
* `cassiopeia/plotting/local.py`
* `cassiopeia/plotting/local_3d.py`
## Symbols
### Symbol Name
#### `__all__`
#### Description
A list of symbols to be exported when using `from cassiopeia.plotting import *`. This allows users to directly access these functions and classes without needing to know the specific submodules where they are defined.
#### Inputs
N/A - This is a variable definition, not a function.
#### Outputs
N/A - This is a variable definition, not a function.
#### Internal Logic
The list includes the following symbols:
* `upload_and_export_itol`: Uploads a tree to the iTOL web service and exports the visualization. (from `itol_utilities.py`)
* `plot_matplotlib`: Generates a static plot of a tree using Matplotlib. (from `local.py`)
* `plot_plotly`: Generates a dynamic plot of a tree using Plotly. (from `local.py`)
* `labels_from_coordinates`: Creates a synthetic labels array from cell coordinates for 3D plotting. (from `local_3d.py`)
* `Tree3D`: A class for creating and manipulating 3D projections of trees onto 2D surfaces. (from `local_3d.py`)
# itol_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/itol_utilities.py
## High-level description
This file provides utility functions for generating visualizations of phylogenetic trees using the iTOL (Interactive Tree Of Life) web interface. It includes functions for uploading trees and associated data to iTOL, exporting the visualizations, and creating various data files that iTOL uses for customization, such as gradient files, colorbars, and indel heatmaps.
## Code Structure
The code consists of several functions that are used to create files for iTOL and one main function, `upload_and_export_itol`, which orchestrates the process of uploading a tree and associated data to iTOL and exporting the visualization. The other functions are called by `upload_and_export_itol` to create the necessary data files.
## References
This code references the `CassiopeiaTree` class from `cassiopeia.data` and the `Itol` and `ItolExport` classes from the `itolapi` package. It also uses utility functions from `cassiopeia.plotting.utilities` for color manipulation and data preparation.
## Symbols
### `upload_and_export_itol`
#### Description
Uploads a phylogenetic tree to iTOL and exports the visualization to a local file. This function handles the entire process of preparing the tree and associated data for iTOL, uploading it, and exporting the visualization in the desired format.
#### Inputs
| Name | Type | Description |
| :------------------- | :------------------------------- | :------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | A CassiopeiaTree object containing the phylogenetic tree. |
| tree\_name | str | Name of the tree to be used in iTOL. |
| export\_filepath | str | Path to the file where the exported visualization will be saved. |
| itol\_config | str | Path to the iTOL configuration file containing API key and project name. Defaults to "\~/.itolconfig". |
| api\_key | Optional\[str] | iTOL API key. If not provided, it will be read from the `itol_config` file. |
| project\_name | Optional\[str] | iTOL project name. If not provided, it will be read from the `itol_config` file. |
| meta\_data | List\[str] | List of cell metadata columns to be plotted alongside the tree. |
| allele\_table | Optional\[pd.DataFrame] | Allele table to be plotted as an indel heatmap. |
| indel\_colors | Optional\[pd.DataFrame] | Color mapping for indels in the heatmap. |
| indel\_priors | Optional\[pd.DataFrame] | Prior probabilities for each indel, used for color mapping if `indel_colors` is not provided. |
| rect | bool | Whether to export the tree as a rectangle (True) or a circle (False). Defaults to False. |
| include\_legend | bool | Whether to include a legend for the colorbar. Defaults to False. |
| use\_branch\_lengths | bool | Whether to use branch lengths when exporting the tree. Defaults to False. |
| palette | List\[str] | List of colors in hex format to be used for categorical metadata. Defaults to `bokeh.palettes.Category20[20]`. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
| user\_dataset\_files | Optional\[List\[str]] | List of paths to additional dataset files to upload. |
| verbose | bool | Whether to print verbose output. Defaults to True. |
| \*\*kwargs | dict | Additional keyword arguments passed as iTOL export parameters. |
#### Outputs
This function does not return any values. It uploads the tree and associated data to iTOL and exports the visualization to the specified file.
#### Internal Logic
1. **Credentials:** The function first checks for iTOL API key and project name, either provided as arguments or read from the `itol_config` file.
2. **Temporary Directory:** A temporary directory is created to store the files that will be uploaded to iTOL.
3. **Tree File:** The tree is written to a Newick format file in the temporary directory.
4. **Data Files:** Additional data files are created based on the provided arguments:
* **Indel Heatmap:** If `allele_table` is provided, a set of files for the indel heatmap is created using `create_indel_heatmap`.
* **Metadata:** For each metadata column in `meta_data`, a gradient file or a colorbar file is created depending on the data type, using `create_gradient_from_df` or `create_colorbar` respectively.
5. **Upload:** The tree file and all data files are uploaded to iTOL using the `itolapi` package.
6. **Export:** The visualization is exported to the specified file using the `itolapi` package, with the specified format and parameters.
7. **Cleanup:** The temporary directory is removed.
#### Side Effects
* Creates a temporary directory and files within it.
* Uploads data to the user's iTOL account.
* Creates a local file containing the exported visualization.
### `create_gradient_from_df`
#### Description
Creates a gradient file for iTOL from a pandas Series containing numerical data. The gradient file defines a color gradient that iTOL uses to color leaves based on the numerical values.
#### Inputs
| Name | Type | Description |
| :---------------- | :------------- | :--------------------------------------------------------------------- |
| df | pd.Series | Pandas Series containing numerical data. |
| tree | CassiopeiaTree | CassiopeiaTree object. |
| dataset\_name | str | Name of the dataset. |
| output\_directory | str | Directory where the gradient file will be saved. Defaults to "./tmp/". |
| color\_min | str | Hex color code for the minimum value. Defaults to "#ffffff". |
| color\_max | str | Hex color code for the maximum value. Defaults to "#000000". |
#### Outputs
| Name | Type | Description |
| :---- | :--- | :--------------------------------- |
| outfp | str | Path to the created gradient file. |
#### Internal Logic
1. **Leaf Data:** The function extracts the numerical data for the leaves of the tree from the input Series.
2. **File Content:** It creates a pandas DataFrame with two columns: "cellBC" (leaf names) and "gradient" (numerical values).
3. **Header:** It constructs a header for the gradient file according to the iTOL format.
4. **File Writing:** It writes the header, followed by the DataFrame content in tab-separated format, to the output file.
### `create_colorbar`
#### Description
Creates a colorbar file for iTOL from a pandas Series containing categorical data. The colorbar file defines a color mapping for each unique category, which iTOL uses to color leaves.
#### Inputs
| Name | Type | Description |
| :---------------- | :-------------------------------- | :----------------------------------------------------------------------------------------- |
| labels | pd.DataFrame | Pandas Series containing categorical data. |
| tree | CassiopeiaTree | CassiopeiaTree object. |
| colormap | Dict\[str, Tuple\[int, int, int]] | Dictionary mapping categories to RGB color tuples. |
| dataset\_name | str | Name of the dataset. |
| output\_directory | str | Directory where the colorbar file will be saved. Defaults to ".tmp/". |
| create\_legend | bool | Whether to include a legend for the colorbar in the iTOL visualization. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :---- | :--- | :--------------------------------- |
| outfp | str | Path to the created colorbar file. |
#### Internal Logic
1. **Leaf Data:** The function extracts the categorical data for the leaves of the tree from the input Series.
2. **Color Mapping:** It converts the RGB color tuples in the `colormap` to iTOL's "rgb(R,G,B)" format.
3. **File Content:** It creates a pandas DataFrame with two columns: "cellBC" (leaf names) and "color" (color strings).
4. **Header:** It constructs a header for the colorbar file according to the iTOL format.
5. **Legend:** If `create_legend` is True, it adds legend information to the header, including legend title, shapes, colors, and labels.
6. **File Writing:** It writes the header, followed by the DataFrame content in tab-separated format, to the output file.
### `create_indel_heatmap`
#### Description
Creates a set of files for displaying an indel heatmap alongside the tree in iTOL. Each file corresponds to a character (cut site) in the allele table and defines a color mapping for the indels observed at that site.
#### Inputs
| Name | Type | Description |
| :---------------- | :------------------------------- | :----------------------------------------------------------------------------------------------------------------- |
| alleletable | pd.DataFrame | Allele table containing indel information. |
| tree | CassiopeiaTree | CassiopeiaTree object. |
| dataset\_name | str | Name of the dataset. |
| output\_directory | str | Directory where the heatmap files will be saved. |
| indel\_colors | Optional\[pd.DataFrame] | DataFrame mapping indels to HSV color tuples. If not provided, random colors are generated. |
| indel\_priors | Optional\[pd.DataFrame] | DataFrame containing prior probabilities for each indel. Used for color mapping if `indel_colors` is not provided. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :------------ | :--------- | :------------------------------------------ |
| allele\_files | List\[str] | List of paths to the created heatmap files. |
#### Internal Logic
1. **Lineage Profile:** The function converts the allele table to a lineage profile using `utilities.prepare_alleletable`.
2. **Color Mapping:** If `indel_colors` is not provided, it generates random colors for each unique indel using `utilities.get_random_indel_colors` or `utilities.get_indel_colors` based on `indel_priors`.
3. **Heatmap Data:** It creates a numpy array representing the heatmap, where each row corresponds to a leaf and each column corresponds to a character. The array values are RGB color tuples based on the indel observed at each position.
4. **File Writing:** For each character, it creates a colorbar file similar to `create_colorbar`, using the corresponding column from the heatmap array as the color data. The file names are formatted as "indelColors\_0.txt".
## Dependencies
This code depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :--------------------------------------------------------------- |
| itolapi | Interacting with the iTOL API for uploading and exporting trees. |
| bokeh | Providing color palettes for categorical metadata. |
| matplotlib | Color manipulation and conversion between color spaces. |
| numpy | Numerical operations and array manipulation. |
| pandas | Data manipulation and handling dataframes. |
| networkx | Working with tree structures and graph algorithms. |
| tqdm | Displaying progress bars for long-running operations. |
## Error Handling
The code raises `iTOLError` exceptions for various error conditions, such as invalid iTOL credentials, unsupported output formats, missing metadata, or errors encountered during interaction with the iTOL API.
# local.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/local.py
## High-level description
The `local.py` file in the `cassiopeia/plotting` module provides functions for plotting phylogenetic trees locally using either Matplotlib (for static plots) or Plotly (for dynamic plots). These functions offer an alternative to using iTOL, a cloud-based tree plotting service, for visualizing CassiopeiaTree objects.
## Code Structure
The code defines several helper functions for placing tree elements and annotations on a coordinate system, creating colorstrips for categorical and continuous variables, and generating an indel heatmap. These helper functions are then used by the main plotting functions, `plot_matplotlib` and `plot_plotly`, to generate the final tree visualizations.
## References
This file references symbols from the `cassiopeia.data` module, specifically the `CassiopeiaTree` class, and the `cassiopeia.plotting.utilities` module for tree placement and coordinate transformations.
## Symbols
### `compute_colorstrip_size`
#### Description
This function calculates the width and height of colorstrip boxes based on the node coordinates, anchor coordinates, and the desired location of the colorstrip relative to the anchor. The height is determined to ensure tight arrangement of boxes, while the width is set to 5% of the tree depth.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------------------------------- | :------------------------------------------------------------------ |
| node\_coords | Dict\[str, Tuple\[float, float]] | A dictionary mapping node names to their (x, y) coordinates. |
| anchor\_coords | Dict\[str, Tuple\[float, float]] | A dictionary mapping anchor node names to their (x, y) coordinates. |
| loc | Literal\["left", "right", "up", "down", "polar"] | The location of the colorstrip relative to the anchor. |
#### Outputs
| Name | Type | Description |
| :------------ | :------------------- | :-------------------------------------------------------------------------- |
| width, height | Tuple\[float, float] | A tuple containing the calculated width and height of the colorstrip boxes. |
#### Internal Logic
The function iterates through the node coordinates to determine the minimum and maximum positions and depths based on the specified location. The width is calculated as 5% of the difference between the maximum and minimum depths. The height is calculated by dividing the difference between the maximum and minimum positions by the number of anchor coordinates.
### `create_categorical_colorstrip`
#### Description
This function creates a colorstrip for a categorical variable, assigning a unique color from a specified colormap to each unique value.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------------------------------- | :------------------------------------------------------------------------------------ |
| values | Dict\[str, str] | A dictionary mapping node names to their categorical values. |
| anchor\_coords | Dict\[str, Tuple\[float, float]] | A dictionary mapping anchor node names to their (x, y) coordinates. |
| width | float | The width of the colorstrip. |
| height | float | The height of each box in the colorstrip. |
| spacing | float | The padding between consecutive colorstrip boxes. |
| loc | Literal\["left", "right", "up", "down", "polar"] | The location of the colorstrip relative to the anchor. |
| cmap | Union\[str, mpl.colors.Colormap] | The colormap to use for assigning colors. Defaults to "tab10". |
| value\_mapping | Optional\[Dict\[str, int]] | An optional dictionary mapping string values to integer indices for color assignment. |
#### Outputs
| Name | Type | Description |
| :------------------------------- | :------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| colorstrip, next\_anchor\_coords | Tuple\[Dict\[str, Tuple\[List\[float], List\[float], Tuple\[float, float, float], str]], Dict\[str, Tuple\[float, float]]] | A tuple containing two dictionaries: the first maps node names to their colorstrip box coordinates, color, and text label; the second maps node names to the next set of anchor coordinates for placing additional colorstrips. |
#### Internal Logic
The function first determines the unique categorical values and assigns integer indices to them, either using the provided `value_mapping` or generating a pseudo-random mapping. It then uses the `utilities.place_colorstrip` function to calculate the box coordinates and next anchor coordinates. Finally, it iterates through the input values, assigns a color from the colormap based on the integer index, and populates the `colorstrip` dictionary with the box coordinates, color, and text label for each node.
### `create_continuous_colorstrip`
#### Description
This function creates a colorstrip for a continuous variable, assigning colors from a specified colormap based on the normalized values within a specified range.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------------------------------- | :------------------------------------------------------------------ |
| values | Dict\[str, float] | A dictionary mapping node names to their continuous values. |
| anchor\_coords | Dict\[str, Tuple\[float, float]] | A dictionary mapping anchor node names to their (x, y) coordinates. |
| width | float | The width of the colorstrip. |
| height | float | The height of each box in the colorstrip. |
| spacing | float | The padding between consecutive colorstrip boxes. |
| loc | Literal\["left", "right", "up", "down", "polar"] | The location of the colorstrip relative to the anchor. |
| cmap | Union\[str, mpl.colors.Colormap] | The colormap to use for assigning colors. Defaults to "viridis". |
| vmin | Optional\[float] | The minimum value for the color scale. |
| vmax | Optional\[float] | The maximum value for the color scale. |
#### Outputs
| Name | Type | Description |
| :------------------------------- | :------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| colorstrip, next\_anchor\_coords | Tuple\[Dict\[str, Tuple\[List\[float], List\[float], Tuple\[float, float, float], str]], Dict\[str, Tuple\[float, float]]] | A tuple containing two dictionaries: the first maps node names to their colorstrip box coordinates, color, and text label; the second maps node names to the next set of anchor coordinates for placing additional colorstrips. |
#### Internal Logic
The function first determines the minimum and maximum values for the color scale, using the provided `vmin` and `vmax` or the minimum and maximum values from the input `values`. It then uses the `utilities.place_colorstrip` function to calculate the box coordinates and next anchor coordinates. Finally, it iterates through the input values, normalizes them to the range \[0, 1], assigns a color from the colormap based on the normalized value, and populates the `colorstrip` dictionary with the box coordinates, color, and text label for each node.
### `create_indel_heatmap`
#### Description
This function creates an indel heatmap, visualizing the indels present in an allele table using a color mapping.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------------------------------- | :------------------------------------------------------------------- |
| allele\_table | pd.DataFrame | The allele table containing indel information. |
| anchor\_coords | Dict\[str, Tuple\[float, float]] | A dictionary mapping anchor node names to their (x, y) coordinates. |
| width | float | The width of the heatmap. |
| height | float | The height of each box in the heatmap. |
| spacing | float | The padding between consecutive heatmap boxes. |
| loc | Literal\["left", "right", "up", "down", "polar"] | The location of the heatmap relative to the anchor. |
| indel\_colors | Optional\[pd.DataFrame] | An optional DataFrame mapping indels to colors. |
| indel\_priors | Optional\[pd.DataFrame] | An optional DataFrame containing prior probabilities for each indel. |
| random\_state | Optional\[np.random.RandomState] | An optional random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :---------------------- | :-------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| heatmap, anchor\_coords | Tuple\[List\[Dict\[str, Tuple\[List\[float], List\[float], Tuple\[float, float, float], str]]], Dict\[str, Tuple\[float, float]]] | A tuple containing two elements: the first is a list of colorstrips, where each colorstrip is a dictionary mapping node names to their box coordinates, color, and text label; the second is a dictionary mapping node names to the next set of anchor coordinates for placing additional colorstrips. |
#### Internal Logic
The function first prepares the allele table and indel colors using the `utilities.prepare_alleletable` function. It then iterates through each site in the allele table, calculates the box coordinates and next anchor coordinates using the `utilities.place_colorstrip` function, and assigns a color to each indel based on the provided `indel_colors` or a randomly generated color mapping. The resulting colorstrip for each site is appended to the `heatmap` list.
### `create_clade_colors`
#### Description
This function assigns colors to nodes and branches based on clade membership, using a provided dictionary mapping clade root nodes to colors.
#### Inputs
| Name | Type | Description |
| :------------ | :-------------------------------------- | :--------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object. |
| clade\_colors | Dict\[str, Tuple\[float, float, float]] | A dictionary mapping clade root nodes to RGB color tuples. |
#### Outputs
| Name | Type | Description |
| :--------------------------- | :---------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| node\_colors, branch\_colors | Tuple\[Dict\[str, Tuple\[float, float, float]], Dict\[Tuple\[str, str], Tuple\[float, float, float]]] | A tuple containing two dictionaries: the first maps node names to their assigned colors; the second maps branch tuples (parent, child) to their assigned colors. |
#### Internal Logic
The function first identifies all descendants of each clade root node. It then iterates through the clade root nodes in descending order of their descendant set size, assigning the corresponding color to all nodes and branches within each clade.
### `place_tree_and_annotations`
#### Description
This helper function places the tree and all requested annotations on a coordinate system, preparing the data for plotting.
#### Inputs
| Name | Type | Description |
| :------------------ | :----------------------------------------------------- | :-------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object to plot. |
| depth\_key | Optional\[str] | The node attribute to use as the depth of the nodes. |
| meta\_data | Optional\[List\[str]] | A list of metadata columns to plot alongside the tree. |
| allele\_table | Optional\[pd.DataFrame] | The allele table to plot alongside the tree. |
| indel\_colors | Optional\[pd.DataFrame] | A DataFrame mapping indels to colors for the indel heatmap. |
| indel\_priors | Optional\[pd.DataFrame] | A DataFrame containing prior probabilities for each indel. |
| orient | Union\[Literal\["up", "down", "left", "right"], float] | The orientation of the tree. |
| extend\_branches | bool | Whether to extend branch lengths to equalize distances from the root. |
| angled\_branches | bool | Whether to display branches as angled lines. |
| add\_root | bool | Whether to add a synthetic root node. |
| colorstrip\_width | Optional\[float] | The width of the colorstrips. |
| colorstrip\_spacing | Optional\[float] | The spacing between consecutive colorstrips. |
| clade\_colors | Optional\[Dict\[str, Tuple\[float, float, float]]] | A dictionary mapping clade root nodes to colors. |
| continuous\_cmap | Union\[str, mpl.colors.Colormap] | The colormap to use for continuous variables. |
| vmin | Optional\[float] | The minimum value for the continuous color scale. |
| vmax | Optional\[float] | The maximum value for the continuous color scale. |
| categorical\_cmap | Union\[str, mpl.colors.Colormap] | The colormap to use for categorical variables. |
| value\_mapping | Optional\[Dict\[str, int]] | A dictionary mapping string values to integer indices for categorical color assignment. |
| random\_state | Optional\[np.random.RandomState] | A random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :---------------------------------------------------------------------- | :----------------------------------- | :----------------------------------------------------------------------------------------------------------------------------- |
| node\_coords, branch\_coords, node\_colors, branch\_colors, colorstrips | Tuple\[Dict, Dict, Dict, Dict, List] | A tuple containing five elements: node coordinates, branch coordinates, node colors, branch colors, and a list of colorstrips. |
#### Internal Logic
The function first places the tree on the coordinate system using the `utilities.place_tree` function. It then calculates the initial anchor coordinates for the colorstrips, which are the coordinates of the leaf nodes. Next, it creates the indel heatmap, if an allele table is provided, and iterates through the metadata columns, creating colorstrips for each column based on its data type. Finally, it assigns colors to nodes and branches based on clade membership, if `clade_colors` is provided.
### `plot_matplotlib`
#### Description
This function generates a static plot of a CassiopeiaTree using Matplotlib.
#### Inputs
| Name | Type | Description |
| :--------- | :------------- | :--------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object to plot. |
| depth\_key | Optional\[str] | The node attribute to use as the depth of the nodes. |
# local_3d.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/local_3d.py
## High-level description
The `cassiopeia/plotting/local_3d.py` file provides functionality for creating and displaying 3D visualizations of CassiopeiaTree objects. It leverages the PyVista library for 3D rendering and offers features like node and branch placement, subclone shading, image integration, and interactive widgets for exploring the tree structure.
## Code Structure
The code defines several helper functions for color manipulation, branch interpolation, and PyVista object creation. The core functionality is encapsulated in the `Tree3D` class, which manages the tree data, labels, images, and PyVista plotter. The class provides methods for adding images, setting visualization parameters (e.g., root, time, blur strength), updating displayed elements, and adding interactive widgets.
## References
The code references the `CassiopeiaTree` class from `cassiopeia/data` for tree data and the `palettes` module from `cassiopeia/plotting` for colormaps.
## Symbols
### `interpolate_branch`
#### Description
Interpolates the branch between a parent and child node in 3D space, ensuring a 90-degree angle.
#### Inputs
| Name | Type | Description |
| :----- | :-------------------------- | :---------------------------------------- |
| parent | Tuple\[float, float, float] | Coordinates of the parent node (x, y, z). |
| child | Tuple\[float, float, float] | Coordinates of the child node (x, y, z). |
#### Outputs
| Name | Type | Description |
| :------------- | :--------- | :------------------------------------------------------------------------- |
| branch\_coords | np.ndarray | Numpy array containing the interpolated x, y, z coordinates of the branch. |
#### Internal Logic
The function calculates the intermediate point on the branch by taking the x and y coordinates of the child and the z coordinate of the parent. It then returns an array containing the parent, intermediate point, and child coordinates.
### `polyline_from_points`
#### Description
Creates a PyVista `PolyData` object representing a polyline connecting a set of points.
#### Inputs
| Name | Type | Description |
| :----- | :--------- | :------------------------------------------------------------ |
| points | np.ndarray | Numpy array containing the x, y, z coordinates of the points. |
#### Outputs
| Name | Type | Description |
| :--- | :---------- | :--------------------------------------------------- |
| poly | pv.PolyData | PyVista `PolyData` object representing the polyline. |
#### Internal Logic
The function creates a `PolyData` object, sets its points to the input `points`, and defines a cell connecting all points in sequence.
### `average_mixing`
#### Description
Mixes a set of colors by averaging the RGB values of each color.
#### Inputs
| Name | Type | Description |
| :--- | :-------------------------- | :--------------------------------------------- |
| \*c | Tuple\[float, float, float] | Variable number of color tuples in RGB format. |
#### Outputs
| Name | Type | Description |
| :----------- | :-------------------------- | :----------------------------- |
| mixed\_color | Tuple\[float, float, float] | The mixed color in RGB format. |
#### Internal Logic
The function converts the input colors to RGBA arrays using `to_rgba_array`, extracts the RGB channels, calculates the mean across all colors for each channel, and returns the resulting RGB tuple.
### `highlight`
#### Description
Highlights a color by increasing its value (V) component in the HSV color space.
#### Inputs
| Name | Type | Description |
| :--- | :-------------------------- | :------------------------- |
| c | Tuple\[float, float, float] | Color tuple in RGB format. |
#### Outputs
| Name | Type | Description |
| :----------------- | :-------------------------- | :----------------------------------- |
| highlighted\_color | Tuple\[float, float, float] | The highlighted color in RGB format. |
#### Internal Logic
The function converts the input color to HSV using `rgb_to_hsv`, increases the V component by 0.5 (capped at 1), and converts back to RGB using `hsv_to_rgb`.
### `lowlight`
#### Description
Dims out a color by decreasing its value (V) component in the HSV color space.
#### Inputs
| Name | Type | Description |
| :--- | :-------------------------- | :------------------------- |
| c | Tuple\[float, float, float] | Color tuple in RGB format. |
#### Outputs
| Name | Type | Description |
| :------------ | :-------------------------- | :------------------------------ |
| dimmed\_color | Tuple\[float, float, float] | The dimmed color in RGB format. |
#### Internal Logic
The function converts the input color to HSV using `rgb_to_hsv`, decreases the V component by 0.5 (capped at 0), and converts back to RGB using `hsv_to_rgb`.
### `labels_from_coordinates`
#### Description
Creates a synthetic labels array for 3D plotting based on the spatial coordinates of cells in a CassiopeiaTree.
#### Inputs
| Name | Type | Description |
| :------------- | :------------- | :-------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing cell spatial coordinates. |
| attribute\_key | str | Attribute name in the `cell_meta` of the tree containing coordinates. |
| shape | tuple | Shape of the output labels array (height, width). |
#### Outputs
| Name | Type | Description |
| :----- | :--------- | :------------------------------------------------------------------------------------------------------- |
| labels | np.ndarray | A synthetic labels array where each cell is represented by a circle centered at its spatial coordinates. |
#### Internal Logic
The function extracts spatial coordinates from the tree's `cell_meta`, normalizes them to the range \[0.05, 0.95], and scales the circle radius based on the number of cells. It then iterates through the cells, drawing circles on the labels array using OpenCV's `ellipse` function.
### `Tree3D`
#### Description
A class for creating and displaying 3D visualizations of CassiopeiaTree objects.
#### Inputs
| Name | Type | Description |
| :------------- | :-------------------- | :------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object to visualize. |
| labels | np.ndarray | Optional labels array for cell positions. If None, labels are generated using `labels_from_coordinates`. |
| offset | float | Offset for tree and subclone shading to prevent clipping. |
| downscale | float | Downscaling factor for images and labels. |
| cmap | Optional\[List\[str]] | Colormap to use for nodes and branches. |
| attribute\_key | str | Attribute key for leaf labels in the `cell_meta`. |
#### Internal Logic
The class initializes PyVista objects, caches for tree data, and visualization parameters. It provides methods for adding images, setting visualization parameters, updating displayed elements (nodes, branches, subclones, images), and adding interactive widgets (sliders, checkboxes). The `plot` method renders the 3D visualization using PyVista.
## Dependencies
| Dependency | Purpose |
| :-------------- | :--------------------------------------------- |
| cv2 (OpenCV) | Image processing and circle drawing. |
| pyvista | 3D rendering and visualization. |
| skimage.measure | Region properties analysis for cell positions. |
| vtk | Visualization Toolkit for PyVista. |
## Error Handling
The code raises `PlottingError` exceptions for invalid input dimensions, image shapes, and missing dependencies. It also issues `PlottingWarning` warnings for missing cell meta data.
## Configuration
The `Tree3D` class allows configuration of visualization parameters through its methods, such as `set_subclone_sigma`, `set_height`, `set_time`, `set_root`, and `select_node`.
## Logging
The code does not implement explicit logging mechanisms.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/overview
## High-level description
The `cassiopeia/plotting` directory contains modules for visualizing phylogenetic trees and related data using various methods and libraries. It provides functionality for both local plotting using Matplotlib and Plotly, as well as cloud-based visualization using the iTOL (Interactive Tree Of Life) web service. The package also includes utilities for 3D tree visualization and color palette management.
## What does it do?
This package enables researchers and data scientists to create visual representations of phylogenetic trees and associated data. It offers the following key functionalities:
1. Local tree plotting using Matplotlib (static) and Plotly (interactive).
2. Cloud-based tree visualization using iTOL, including the ability to upload trees and export visualizations.
3. 3D tree visualization using PyVista.
4. Creation of colorstrips and indel heatmaps to annotate trees with additional data.
5. Utility functions for tree placement, color manipulation, and data preparation.
These visualizations help in analyzing the structure of phylogenetic trees, comparing different clades, and identifying patterns in associated metadata or genetic information.
## Entry points
The main entry points for users of this package are:
1. `plot_matplotlib` and `plot_plotly` functions in `local.py` for local tree plotting.
2. `upload_and_export_itol` function in `itol_utilities.py` for cloud-based visualization using iTOL.
3. `Tree3D` class in `local_3d.py` for 3D tree visualization.
The `__init__.py` file exports these key functions and classes, making them directly accessible when importing from `cassiopeia.plotting`.
## Key Files
1. `local.py`: Contains functions for local tree plotting using Matplotlib and Plotly.
2. `itol_utilities.py`: Provides utilities for interacting with the iTOL web service.
3. `local_3d.py`: Implements 3D tree visualization using PyVista.
4. `utilities.py`: Contains various utility functions for tree placement, color manipulation, and data preparation.
5. `palettes.py`: Defines a color palette for use in visualizations.
## Dependencies
The package relies on several external libraries:
* Matplotlib and Plotly for 2D plotting
* PyVista for 3D visualization
* itolapi for interacting with the iTOL web service
* pandas and numpy for data manipulation
* networkx for graph operations
* OpenCV (cv2) for image processing in 3D visualization
## Configuration
The package does not require extensive configuration. However, users can customize various aspects of the visualizations through function parameters, such as:
* Tree orientation and layout
* Color schemes and palettes
* Metadata inclusion and representation
* 3D visualization parameters
For iTOL-based visualizations, users need to provide their iTOL API key and project name, either directly or through a configuration file.
In summary, the `cassiopeia/plotting` package provides a comprehensive set of tools for visualizing phylogenetic trees and associated data, offering flexibility in terms of visualization methods and customization options to suit various research needs.
# palettes.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/palettes.py
## High-level description
This code defines a single variable, `godsnot_102`, which is a list of 102 color hex codes. This palette is intended to be used for plotting.
## Symbols
### Symbol Name
#### `godsnot_102`
#### Description
A list of 102 color hex codes, likely intended to be used as a color palette for plotting. The name suggests it's an alternative to a palette from another library (potentially `scanpy`).
#### Inputs
*None*
#### Outputs
*None*
#### Internal Logic
The variable is a hardcoded list of strings, where each string is a hex color code.
```python theme={null}
godsnot_102 = [
"#FFFF00",
"#1CE6FF",
"#FF34FF",
...
"#324E72",
]
```
#### Side Effects
*None*
#### Performance Considerations
*None*
```
```
# utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/plotting/utilities.py
## High-level description
This file contains utility functions for plotting phylogenetic trees locally using libraries like Matplotlib and Plotly. It provides functions for placing tree nodes and branches on a coordinate system, generating color strips for annotations, and creating indel heatmaps.
## Code Structure
The code consists of several independent utility functions that can be used for plotting phylogenetic trees. These functions include: `degrees_to_radians`, `polar_to_cartesian`, `polars_to_cartesians`, `place_tree`, `place_colorstrip`, `generate_random_color`, `get_random_indel_colors`, `get_indel_colors`, `hex_to_rgb`, `rgb_to_hex`, and `prepare_alleletable`.
## References
This file references the `CassiopeiaTree` class from `cassiopeia.data` and the `convert_alleletable_to_lineage_profile` function from `cassiopeia.preprocess.utilities`.
## Symbols
### `degrees_to_radians`
#### Description
Converts an angle from degrees to radians.
#### Inputs
| Name | Type | Description |
| :------ | :---- | :---------------- |
| degrees | float | Angle in degrees. |
#### Outputs
| Name | Type | Description |
| :---- | :---- | :---------------- |
| float | float | Angle in radians. |
### `polar_to_cartesian`
#### Description
Converts polar coordinates (angle and radius) to Cartesian coordinates (x and y).
#### Inputs
| Name | Type | Description |
| :------ | :---- | :---------------- |
| degrees | float | Angle in degrees. |
| r | float | Radius. |
#### Outputs
| Name | Type | Description |
| :------------------- | :------------------- | :------------------------------------------ |
| Tuple\[float, float] | Tuple\[float, float] | A tuple containing the x and y coordinates. |
### `polars_to_cartesians`
#### Description
Converts a list of polar coordinates to a list of Cartesian coordinates.
#### Inputs
| Name | Type | Description |
| :------ | :----------- | :------------------------- |
| degrees | List\[float] | List of angles in degrees. |
| rs | List\[float] | List of radii. |
#### Outputs
| Name | Type | Description |
| :--------------------------------- | :--------------------------------- | :---------------------------------------------------------- |
| Tuple\[List\[float], List\[float]] | Tuple\[List\[float], List\[float]] | Two lists, containing the x and y coordinates respectively. |
### `place_tree`
#### Description
Computes the coordinates of nodes and branches in a `CassiopeiaTree` for visualization. It supports both rectangular and polar coordinate systems and offers options for customizing the tree layout.
#### Inputs
| Name | Type | Description |
| :------------------------------ | :----------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree to place on the coordinate grid. |
| depth\_key | Optional\[str] | The node attribute to use as the depth of the nodes. If not provided, the distances from the root are used. |
| orient | Union\[Literal\["down", "up", "left", "right"], float] | The orientation of the tree. Can be "down", "up", "left", "right" for rectangular plots, or a number for polar plots. |
| depth\_scale | float | Scale the depth of the tree by this amount. |
| width\_scale | float | Scale the width of the tree by this amount. |
| extend\_branches | bool | Extend branch lengths to equalize distances from the root to all nodes. |
| angled\_branches | bool | Display branches as angled lines instead of straight lines. |
| polar\_interpolation\_threshold | float | Threshold for adding interpolation points in polar plots to smooth out large angles. |
| polar\_interpolation\_step | float | Interpolation step for polar plots. |
| add\_root | bool | Add a synthetic root node to the tree. |
#### Outputs
| Name | Type | Description |
| :---------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------- |
| Tuple\[Dict\[str, Tuple\[float, float]], Dict\[Tuple\[str, str], Tuple\[List\[float], List\[float]]]] | Tuple\[Dict\[str, Tuple\[float, float]], Dict\[Tuple\[str, str], Tuple\[List\[float], List\[float]]]] | Two dictionaries: the first contains node coordinates, and the second contains branch coordinates. |
#### Internal Logic
The function first places the leaves of the tree, then iteratively places internal nodes based on the positions of their children. It supports both rectangular and polar coordinate systems and offers options for customizing the tree layout, such as extending branches, angling branches, and adding a synthetic root node.
### `place_colorstrip`
#### Description
Computes the coordinates of boxes for a color strip annotation, which can be used to visualize categorical or continuous data alongside the tree.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------------------------------- | :---------------------------------------------------------- |
| anchor\_coords | Dict\[str, Tuple\[float, float]] | Dictionary of anchor coordinates for the color strip boxes. |
| width | float | Width of the box. |
| height | float | Height of the box. |
| spacing | float | Space between consecutive color strips. |
| loc | Literal\["left", "right", "up", "down", "polar"] | Location of the boxes relative to the anchors. |
#### Outputs
| Name | Type | Description |
| :--------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------- |
| Tuple\[Dict\[str, Tuple\[List\[float], List\[float]]], Dict\[str, Tuple\[float, float]]] | Tuple\[Dict\[str, Tuple\[List\[float], List\[float]]], Dict\[str, Tuple\[float, float]]] | A dictionary of box coordinates and a dictionary of new anchor coordinates. |
### `generate_random_color`
#### Description
Generates a random RGB color within specified ranges for each color channel.
#### Inputs
| Name | Type | Description |
| :------------ | :------------------------------- | :------------------------------------- |
| r\_range | Tuple\[float, float] | Range of values for the red channel. |
| g\_range | Tuple\[float, float] | Range of values for the green channel. |
| b\_range | Tuple\[float, float] | Range of values for the blue channel. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :-------------------- | :-------------------- | :---------------------------------------------------------- |
| Tuple\[int, int, int] | Tuple\[int, int, int] | A tuple containing the randomly generated RGB color values. |
### `get_random_indel_colors`
#### Description
Assigns random HSV colors to each unique indel in a lineage profile.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------------------------- | :--------------------------------------------- |
| lineage\_profile | pd.DataFrame | Lineage profile containing indel observations. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :----------- | :----------- | :---------------------------------------- |
| pd.DataFrame | pd.DataFrame | A DataFrame mapping indels to HSV colors. |
### `get_indel_colors`
#### Description
Maps indels to HSV colors based on their prior probabilities, with rarer indels receiving brighter colors.
#### Inputs
| Name | Type | Description |
| :------------ | :------------------------------- | :----------------------------------------- |
| indel\_priors | pd.DataFrame | DataFrame mapping indels to probabilities. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :----------- | :----------- | :---------------------------------------- |
| pd.DataFrame | pd.DataFrame | A DataFrame mapping indels to HSV colors. |
### `hex_to_rgb`
#### Description
Converts a hexadecimal color code to an RGB tuple.
#### Inputs
| Name | Type | Description |
| :---- | :--- | :---------------------------------------- |
| value | str | Hexadecimal color code (e.g., "#FF0000"). |
#### Outputs
| Name | Type | Description |
| :-------------------- | :-------------------- | :--------------------------------------- |
| Tuple\[int, int, int] | Tuple\[int, int, int] | A tuple containing the RGB color values. |
### `rgb_to_hex`
#### Description
Converts an RGB tuple to a hexadecimal color code.
#### Inputs
| Name | Type | Description |
| :--- | :-------------------- | :--------------------------------------- |
| rgb | Tuple\[int, int, int] | A tuple containing the RGB color values. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :---------------------------------------- |
| str | str | Hexadecimal color code (e.g., "#FF0000"). |
### `prepare_alleletable`
#### Description
Prepares an allele table for plotting by generating indel colors and creating a lineage profile.
#### Inputs
| Name | Type | Description |
| :------------ | :------------------------------- | :------------------------------------------ |
| allele\_table | pd.DataFrame | Allele table containing indel observations. |
| leaves | List\[str] | List of leaf node names in the tree. |
| indel\_priors | Optional\[pd.DataFrame] | DataFrame mapping indels to probabilities. |
| random\_state | Optional\[np.random.RandomState] | Random state for reproducibility. |
#### Outputs
| Name | Type | Description |
| :--------------------------------- | :--------------------------------- | :--------------------------------------------------------------------- |
| Tuple\[pd.DataFrame, pd.DataFrame] | Tuple\[pd.DataFrame, pd.DataFrame] | A tuple containing the lineage profile and indel colors as DataFrames. |
#### Internal Logic
The function first converts the allele table to a lineage profile using the `convert_alleletable_to_lineage_profile` function. Then, it generates indel colors either randomly or based on prior probabilities, depending on whether `indel_priors` is provided.
# UMI_utils.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/UMI_utils.py
## High-level description
This file provides functions for collapsing and preprocessing Unique Molecular Identifiers (UMIs) in sequencing data. It handles tasks like grouping reads by cell barcodes and UMIs, forming consensus sequences for each UMI cluster, and error-correcting UMI sequences.
## Code Structure
The code defines several functions that work together to perform UMI collapsing and error correction. The main functions are `sort_bam`, `form_collapsed_clusters`, `form_clusters`, `form_clusters_likelihood`, and `correct_umis_in_group`. `sort_bam` sorts the input BAM file by cell barcode and UMI. `form_collapsed_clusters` groups reads with the same cell barcode and UMI, and then calls either `form_clusters` or `form_clusters_likelihood` to form consensus sequences for each UMI cluster. `correct_umis_in_group` further groups reads by cell barcode, UMI, and optionally allele, and then performs error correction on the UMI sequences within each group.
## References
This file references the following symbols:
* `cassiopeia.preprocess.constants.BAM_CONSTANTS`
* `cassiopeia.preprocess.utilities.convert_bam_to_df`
* `cassiopeia.mixins.logger`
* `cassiopeia.mixins.PreprocessError`
* `cassiopeia.mixins.PreprocessWarning`
## Symbols
### `detect_cell_bc_tag`
#### Description
This function determines the appropriate BAM tag for cell barcodes by checking if the corrected cell barcode tag exists in the BAM file. If it does, it returns the corrected tag; otherwise, it returns the raw cell barcode tag.
#### Inputs
| Name | Type | Description |
| :------ | :--- | :-------------------- |
| bam\_fp | str | Path to the BAM file. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :-------------------------------------- |
| | str | The BAM tag to use as the cell barcode. |
### `group_bam_by_key`
#### Description
This generator function iterates through a sorted BAM file and yields groups of alignments that share the same key, defined by the `sort_key` function.
#### Inputs
| Name | Type | Description |
| :--------- | :------- | :---------------------------------------------------------------- |
| sorted\_fn | str | Path to the sorted BAM file. |
| sort\_key | Callable | Function that takes an alignment and returns the key to group by. |
| n\_threads | int | Number of threads to use. |
#### Outputs
| Name | Type | Description |
| :--- | :----------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------- |
| | Generator\[Tuple\[Union\[str, Tuple\[str, ...]], List\[pysam.AlignedSegment]], None, None] | A generator yielding tuples of two elements: the grouping key and a list of alignments with that key. |
### `sort_bam`
#### Description
This function sorts a BAM file by a specified key and filters alignments based on a filter function. It splits the BAM file into chunks, sorts and filters each chunk, and then merges the sorted chunks into a final sorted BAM file.
#### Inputs
| Name | Type | Description |
| :----------- | :------- | :-------------------------------------------------------------------------------------------------- |
| bam\_fp | str | Path to the input BAM file. |
| sorted\_fn | str | Path to the output sorted BAM file. |
| sort\_key | Callable | Function that takes an alignment and returns the key to sort by. |
| filter\_func | Callable | Function that takes an alignment and returns True if the alignment should be kept, False otherwise. |
| n\_threads | int | Number of threads to use. |
#### Outputs
| Name | Type | Description |
| :---------------- | :--- | :-------------------------------------------------------- |
| max\_read\_length | int | The maximum read length observed in the BAM file. |
| total\_reads\_out | int | The total number of reads written to the sorted BAM file. |
### `form_collapsed_clusters`
#### Description
This function groups reads with the same cell barcode and UMI, and then forms consensus sequences for each UMI cluster. It uses either a hard cutoff method or a likelihood-based method to form the initial sequence clusters, and then attempts to merge clusters with similar sequences.
#### Inputs
| Name | Type | Description |
| :------------------ | :------------------------------- | :----------------------------------------------------------------------------------- |
| sorted\_fn | str | Path to the sorted BAM file. |
| collapsed\_fn | str | Path to the output collapsed BAM file. |
| max\_hq\_mismatches | int | Maximum number of high-quality mismatches allowed between sequences to be collapsed. |
| max\_indels | int | Maximum number of indels allowed between sequences to be collapsed. |
| cell\_key | Callable | Function that takes an alignment and returns the cell barcode. |
| UMI\_key | Callable | Function that takes an alignment and returns the UMI sequence. |
| method | Literal\["cutoff", "likelihood"] | Method to use for forming initial sequence clusters. |
| n\_threads | int | Number of threads to use. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :--------------------------------------------------- |
| | None | This function writes the collapsed BAM file to disk. |
### `form_clusters`
#### Description
This function implements the hard cutoff method for forming consensus sequences for UMI clusters. It iteratively finds the most frequent sequence in the remaining alignments and groups all sequences within a certain Hamming distance from it.
#### Inputs
| Name | Type | Description |
| :------------------ | :-------------------------- | :----------------------------------------------------------------------------------- |
| als | List\[pysam.AlignedSegment] | List of alignments to cluster. |
| max\_read\_length | int | Maximum read length in the dataset. |
| max\_hq\_mismatches | int | Maximum number of high-quality mismatches allowed between sequences to be collapsed. |
#### Outputs
| Name | Type | Description |
| :--- | :-------------------------- | :------------------------------------------------------------------------------- |
| | List\[pysam.AlignedSegment] | A list of annotated aligned segments representing the consensus of each cluster. |
### `form_clusters_likelihood`
#### Description
This function implements the likelihood-based method for forming consensus sequences for UMI clusters. It uses the quality scores of the bases to calculate the probability of each base being correct and forms consensus sequences based on these probabilities.
#### Inputs
| Name | Type | Description |
| :----------- | :-------------------------- | :------------------------------------------------------------------------------- |
| als | List\[pysam.AlignedSegment] | List of alignments to cluster. |
| q\_threshold | Optional\[int] | Optional PHRED quality threshold. |
| proportion | float | Proportion of each sequence to allow mismatched bases to be above `q_threshold`. |
#### Outputs
| Name | Type | Description |
| :--- | :-------------------------- | :------------------------------------------------------------------------------- |
| | List\[pysam.AlignedSegment] | A list of annotated aligned segments representing the consensus of each cluster. |
### `align_clusters`
#### Description
This function calculates the number of indels and high-quality mismatches between two aligned segments using the Smith-Waterman algorithm.
#### Inputs
| Name | Type | Description |
| :----- | :------------------- | :-------------------------- |
| first | pysam.AlignedSegment | The first aligned segment. |
| second | pysam.AlignedSegment | The second aligned segment. |
#### Outputs
| Name | Type | Description |
| :------------------ | :--- | :--------------------------------------------------------------- |
| indels | int | The number of differing indels between the two sequences. |
| num\_hq\_mismatches | int | The number of high-quality mismatches between the two sequences. |
### `within_radius_of_seed`
#### Description
This function takes a seed sequence and a list of alignments and returns two lists: one containing alignments whose sequences are within a certain Hamming distance from the seed sequence, and another containing the remaining alignments.
#### Inputs
| Name | Type | Description |
| :------------------ | :-------------------------- | :------------------------------------------------------------------------------------------ |
| seed | str | The seed sequence. |
| als | List\[pysam.AlignedSegment] | List of alignments to compare to the seed sequence. |
| max\_hq\_mismatches | int | Maximum number of high-quality mismatches allowed between a sequence and the seed sequence. |
#### Outputs
| Name | Type | Description |
| :--------- | :-------------------------- | :---------------------------------------- |
| near\_seed | List\[pysam.AlignedSegment] | Alignments within radius of the seed. |
| remaining | List\[pysam.AlignedSegment] | Alignments not within radius of the seed. |
### `propose_seed`
#### Description
This function proposes a seed sequence for clustering by finding the most frequent sequence in a list of alignments. If there is no most frequent sequence, it calls `call_consensus` to generate a consensus sequence.
#### Inputs
| Name | Type | Description |
| :---------------- | :-------------------------- | :-------------------------------------------------- |
| als | List\[pysam.AlignedSegment] | List of alignments to propose a seed sequence from. |
| max\_read\_length | int | Maximum read length in the dataset. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :-------------------------- |
| | str | The proposed seed sequence. |
### `make_singleton_cluster`
#### Description
This function creates a singleton cluster from a single alignment.
#### Inputs
| Name | Type | Description |
| :--- | :------------------- | :------------------------------------------------ |
| al | pysam.AlignedSegment | The alignment to create a singleton cluster from. |
#### Outputs
| Name | Type | Description |
| :--- | :------------------- | :--------------------------------------------------------------- |
| | pysam.AlignedSegment | An annotated aligned segment representing the singleton cluster. |
### `call_consensus`
#### Description
This function generates a consensus sequence from a list of alignments by taking the most frequent high-quality base at each position.
#### Inputs
| Name | Type | Description |
| :---------------- | :-------------------------- | :-------------------------------------------------------- |
| als | List\[pysam.AlignedSegment] | List of alignments to generate a consensus sequence from. |
| max\_read\_length | int | Maximum read length in the dataset. |
#### Outputs
| Name | Type | Description |
| :--- | :------------------- | :------------------------------------- |
| | pysam.AlignedSegment | A consensus annotated aligned segment. |
### `merge_annotated_clusters`
#### Description
This function merges two annotated clusters by adding the read count of the smaller cluster to the larger cluster and updating the cluster ID.
#### Inputs
| Name | Type | Description |
| :------ | :------------------- | :------------------------------- |
| biggest | pysam.AlignedSegment | The larger of the two clusters. |
| other | pysam.AlignedSegment | The smaller of the two clusters. |
#### Outputs
| Name | Type | Description |
| :--- | :------------------- | :------------------------------------------------------------- |
| | pysam.AlignedSegment | The annotated aligned segment representing the merged cluster. |
### `correct_umis_in_group`
#### Description
This function performs error correction on UMI sequences within a group of alignments that share the same cell barcode and integration barcode. It uses a Hamming distance threshold to determine which UMIs should be merged.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :------------------------------------------------------------------------- |
| cell\_group | pd.DataFrame | DataFrame of alignments for a single cell barcode and integration barcode. |
| max\_umi\_distance | int | Maximum Hamming distance between UMIs for error correction. |
#### Outputs
| Name | Type | Description |
| :--------------------- | :----------- | :------------------------------------------------- |
| corrected\_cell\_group | pd.DataFrame | DataFrame of alignments with error-corrected UMIs. |
| len(corrections) | int | Number of corrected UMIs. |
#### Internal Logic
The function first calculates the Hamming distance between all pairs of UMIs in the group. It then identifies UMIs that are within the specified distance threshold and merges them into the more abundant UMI.
#### Side Effects
This function modifies the input DataFrame in place.
#### Performance Considerations
The function uses a Cython implementation of the Hamming distance calculation to improve performance.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/__init__.py
## High-level description
This file defines the public API for the `cassiopeia.preprocess` module, exposing functions for preprocessing sequencing data into character matrices suitable for phylogenetic inference. It imports and re-exports specific functions from submodules, making them directly accessible through `cassiopeia.preprocess`.
## References
This file imports and re-exports symbols from the following files:
* `cassiopeia/preprocess/pipeline.py`
* `cassiopeia/preprocess/utilities.py`
* `cassiopeia/preprocess/setup_utilities.py`
## Symbols
### Symbol Name
#### `__all__`
#### Description
A list of strings defining the public API of the `cassiopeia.preprocess` module. It specifies which symbols are exported when using `from cassiopeia.preprocess import *`.
#### Inputs
N/A - This is a variable, not a function.
#### Outputs
N/A - This is a variable, not a function.
#### Internal Logic
The list includes functions for various preprocessing steps like sequence alignment, allele calling, UMI collapsing, data filtering, and lineage group calling. It also includes utility functions for data conversion and setup.
```python theme={null}
__all__ = [
"align_sequences",
"call_alleles",
"collapse_umis",
"convert_fastqs_to_unmapped_bam",
"error_correct_cellbcs_to_whitelist",
"error_correct_intbcs_to_whitelist",
"error_correct_umis",
"filter_bam",
"resolve_umi_sequence",
"filter_molecule_table",
"call_lineage_groups",
"compute_empirical_indel_priors",
"convert_alleletable_to_character_matrix",
"convert_alleletable_to_lineage_profile",
"convert_lineage_profile_to_character_matrix",
"filter_cells",
"filter_umis",
"setup",
]
```
***
# alignment_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/alignment_utilities.py
## High-level description
This file provides utility functions for handling sequence alignments, particularly for processing TargetSite sequencing data. It includes functions for performing local and global alignment, parsing CIGAR strings to identify indels, and extracting integration barcodes.
## Code Structure
The `parse_cigar` function utilizes the output of the `align_local` and `align_global` functions, specifically the CIGAR string, to identify indels.
## Symbols
### Symbol Name
#### `align_local`
#### Description
Performs local alignment of a query sequence (`seq`) to a reference sequence (`ref`) using the Smith-Waterman algorithm.
#### Inputs
| Name | Type | Description |
| :------------------- | :-------------------------- | :-------------------------------------------------- |
| ref | str | The reference sequence. |
| seq | str | The query sequence. |
| substitution\_matrix | Dict\[str, Dict\[str, int]] | Nested dictionary encoding the substitution matrix. |
| gap\_open\_penalty | int | Gap open penalty. |
| gap\_extend\_penalty | int | Gap extend penalty. |
#### Outputs
| Name | Type | Description |
| :----------- | :---- | :--------------------------------------------------------- |
| cigar | str | The CIGAR string representing the alignment. |
| query\_start | int | Start position of the alignment in the query sequence. |
| ref\_start | int | Start position of the alignment in the reference sequence. |
| score | float | Alignment score. |
| seq | str | The input query sequence. |
#### Internal Logic
1. Initializes a Smith-Waterman aligner using the provided substitution matrix and gap penalties.
2. Performs local alignment using the `align` method of the aligner.
3. Extracts the alignment results, including the aligned sequences, start positions, and alignment score.
4. Converts the alignment results to a CIGAR string using the `ngs.sequence.alignment_to_cigar` function.
5. Returns the CIGAR string, start positions, alignment score, and the input query sequence.
### Symbol Name
#### `align_global`
#### Description
Performs global alignment of a query sequence (`seq`) to a reference sequence (`ref`) using the Needleman-Wunsch algorithm.
#### Inputs
| Name | Type | Description |
| :------------------- | :-------------------------- | :-------------------------------------------------- |
| ref | str | The reference sequence. |
| seq | str | The query sequence. |
| substitution\_matrix | Dict\[str, Dict\[str, int]] | Nested dictionary encoding the substitution matrix. |
| gap\_open\_penalty | int | Gap open penalty. |
| gap\_extend\_penalty | int | Gap extend penalty. |
#### Outputs
| Name | Type | Description |
| :----------- | :---- | :--------------------------------------------------------- |
| cigar | str | The CIGAR string representing the alignment. |
| query\_start | int | Start position of the alignment in the query sequence. |
| ref\_start | int | Start position of the alignment in the reference sequence. |
| score | float | Alignment score. |
| seq | str | The input query sequence. |
#### Internal Logic
1. Initializes a Needleman-Wunsch aligner using the provided substitution matrix and gap penalties.
* Note: The `no_end_gap_penalty` parameter is set to `True` because reads are expected to be shorter than the reference.
2. Performs global alignment using the `align` method of the aligner.
3. Extracts the alignment results, including the aligned sequences, start positions, and alignment score.
4. Converts the alignment results to a CIGAR string using the `ngs.sequence.alignment_to_cigar` function.
5. Returns the CIGAR string, start positions, alignment score, and the input query sequence.
### Symbol Name
#### `parse_cigar`
#### Description
Parses a CIGAR string to identify insertions and deletions (indels) relative to a reference sequence, focusing on regions around specified cutsites and a barcode interval.
#### Inputs
| Name | Type | Description |
| :---------------- | :--------------- | :------------------------------------------------------------------------------------------------- |
| cigar | str | The CIGAR string to parse. |
| seq | str | The query sequence. |
| ref | str | The reference sequence. |
| ref\_start | int | Start position of the alignment in the reference sequence. |
| query\_start | int | Start position of the alignment in the query sequence. |
| barcode\_interval | Tuple\[int, int] | Interval in the reference sequence containing the integration barcode. |
| cutsites | List\[int] | List of cutsite locations in the reference sequence. |
| cutsite\_window | int | Number of nucleotides to the left and right of each cutsite to consider for indels. Defaults to 0. |
| context | bool | Whether to include nucleotide context around indels. Defaults to True. |
| context\_size | int | Number of bases to include as context on each side of an indel. Defaults to 5. |
#### Outputs
| Name | Type | Description |
| :----- | :--------- | :------------------------------------------------------------------ |
| intBC | str | The extracted integration barcode. |
| indels | List\[str] | A list of strings, one per cutsite, describing the observed indels. |
#### Internal Logic
1. **Initialization:**
* Creates lists to store indels (`indels`) and indels occurring outside cutsite windows (`uncut_indels`) for each cutsite.
* Initializes the integration barcode (`intBC`) to "NC" (not captured).
* Extracts the anchor sequence from the reference for later intBC extraction.
* Sets up pointers to track the current position in the reference (`ref_pointer`) and query (`query_pointer`) sequences.
* Initializes a variable `query_pad` to account for potential insertions or deletions at the beginning of the intBC.
2. **CIGAR Parsing:**
* Iterates through each chunk (operation and length) in the CIGAR string.
* **Match (M):**
* Updates `ref_pointer` and `query_pointer`.
* If the match overlaps with the `barcode_interval`, extracts the `intBC`.
* If the match overlaps with any `cutsite` window and no indel has been recorded for that cutsite, records "None" (no indel) with optional context.
* Handles partial deletions within the `barcode_interval`.
* **Insertion (I):**
* Increments `query_pointer`.
* If the insertion occurs within any `cutsite` window, records the insertion with its position and length, along with optional context.
* **Deletion (D):**
* Increments `ref_pointer`.
* If the deletion occurs within any `cutsite` window, records the deletion with its position and length, along with optional context.
* **Hard Clip (H):**
* Increments `query_pointer`.
* **Unknown Operation:**
* Raises an `UnknownCigarStringError`.
3. **Integration Barcode (intBC) Handling:**
* If `intBC` is still "NC" or its length is less than expected, attempts to extract it based on the anchor sequence.
4. **Indel Finalization:**
* Replaces empty indel strings in `indels` with corresponding entries from `uncut_indels`.
5. **Return:**
* Returns the extracted `intBC` and the list of `indels`.
## References
* This code references the `ngs_tools` library for sequence manipulation and alignment functions.
* It also references the `pyseq_align` library for the Smith-Waterman and Needleman-Wunsch alignment algorithms.
* The code utilizes the `UnknownCigarStringError` from the `cassiopeia.mixins` module.
## Error Handling
* The `parse_cigar` function raises an `UnknownCigarStringError` if it encounters an unsupported CIGAR operation.
# cassiopeia_preprocess.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/cassiopeia_preprocess.py
## High-level description
This script orchestrates the preprocessing of sequencing data for phylogenetic inference using the Cassiopeia pipeline. It takes a configuration file as input, which specifies the desired preprocessing steps and their parameters, and executes them sequentially. The script supports various functionalities such as converting FASTQ files to BAM format, filtering reads, error-correcting barcodes and UMIs, collapsing UMIs, aligning sequences, calling alleles, and filtering molecule tables.
## Code Structure
The `main` function serves as the entry point for the script. It parses command-line arguments, reads the configuration file, sets up the output directory, and creates a pipeline plan based on the specified entry and exit points. It then iterates through the pipeline stages, executing each stage's corresponding function with the appropriate parameters. The `STAGES` dictionary maps stage names to their respective functions defined in the `cassiopeia.preprocess.pipeline` module.
## References
This script references functions from the `cassiopeia.preprocess.pipeline` and `cassiopeia.preprocess.setup_utilities` modules.
## Symbols
### `main`
#### Description
This function is the main entry point for the Cassiopeia preprocessing script. It parses the configuration file, sets up the pipeline, and executes the preprocessing stages.
#### Inputs
This function takes no direct inputs. It uses command-line arguments to obtain the path to the configuration file.
#### Outputs
This function has no direct outputs. It writes the processed data to files in the specified output directory.
#### Internal Logic
1. **Parse command-line arguments:** Retrieves the path to the configuration file.
2. **Read configuration file:** Parses the configuration file and extracts pipeline parameters.
3. **Validate configuration:** Checks if all specified stages are valid.
4. **Setup output directory:** Creates the output directory if it doesn't exist.
5. **Create pipeline plan:** Determines the stages to execute based on the entry and exit points defined in the configuration.
6. **Load input data:** Loads the input data based on the entry point.
7. **Execute pipeline stages:** Iterates through the pipeline stages, executing each stage's function with the corresponding parameters.
8. **Write output data:** Writes the processed data for each stage to a separate file in the output directory.
## Side Effects
This script modifies the file system by creating directories and writing output files. It also logs information and errors to the console and log files.
## Dependencies
This script depends on the following external libraries:
| Dependency | Purpose |
| :------------------------------------- | :------------------------------- |
| argparse | Parsing command-line arguments |
| logging | Logging information and errors |
| pandas | Data manipulation and analysis |
| typing | Type hinting |
| cassiopeia.mixins | Cassiopeia-specific mixins |
| cassiopeia.preprocess.pipeline | Preprocessing pipeline functions |
| cassiopeia.preprocess.setup\_utilities | Setup utilities for the pipeline |
### Configuration
The script relies on a configuration file to define the pipeline parameters. The configuration file should be in INI format and contain sections for each stage, as well as a "general" section for global parameters. See `cassiopeia/preprocess/constants.py` for the default pipeline parameters and their descriptions.
## Error Handling
The script raises `PreprocessError` exceptions for invalid configuration parameters or pipeline stages. It also logs errors to the console and error log file.
## Logging
The script uses the `logging` module to log information and errors to the console and log files. The verbosity of the logging can be controlled using the `verbose` parameter in the configuration file.
# constants.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/constants.py
## High-level description
This file defines constants and default parameters used in the Cassiopeia preprocessing pipeline. These constants include BAM tag names, quality scores, a DNA substitution matrix, and default parameters for various pipeline stages.
## Code Structure
The code defines several dictionaries: `BAM_CONSTANTS`, `SINGLE_CELL_BAM_TAGS`, `SPATIAL_BAM_TAGS`, `CHEMISTRY_BAM_TAGS`, `DNA_SUBSTITUTION_MATRIX`, and `DEFAULT_PIPELINE_PARAMETERS`. The first four dictionaries define BAM tag names for different sequencing chemistries. `DNA_SUBSTITUTION_MATRIX` defines a substitution matrix for DNA alignment. `DEFAULT_PIPELINE_PARAMETERS` defines default parameters for each stage of the preprocessing pipeline.
## Symbols
### Symbol Name: `BAM_CONSTANTS`
#### Description:
This dictionary stores constants related to BAM file tags used in the preprocessing pipeline.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
The dictionary maps descriptive names to their corresponding BAM tag strings. For example, `RAW_CELL_BC_TAG` maps to `"CR"`, which represents the tag for the raw cell barcode sequence.
### Symbol Name: `SINGLE_CELL_BAM_TAGS`
#### Description:
This dictionary defines BAM tag names for single-cell sequencing chemistries.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
The dictionary maps data types (`umi`, `cell_barcode`) to tuples of BAM tag names. Each tuple contains two tags: one for the sequence and one for the quality scores.
### Symbol Name: `SPATIAL_BAM_TAGS`
#### Description:
This dictionary defines BAM tag names for spatial sequencing chemistries.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
Similar to `SINGLE_CELL_BAM_TAGS`, this dictionary maps data types (`umi`, `spot_barcode`) to tuples of BAM tag names.
### Symbol Name: `CHEMISTRY_BAM_TAGS`
#### Description:
This dictionary maps specific sequencing chemistries to their corresponding BAM tag dictionaries.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
The dictionary maps chemistry names (e.g., 'dropseq', '10xv2') to either `SINGLE_CELL_BAM_TAGS` or `SPATIAL_BAM_TAGS` based on the chemistry type.
### Symbol Name: `DNA_SUBSTITUTION_MATRIX`
#### Description:
This dictionary defines a substitution matrix for DNA sequence alignment.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
The dictionary represents a matrix where keys are nucleotides (A, T, C, G, Z, N) and values are dictionaries mapping each nucleotide to a score. This matrix is used to score alignments between DNA sequences.
### Symbol Name: `DEFAULT_PIPELINE_PARAMETERS`
#### Description:
This dictionary stores default parameters for each stage of the Cassiopeia preprocessing pipeline.
#### Inputs:
N/A - This is a constant dictionary.
#### Outputs:
N/A - This is a constant dictionary.
#### Internal Logic:
The dictionary maps stage names (e.g., 'general', 'convert', 'filter\_bam') to dictionaries containing parameter names and their default values. These parameters control the behavior of each stage in the pipeline.
# doublet_utils.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/doublet_utils.py
## High-level description
The `doublet_utils.py` file provides functions for identifying and filtering potential cell doublets in single-cell sequencing data. It offers methods to detect both intra-lineage and inter-lineage doublets based on the consistency of allele information within and across lineage groups.
## Code Structure
The `filter_intra_doublets` function identifies cells with conflicting allele information within themselves, while `get_intbc_set` and `compute_lg_membership` are used by `filter_inter_doublets` to identify cells with ambiguous lineage assignments.
## Symbols
### `filter_intra_doublets`
#### Description
This function identifies and filters out cells that exhibit a high degree of conflicting allele information within themselves, suggesting they might be doublets. It calculates the proportion of UMIs supporting conflicting alleles for each cell and filters those exceeding a specified threshold.
#### Inputs
| Name | Type | Description |
| :-------------- | :--------------- | :---------------------------------------------------------------------------------------------------------------- |
| molecule\_table | pandas.DataFrame | A DataFrame representing a molecule table with cellBC-UMI pairs. |
| prop | float | The threshold for the proportion of conflicting UMIs above which a cell is considered a doublet. Defaults to 0.1. |
#### Outputs
| Name | Type | Description |
| :--------------- | :----------------------------------------------------------------------- | :---------- |
| pandas.DataFrame | A filtered molecule table with potential intra-lineage doublets removed. | |
#### Internal Logic
1. Groups UMIs by cell barcode (cellBC), integration barcode (intBC), and allele, counting UMIs per group.
2. Identifies the most common allele for each cellBC-intBC pair.
3. Calculates the proportion of UMIs supporting alleles different from the most common one for each cellBC.
4. Filters out cellBCs with a conflicting UMI proportion exceeding the specified `prop` threshold.
### `get_intbc_set`
#### Description
This function identifies the set of intBCs present in a lineage group, optionally filtering out intBCs with low prevalence within the group.
#### Inputs
| Name | Type | Description |
| :----- | :--------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| lg | pandas.DataFrame | A DataFrame representing an allele table for a single lineage group. |
| thresh | int | The minimum proportion of cells in the lineage group required to have an intBC for it to be included in the set. If None, no filtering is performed. Defaults to None. |
#### Outputs
| Name | Type | Description |
| :---- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- |
| tuple | A tuple containing: (1) a set of intBCs present in the lineage group (potentially filtered by `thresh`), and (2) a dictionary mapping each intBC to the proportion of cells in the lineage group lacking that intBC. | |
#### Internal Logic
1. Calculates the number of unique cells in the lineage group.
2. Counts the number of unique cellBCs associated with each intBC.
3. Calculates the proportion of cells lacking each intBC.
4. If `thresh` is provided, filters out intBCs with proportions exceeding the threshold.
5. Returns the set of intBCs and the dictionary of dropout proportions.
### `compute_lg_membership`
#### Description
This function calculates a "kinship" score for a given cell against each lineage group, representing the weighted proportion of shared intBCs.
#### Inputs
| Name | Type | Description |
| :----------- | :--------------- | :------------------------------------------------------------------------------------------------------------- |
| cell | pandas.DataFrame | A DataFrame representing an allele table subsetted to a single cellBC. |
| intbc\_sets | dict | A dictionary mapping lineage group identifiers to their corresponding sets of intBCs. |
| lg\_dropouts | dict | A dictionary mapping lineage group identifiers to dictionaries of intBC dropout proportions within that group. |
#### Outputs
| Name | Type | Description |
| :--- | :------------------------------------------------------------------------------------------------- | :---------- |
| dict | A dictionary mapping lineage group identifiers to the calculated kinship score for the given cell. | |
#### Internal Logic
1. Identifies the set of intBCs present in the given cell.
2. For each lineage group:
* Calculates the intersection between the cell's intBC set and the lineage group's intBC set.
* Calculates a weighted intersection size, considering the dropout proportions of shared intBCs within the lineage group.
* Normalizes the weighted intersection by the total weighted size of the lineage group's intBC set.
3. Normalizes the kinship scores across all lineage groups to sum to 1.
### `filter_inter_doublets`
#### Description
This function identifies and filters out cells with ambiguous lineage assignments, suggesting they might be doublets formed from cells belonging to different lineages. It calculates a kinship score between each cell and its assigned lineage group and filters out cells with scores below a specified threshold.
#### Inputs
| Name | Type | Description |
| :--- | :--------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| at | pandas.DataFrame | An allele table containing cellBC-intBC-allele information. |
| rule | float | The minimum kinship score required for a cell to be retained. Cells with kinship scores below this threshold are considered potential inter-lineage doublets. Defaults to 0.35. |
#### Outputs
| Name | Type | Description |
| :--------------- | :--------------------------------------------------------------------- | :---------- |
| pandas.DataFrame | A filtered allele table with potential inter-lineage doublets removed. | |
#### Internal Logic
1. Determines the set of intBCs and their dropout proportions for each lineage group.
2. For each cell:
* Calculates the kinship score between the cell and each lineage group using the `compute_lg_membership` function.
* Compares the kinship score of the cell's assigned lineage group to the `rule` threshold.
* Filters out the cell if its kinship score with its assigned lineage group is below the threshold.
## Error Handling
The `error_correct_umis` function raises a `PreprocessError` if it encounters non-unique cellBC-UMI pairs, indicating an issue with prior UMI resolution steps.
# lineage_utils.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/lineage_utils.py
## High-level description
This file provides functions for calling lineage groups in single-cell sequencing data. It includes algorithms for identifying potential lineage groups, assigning cells to these groups based on shared integration barcodes (intBCs), and filtering out potential doublets.
## Code Structure
The code consists of several functions that work together to identify and assign cells to lineage groups. The main functions are `assign_lineage_groups`, `find_top_lg`, `filter_intbcs_lg_sets`, `score_lineage_kinships`, `annotate_lineage_groups`, `filter_intbcs_final_lineages`, and `filtered_lineage_group_to_allele_table`. These functions are called sequentially to process the input data and generate the final lineage group assignments. Additionally, there are helper functions for plotting and data manipulation.
## References
This file is referenced by `cassiopeia/preprocess/pipeline.py` in the function `call_lineage_groups`.
## Symbols
### `assign_lineage_groups`
#### Description
This function is a wrapper function that iteratively identifies and assigns cells to lineage groups. It repeatedly calls the `find_top_lg` function to find the next largest lineage group until the remaining unassigned cells form a group smaller than the specified minimum cluster size.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :------------------------------------------------------------------------------------------------------------------------- |
| pivot\_in | pd.DataFrame | The input pivot table of UMI counts for each cellBC-intBC pair |
| min\_clust\_size | int | The minimum number of cells required for a group |
| min\_intbc\_thresh | float | Minimum proportion of cells shared between an intBC and the most frequent intBC to be included in the group (default: 0.2) |
| kinship\_thresh | float | Minimum proportion of intBCs a cell needs to share with a group to be included in that group (default: 0.2) |
#### Outputs
| Name | Type | Description |
| :------------ | :----------- | :--------------------------------------------------------------- |
| piv\_assigned | pd.DataFrame | A pivot table of cells labeled with their assigned lineage group |
#### Internal Logic
The function initializes two empty DataFrames, `piv_assigned` to store the assigned cells and `pivot_in` to keep track of the remaining unassigned cells. It then enters a while loop that continues until the number of cells in the latest identified lineage group (`prev_clust_size`) is less than or equal to the specified `min_clust_size`.
Inside the loop, the function calls `find_top_lg` to identify the next largest lineage group based on the remaining unassigned cells in `pivot_in`. The identified lineage group is then concatenated with the `piv_assigned` DataFrame. The `pivot_in` DataFrame is updated by removing the cells that were assigned to the latest lineage group. This process repeats until the stopping criterion is met.
### `find_top_lg`
#### Description
This function identifies the next largest lineage group from a pivot table of UMI counts for each cellBC-intBC pair. It first identifies the most frequent intBC and then selects all intBCs that share a certain proportion of cells with the most frequent one. This set of intBCs forms the initial lineage group. The function then expands this group by including cells that share a certain proportion of intBCs with the initial group.
#### Inputs
| Name | Type | Description |
| :--------------- | :----------- | :------------------------------------------------------------------------------------------------------------------------- |
| PIVOT\_in | pd.DataFrame | The input pivot table of UMI counts for each cellBC-intBC pair |
| iteration | int | The current iteration number of the lineage group assignment process |
| min\_intbc\_prop | float | Minimum proportion of cells shared between an intBC and the most frequent intBC to be included in the group (default: 0.2) |
| kinship\_thresh | float | Minimum proportion of intBCs a cell needs to share with a group to be included in that group (default: 0.2) |
#### Outputs
| Name | Type | Description |
| :-------- | :----------- | :-------------------------------------------------------------- |
| PIV\_LG | pd.DataFrame | A pivot table of cells assigned to the identified lineage group |
| PIV\_noLG | pd.DataFrame | A pivot table of the remaining unassigned cells |
#### Internal Logic
1. **Identify the most frequent intBC:** Calculate the sum of UMIs for each intBC and select the intBC with the highest sum.
2. **Select intBCs that share a significant proportion of cells with the most frequent intBC:** Create a subset of the input pivot table containing only cells that have the most frequent intBC. Calculate the proportion of cells that each intBC shares with the most frequent intBC and select those above the `min_intbc_prop` threshold.
3. **Identify cells that share a significant proportion of intBCs with the selected intBCs:** Calculate the proportion of intBCs that each cell shares with the selected intBCs (kinship) and select cells above the `kinship_thresh` threshold.
4. **Create output DataFrames:** Generate two DataFrames: one containing the cells assigned to the identified lineage group (`PIV_LG`) and another containing the remaining unassigned cells (`PIV_noLG`).
### `filter_intbcs_lg_sets`
#### Description
This function filters out intBCs from each lineage group that are present in a low proportion of cells within that group. This step helps to remove noise and focus on the most informative intBCs for lineage reconstruction.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :-------------------------------------------------------------------------------------------------------------- |
| PIV\_assigned | pd.DataFrame | A pivot table of cells labeled with lineage group assignments |
| min\_intbc\_thresh | float | The minimum proportion of cells in a lineage group that must have an intBC for it to be retained (default: 0.2) |
#### Outputs
| Name | Type | Description |
| :------------- | :----------------------- | :------------------------------------------------------------------------------ |
| master\_LGs | List\[int] | A list of lineage group numbers |
| master\_intBCs | Dict\[int, pd.DataFrame] | A dictionary mapping each lineage group number to a list of its retained intBCs |
#### Internal Logic
1. **Iterate through each lineage group:** For each lineage group, create a binary version of the pivot table where values greater than 0 are set to 1.
2. **Calculate intBC proportions:** Calculate the proportion of cells in the lineage group that have each intBC.
3. **Filter intBCs:** Retain only the intBCs with proportions greater than or equal to `min_intbc_thresh`.
4. **Update master data structures:** Add the lineage group number to `master_LGs` and map the lineage group number to its retained intBCs in `master_intBCs`.
### `score_lineage_kinships`
#### Description
This function calculates the kinship score between each cell and each lineage group based on the shared intBCs. The kinship score represents the total UMI count of shared intBCs between a cell and a lineage group.
#### Inputs
| Name | Type | Description |
| :------------- | :----------------------- | :------------------------------------------------------------------------------ |
| PIV | pd.DataFrame | A pivot table of cells labeled with lineage group assignments |
| master\_LGs | List\[int] | A list of lineage group numbers |
| master\_intBCs | Dict\[int, pd.DataFrame] | A dictionary mapping each lineage group number to a list of its retained intBCs |
#### Outputs
| Name | Type | Description |
| :--------------- | :----------- | :------------------------------------------------------------------------------------ |
| max\_kinship\_LG | pd.DataFrame | A DataFrame containing the lineage group with the highest kinship score for each cell |
#### Internal Logic
1. **Create a binary matrix of lineage groups and their intBCs:** Generate a DataFrame (`dfLG2intBC`) where rows represent lineage groups and columns represent intBCs. A value of 1 indicates that the intBC is present in the lineage group, and 0 indicates absence.
2. **Subset the input pivot table:** Create a subset of the input pivot table (`subPIVOT`) containing only the retained intBCs.
3. **Calculate kinship scores:** Perform matrix multiplication between `subPIVOT` and the transpose of `dfLG2intBC`. This results in a matrix where each row represents a cell and each column represents a lineage group, with the values representing the kinship scores.
4. **Identify the lineage group with the highest kinship score:** For each cell, select the lineage group with the highest kinship score.
### `annotate_lineage_groups`
#### Description
This function annotates the input allele table with the lineage group assignments determined by the `score_lineage_kinships` function.
#### Inputs
| Name | Type | Description |
| :--------------- | :----------------------- | :------------------------------------------------------------------------------------ |
| dfMT | pd.DataFrame | An allele table of cellBC-UMI-allele groups |
| max\_kinship\_LG | pd.DataFrame | A DataFrame containing the lineage group with the highest kinship score for each cell |
| master\_intBCs | Dict\[int, pd.DataFrame] | A dictionary mapping each lineage group number to a list of its retained intBCs |
#### Outputs
| Name | Type | Description |
| :--- | :----------- | :-------------------------------------------------------------- |
| dfMT | pd.DataFrame | The input allele table annotated with lineage group assignments |
#### Internal Logic
1. **Create a dictionary mapping cellBCs to lineage groups:** Convert the `max_kinship_LG` DataFrame into a dictionary where keys are cellBCs and values are the corresponding lineage group assignments.
2. **Map lineage group assignments to the allele table:** Add a new column named "lineageGrp" to the input allele table (`dfMT`) and map the cellBCs to their corresponding lineage group assignments using the dictionary created in the previous step.
3. **Rename lineage groups based on size:** Rename the lineage groups in descending order of their size (number of cells).
### `filter_intbcs_final_lineages`
#### Description
This function performs a final round of filtering on the intBCs within each lineage group after the final lineage group assignments have been determined. It removes alignments associated with intBCs that are present in a low proportion of cells within their assigned lineage group.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :--------------------------------------------------------------------------------------------------------------- |
| at | pd.DataFrame | An allele table of cellBC-UMI-allele groups annotated with final lineage group assignments |
| min\_intbc\_thresh | float | The minimum proportion of cells in a lineage group that must have an intBC for it to be retained (default: 0.05) |
#### Outputs
| Name | Type | Description |
| :--- | :------------------ | :--------------------------------------------------------------------------------------------------------- |
| lgs | List\[pd.DataFrame] | A list of DataFrames, where each DataFrame represents a lineage group and contains the filtered alignments |
#### Internal Logic
1. **Iterate through each lineage group:** For each lineage group, create a pivot table of cellBCs and intBCs, with the values representing the UMI counts.
2. **Calculate intBC proportions:** Calculate the proportion of cells in the lineage group that have each intBC.
3. **Filter intBCs:** Retain only the intBCs with proportions greater than `min_intbc_thresh`.
4. **Filter alignments:** For each lineage group, keep only the alignments associated with the retained intBCs.
### `filtered_lineage_group_to_allele_table`
#### Description
This function takes the list of filtered lineage group DataFrames generated by `filter_intbcs_final_lineages` and combines them into a final allele table.
#### Inputs
| Name | Type | Description |
| :------------ | :------------------ | :--------------------------------------------------------------------------------------------------------- |
| filtered\_lgs | List\[pd.DataFrame] | A list of DataFrames, where each DataFrame represents a lineage group and contains the filtered alignments |
#### Outputs
| Name | Type | Description |
| :-------- | :----------- | :-------------------------------------------------------------- |
| final\_df | pd.DataFrame | The final processed allele table with lineage group assignments |
#### Internal Logic
1. **Concatenate lineage group DataFrames:** Combine the list of lineage group DataFrames into a single DataFrame.
2. **Group and aggregate:** Group the DataFrame by cellBC, intBC, allele, lineage group, and cutsite information, and aggregate the UMI and read counts.
### `plot_overlap_heatmap`
#### Description
This function generates a heatmap showing the overlap of intBCs between cells, providing a visual representation of potential clonal populations.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------- | :----------------------------------------------------- |
| at | pd.DataFrame | An allele table of cellBC-UMI-allele groups |
| at\_pivot\_I | pd.DataFrame | A pivot table indicating which cellBCs have which UMIs |
| output\_directory | str | The directory to save the generated plot |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :------------------------------------------------------------ |
| None | | The function saves the plot to the specified output directory |
#### Internal Logic
1. **Create a list of all unique intBCs:** Extract all unique intBCs from the input allele table.
2. **Subset the pivot table:** Select only the columns in the pivot table that correspond to the unique intBCs.
3. **Generate and save the heatmap:** Create a heatmap using the subset pivot table, with cellBCs as rows and intBCs as columns. The heatmap visually represents the presence or absence of each intBC in each cell.
### `plot_overlap_heatmap_lg`
#### Description
This function generates
# map_utils.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/map_utils.py
## High-level description
This code defines a function `map_intbcs` that processes a molecule table to resolve allele ambiguity for each cellBC/intBC pair. It identifies the most frequent allele based on read count and UMI count, then filters out alignments with other alleles for each pair.
## Symbols
### `map_intbcs`
#### Description
This function takes a molecule table as input and returns a modified allele table where each cellBC/intBC pair is associated with a single allele. It achieves this by grouping the input table by cellBC, intBC, and allele, then selecting the allele with the highest read count (and UMI count as a tie-breaker) for each group. Alignments with other alleles for the same cellBC/intBC pair are removed.
#### Inputs
| Name | Type | Description |
| :-------------- | :--------------- | :--------------------------------------------------------------------------------- |
| molecule\_table | pandas.DataFrame | A molecule table containing cellBC, intBC, allele, readCount, and UMI information. |
#### Outputs
| Name | Type | Description |
| :------------ | :--------------- | :--------------------------------------------------------------------------------------- |
| mapped\_table | pandas.DataFrame | A modified allele table where each cellBC/intBC pair is associated with a single allele. |
#### Internal Logic
1. Drops rows with missing `intBC` values from the input `molecule_table`.
2. Groups the remaining rows by `cellBC`, `intBC`, and `allele`.
3. Aggregates the groups by summing `readCount` and counting `UMI`.
4. Sorts the aggregated table first by `UMI` in descending order, then by `readCount` in descending order.
5. Identifies duplicate `cellBC`/`intBC` pairs, keeping only the first occurrence (which corresponds to the highest read and UMI count).
6. Creates a set of tuples representing the selected `cellBC`, `intBC`, and `allele` combinations.
7. Filters the original `molecule_table` to keep only rows where the `cellBC`, `intBC`, and `allele` combination exists in the set created in step 6.
8. Logs the number of removed alleles and UMIs during the filtering process.
9. Returns the filtered `molecule_table`.
## References
* `logger`: Used for logging debug messages. Imported from `cassiopeia.mixins`.
* `utilities.log_molecule_table`: A decorator function used to log statistics of the molecule table before and after the function execution. Imported from `cassiopeia.preprocess`.
## Side Effects
* Logs debug messages indicating the number of alleles and UMIs removed during the filtering process.
## Performance Considerations
The function uses pandas groupby and apply operations, which can be computationally expensive for large datasets. Consider optimizing these operations or using alternative data structures if performance becomes a bottleneck.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/overview
## High-level description
This directory contains the preprocessing module for Cassiopeia, a tool for phylogenetic inference from single-cell sequencing data. The preprocessing pipeline handles tasks such as converting raw sequencing data to BAM format, filtering reads, error-correcting barcodes and UMIs, collapsing UMIs, aligning sequences, calling alleles, and preparing data for lineage reconstruction.
## What does it do?
The preprocessing pipeline takes raw sequencing data as input and performs the following main steps:
1. Converts FASTQ files to BAM format
2. Filters low-quality reads
3. Error-corrects cell barcodes and UMIs
4. Collapses UMIs to reduce noise
5. Aligns sequences to a reference
6. Calls alleles (indels) at target sites
7. Filters and processes the resulting molecule table
8. Calls lineage groups
9. Converts the processed data into formats suitable for phylogenetic inference
These steps prepare the raw sequencing data for downstream analysis, including the construction of lineage trees.
## Entry points
The main entry point for the preprocessing pipeline is the `cassiopeia_preprocess.py` script. This script orchestrates the execution of the various preprocessing steps based on a user-provided configuration file.
The `pipeline.py` file contains the core functions for each preprocessing step, which are called by the main script.
The `setup_utilities.py` file provides functions for parsing the configuration file and setting up the pipeline.
## Key Files
* `UMI_utils.py`: Contains functions for collapsing and preprocessing Unique Molecular Identifiers (UMIs).
* `alignment_utilities.py`: Provides utilities for sequence alignment and CIGAR string parsing.
* `constants.py`: Defines constants and default parameters used throughout the preprocessing pipeline.
* `doublet_utils.py`: Contains functions for identifying and filtering potential cell doublets.
* `lineage_utils.py`: Provides functions for calling lineage groups and processing lineage-related data.
* `map_utils.py`: Contains functions for resolving allele ambiguity in molecule tables.
* `utilities.py`: Offers various utility functions for data filtering, conversion, and manipulation.
## Dependencies
The preprocessing module relies on several external libraries:
* pandas: For data manipulation and analysis
* numpy: For numerical operations
* pysam: For reading and manipulating BAM files
* ngs\_tools: For sequence manipulation and analysis
* pylab: For plotting and visualization
* tqdm: For progress bar visualization
## Configuration
The preprocessing pipeline is configured using an INI-format configuration file. This file specifies parameters for each preprocessing step, including input/output paths, filtering thresholds, and algorithm-specific settings. The `DEFAULT_PIPELINE_PARAMETERS` dictionary in `constants.py` defines the default values for these parameters.
Key configurable parameters include:
* Input/output file paths
* Sequencing chemistry (e.g., 10xv2, 10xv3)
* Quality thresholds for filtering
* UMI collapsing parameters
* Alignment settings
* Allele calling parameters
* Cell and UMI filtering thresholds
Users can customize these parameters in the configuration file to adapt the pipeline to their specific experimental setup and data characteristics.
In summary, the Cassiopeia preprocessing module provides a comprehensive and flexible pipeline for processing single-cell lineage tracing data, preparing it for downstream phylogenetic analysis.
# pipeline.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/pipeline.py
## High-level description
This file contains functions for preprocessing sequencing data into a character matrix, which is suitable for phylogenetic inference. It provides a series of functions that perform various steps of the preprocessing pipeline, including converting FASTQ files to BAM, filtering reads, collapsing UMIs, aligning sequences, calling alleles, error correcting barcodes and UMIs, filtering molecule tables, and calling lineage groups.
## Code Structure
The code defines several functions that are used sequentially in a preprocessing pipeline. The functions are designed to be modular and can be invoked individually or as part of a larger pipeline. The main functions include `convert_fastqs_to_unmapped_bam`, `filter_bam`, `error_correct_cellbcs_to_whitelist`, `collapse_umis`, `resolve_umi_sequence`, `align_sequences`, `call_alleles`, `error_correct_intbcs_to_whitelist`, `error_correct_umis`, `filter_molecule_table`, and `call_lineage_groups`. Each function performs a specific step in the preprocessing pipeline and takes a DataFrame or a file path as input and returns a processed DataFrame or a file path.
## References
This file references constants defined in `cassiopeia.preprocess.constants`, utility functions from `cassiopeia.preprocess.utilities`, and alignment utilities from `cassiopeia.preprocess.alignment_utilities`. It also uses functions from external libraries such as `ngs_tools`, `pysam`, and `pandas`.
## Symbols
### `convert_fastqs_to_unmapped_bam`
#### Description
Converts FASTQ files to an unmapped BAM file based on the specified chemistry. It adds appropriate BAM tags based on the chemistry used.
#### Inputs
| Name | Type | Description |
| :---------------- | :-------------------------------------------------------------- | :----------------------------------------------------------- |
| fastq\_fps | List\[str] | List of paths to FASTQ files. |
| chemistry | Literal\["dropseq", "10xv2", "10xv3", "indropsv3", "slideseq2"] | Sample-prep/sequencing chemistry used. |
| output\_directory | str | The output directory where the unmapped BAM will be written. |
| name | Optional\[str] | Name of the reads in the FASTQs. |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :------ | :--- | :---------------------------- |
| bam\_fp | str | Path to the written BAM file. |
#### Internal Logic
The function first checks if the provided chemistry is supported. It then retrieves the appropriate BAM tags for the chemistry and converts the FASTQ files to an unmapped BAM file using the `ngs.fastq.fastqs_to_bam_with_chemistry` function from the `ngs_tools` library.
### `filter_bam`
#### Description
Filters reads in a BAM file that have low quality barcode or UMIs.
#### Inputs
| Name | Type | Description |
| :----------------- | :--- | :----------------------------------------------------------------------------------------------------------------------------------------- |
| bam\_fp | str | Input BAM filepath containing reads to filter. |
| output\_directory | str | The output directory where the filtered BAM will be written. |
| quality\_threshold | int | Every base of the barcode and UMI sequence for a given read must have at least this PHRED quality score to pass filtering. Defaults to 10. |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :----------- | :--- | :----------------------------- |
| filtered\_fp | str | Path to the filtered BAM file. |
#### Internal Logic
The function defines a filter function that checks the quality scores of the barcode and UMI sequences. It then uses the `ngs.bam.filter_bam` function from the `ngs_tools` library to filter the input BAM file based on the filter function.
### `error_correct_cellbcs_to_whitelist`
#### Description
Error-corrects cell barcodes in the input BAM file to a provided whitelist.
#### Inputs
| Name | Type | Description |
| :---------------- | :---------------------- | :---------------------------------------------------------------------------------------- |
| bam\_fp | str | Input BAM filepath containing raw barcodes. |
| whitelist | Union\[str, List\[str]] | Either a path to a plaintext file containing the barcode whitelist or a list of barcodes. |
| output\_directory | str | The output directory where the corrected BAM will be written. |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :------------ | :--- | :------------------------------ |
| corrected\_fp | str | Path to the corrected BAM file. |
#### Internal Logic
The function first reads the whitelist from the provided file or list. It then extracts the raw barcodes and their qualities from the input BAM file. It uses the `ngs.sequence.correct_sequences_to_whitelist` function from the `ngs_tools` library to correct the barcodes to the whitelist. Finally, it writes the corrected barcodes to a new BAM file.
### `collapse_umis`
#### Description
Collapses close UMIs together from a BAM file.
#### Inputs
| Name | Type | Description |
| :------------------ | :------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- |
| bam\_fp | str | File path of the BAM file. |
| output\_directory | str | The output directory where the sorted BAM directory, the collapsed BAM directory, and the final collapsed table are written. |
| max\_hq\_mismatches | int | A threshold specifying the maximum number of high quality mismatches between the sequences of two aligned segments to be collapsed. Defaults to 3. |
| max\_indels | int | A threshold specifying the maximum number of differing indels allowed between the sequences of two aligned segments to be collapsed. Defaults to 2. |
| method | Literal\["cutoff", "likelihood"] | Which method to use to form initial sequence clusters. Defaults to "cutoff". |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :--- | :----------- | :------------------------------ |
| df | pd.DataFrame | A DataFrame of collapsed reads. |
#### Internal Logic
The function first sorts the input BAM file by cell barcode and UMI using the `sort_bam` function. It then forms collapsed clusters of UMIs using the `form_collapsed_clusters` function. Finally, it converts the collapsed BAM file to a DataFrame using the `utilities.convert_bam_to_df` function.
### `resolve_umi_sequence`
#### Description
Resolves a consensus sequence for each UMI.
#### Inputs
| Name | Type | Description |
| :------------------------ | :----------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| molecule\_table | pd.DataFrame | Molecule table to resolve. |
| output\_directory | str | Directory to store results. |
| min\_umi\_per\_cell | int | The threshold specifying the minimum number of UMIs in a cell needed to be retained during filtering. Defaults to 10. |
| min\_avg\_reads\_per\_umi | float | The threshold specifying the minimum coverage (i.e. average) reads per UMI in a cell needed for that cell to be retained during filtering. Defaults to 2.0. |
| plot | bool | Whether to plot the number of sequences per UMI and diagnostics after resolving. Defaults to True. |
#### Outputs
| Name | Type | Description |
| :-------------------- | :----------- | :-------------------------------------------------------------- |
| filt\_molecule\_table | pd.DataFrame | A molecule table with unique mappings between cellBC-UMI pairs. |
#### Internal Logic
The function first groups the molecule table by cell barcode and UMI. For each group, it selects the most abundant sequence as the consensus sequence and marks the remaining sequences for filtering. It then filters out the marked sequences and applies cell filtering based on the minimum number of UMIs per cell and the minimum average reads per UMI.
### `align_sequences`
#### Description
Aligns reads to the TargetSite reference.
#### Inputs
| Name | Type | Description |
| :------------------- | :-------------------------- | :---------------------------------------------------- |
| queries | pd.DataFrame | DataFrame storing a list of sequences to align. |
| ref\_filepath | Optional\[str] | Filepath to the reference FASTA. |
| ref | Optional\[str] | Reference sequence. |
| gap\_open\_penalty | float | Gap open penalty. Defaults to 20. |
| gap\_extend\_penalty | float | Gap extension penalty. Defaults to 1. |
| method | Literal\["local", "global"] | What alignment algorithm to use. Defaults to "local". |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :------------ | :----------- | :------------------------------------------------------------------------------------------------ |
| alignment\_df | pd.DataFrame | A DataFrame mapping each sequence name to the CIGAR string, quality, and original query sequence. |
#### Internal Logic
The function first reads the reference sequence from the provided file or string. It then aligns all unique sequences to the reference using either local or global alignment, based on the `method` argument. The alignments are performed using the `alignment_utilities.align_local` or `alignment_utilities.align_global` functions. Finally, the alignments are merged into the input DataFrame.
### `call_alleles`
#### Description
Calls indels from CIGAR strings.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :-------------------------------------------------------------------------------------------------- |
| alignments | pd.DataFrame | Alignments provided in DataFrame. |
| ref\_filepath | Optional\[str] | Filepath to the reference sequence. |
| ref | Optional\[str] | Nucleotide sequence of the reference. |
| barcode\_interval | Tuple\[int, int] | Interval in reference corresponding to the integration barcode. Defaults to (20, 34). |
| cutsite\_locations | List\[int] | A list of all cutsite positions in the reference. Defaults to \[112, 166, 220]. |
| cutsite\_width | int | Number of nucleotides left and right of cutsite location that indels can appear in. Defaults to 12. |
| context | bool | Include sequence context around indels. Defaults to True. |
| context\_size | int | Number of bases to the right and left to include as context. Defaults to 5. |
#### Outputs
| Name | Type | Description |
| :--------- | :----------- | :---------------------------------------------------------------- |
| alignments | pd.DataFrame | A DataFrame mapping each sequence alignment to the called indels. |
#### Internal Logic
The function first reads the reference sequence from the provided file or string. It then iterates through the alignments and parses the CIGAR strings using the `alignment_utilities.parse_cigar` function to extract the indels at each cutsite. The extracted indels are then added to the input DataFrame.
### `error_correct_intbcs_to_whitelist`
#### Description
Corrects all intBCs to the provided whitelist.
#### Inputs
| Name | Type | Description |
| :------------------ | :---------------------- | :-------------------------------------------------------------------------------------------------------------------------------- |
| input\_df | pd.DataFrame | Input DataFrame of alignments. |
| whitelist | Union\[str, List\[str]] | Either a path to a plaintext file containing the barcode whitelist or a list of barcodes. |
| intbc\_dist\_thresh | int | The threshold specifying the maximum Levenshtein distance between the read sequence and whitelist to be corrected. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :-------- | :----------- | :------------------------------------- |
| input\_df | pd.DataFrame | A DataFrame of error corrected intBCs. |
#### Internal Logic
The function first reads the whitelist from the provided file or list. It then iterates through the unique intBCs in the input DataFrame and calculates the Levenshtein distance to each whitelist intBC. If the distance is less than or equal to the threshold and there is only one matching whitelist intBC, the intBC is corrected.
### `error_correct_umis`
#### Description
Within cellBC-intBC pairs, collapses UMIs that have close sequences.
#### Inputs
| Name | Type | Description |
| :----------------------- | :----------- | :-------------------------------------------------------------------------------------------------------------------- |
| input\_df | pd.DataFrame | Input DataFrame of alignments. |
| max\_umi\_distance | int | The threshold specifying the Maximum Hamming distance between UMIs for one to be corrected to another. Defaults to 2. |
| allow\_allele\_conflicts | bool | Whether or not to include the allele when splitting UMIs into allele groups. Defaults to False. |
| n\_threads | int | Number of threads to use. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :------------ | :----------- | :----------------------------------- |
| alignment\_df | pd.DataFrame | A DataFrame of error corrected UMIs. |
#### Internal Logic
The function first groups the input DataFrame by cell barcode, intBC, and optionally allele. For each group, it corrects UMIs that have a Hamming distance less than or equal to the threshold using the `UMI_utils.correct_umis_in_group` function. The corrected UMIs are then merged back into the input DataFrame.
### `filter_molecule_table`
#### Description
Filters and corrects a molecule table of cellBC-UMI pairs.
#### Inputs
| Name | Type | Description |
| :------------------ | :----------- | :------------------------------------------------------- |
| input\_df | pd.DataFrame | A molecule table, i.e. cellBC-UMI pairs. |
| output\_directory | str | The output directory path to store plots. |
| min\_umi\_per\_cell | int | The threshold specifying the minimum number of UMIs in a |
# setup_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/setup_utilities.py
## High-level description
The `setup_utilities.py` file provides utility functions for setting up and configuring the Cassiopeia preprocessing pipeline. It handles tasks such as parsing configuration files, creating output directories, and defining the sequence of preprocessing stages.
## Code Structure
The `setup_utilities.py` file contains three main functions: `setup`, `parse_config`, and `create_pipeline`. The `setup` function initializes the output directory and logging. The `parse_config` function reads and parses the configuration file, setting default values and validating required parameters. The `create_pipeline` function determines the order of preprocessing stages based on the specified entry and exit points.
## References
This code references the following symbols:
* `cassiopeia.preprocess.constants.DEFAULT_PIPELINE_PARAMETERS`: A dictionary containing default parameters for each stage of the preprocessing pipeline.
* `cassiopeia.mixins.UnspecifiedConfigParameterError`: An exception raised when a required configuration parameter is not specified.
* `cassiopeia.preprocess.cassiopeia_preprocess.STAGES`: A dictionary mapping stage names to their corresponding functions in the preprocessing pipeline.
## Symbols
### `setup`
#### Description
Sets up the environment for the preprocessing pipeline by creating the output directory and configuring logging.
#### Inputs
| Name | Type | Description |
| :-------------------------- | :--- | :--------------------------------- |
| output\_directory\_location | str | The path to the output directory. |
| verbose | bool | Whether to enable verbose logging. |
#### Side Effects
* Creates the output directory if it doesn't exist.
* Adds a file handler to the logger, writing logs to `preprocess.log` and `preprocess.err` files in the output directory.
### `parse_config`
#### Description
Parses the configuration file for the preprocessing pipeline, setting default values and validating required parameters.
#### Inputs
| Name | Type | Description |
| :------------- | :--- | :------------------------------------------ |
| config\_string | str | The configuration file content as a string. |
#### Outputs
| Name | Type | Description |
| :--------- | :-------------------------- | :--------------------------------------------------------------- |
| parameters | Dict\[str, Dict\[str, Any]] | A dictionary containing parameters for each preprocessing stage. |
#### Internal Logic
1. Reads the default pipeline parameters from `constants.DEFAULT_PIPELINE_PARAMETERS`.
2. Parses the provided `config_string` using the `configparser` module.
3. Updates the default parameters with values from the parsed configuration.
4. Validates that required parameters are present in the `general` section of the configuration.
5. Adds necessary parameters from the `general` section to other stages.
6. Returns the parsed and validated parameters.
#### Error Handling
Raises an `UnspecifiedConfigParameterError` if any required parameter is missing from the configuration.
### `create_pipeline`
#### Description
Creates a list of preprocessing stages to be executed based on the specified entry and exit points.
#### Inputs
| Name | Type | Description |
| :----- | :-------------- | :----------------------------------------------------------------- |
| entry | str | The name of the stage to start the pipeline from. |
| \_exit | str | The name of the stage to stop the pipeline at. |
| stages | Dict\[str, Any] | A dictionary mapping stage names to their corresponding functions. |
#### Outputs
| Name | Type | Description |
| :--------------- | :--------- | :--------------------------------------------- |
| pipeline\_stages | List\[str] | A list of stage names to be executed in order. |
#### Internal Logic
1. Gets a list of stage names from the `stages` dictionary.
2. Finds the indices of the `entry` and `_exit` stages in the list.
3. Returns a slice of the list from the `entry` index to the `_exit` index (inclusive).
# utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/preprocess/utilities.py
## High-level description
This file provides utility functions for the Cassiopeia-Preprocess pipeline, including filtering cells and UMIs, error correcting integration barcodes, converting allele tables to character matrices and lineage profiles, and computing empirical indel priors. These functions are used to process and prepare sequencing data for phylogenetic inference.
## Code Structure
The file defines several functions that operate on Pandas DataFrames representing molecule tables and allele tables. These functions perform filtering, error correction, and conversion operations to prepare the data for downstream analysis. Some functions are decorated with `log_molecule_table` and `log_runtime` decorators to provide logging and runtime information.
## References
This file references symbols from the `ngs_tools` library for sequence manipulation and analysis, as well as the `cassiopeia.mixins` module for logging and warnings.
## Symbols
### `filter_cells`
#### Description
Filters out cell barcodes that have too few UMIs or too few reads per UMI, based on user-defined thresholds.
#### Inputs
| Name | Type | Description |
| :------------------------ | :----------- | :--------------------------------------------------------------------------------------------------------- |
| molecule\_table | pd.DataFrame | A molecule table of cellBC-UMI pairs to be filtered |
| min\_umi\_per\_cell | int | Minimum number of UMIs per cell for cell to not be filtered. Defaults to 10. |
| min\_avg\_reads\_per\_umi | float | Minimum average reads per UMI in a cell needed in order for that cell not to be filtered. Defaults to 2.0. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :----------- | :------------------------ |
| filtered\_molecule\_table | pd.DataFrame | A filtered molecule table |
#### Internal Logic
1. Groups the molecule table by cell barcode.
2. Calculates the number of UMIs and average reads per UMI for each cell.
3. Filters out cells that do not meet the minimum thresholds for both UMIs per cell and average reads per UMI.
### `filter_umis`
#### Description
Filters out UMIs with too few reads, based on a user-defined threshold.
#### Inputs
| Name | Type | Description |
| :------------------- | :----------- | :--------------------------------------------------------------------------- |
| molecule\_table | pd.DataFrame | A molecule table of cellBC-UMI pairs to be filtered |
| min\_reads\_per\_umi | int | The minimum read count needed for a UMI to not be filtered. Defaults to 100. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :----------- | :------------------------ |
| filtered\_molecule\_table | pd.DataFrame | A filtered molecule table |
#### Internal Logic
1. Filters the molecule table to keep only UMIs with a read count greater than or equal to the minimum threshold.
### `error_correct_intbc`
#### Description
Error corrects close integration barcodes (intBCs) with small enough unique UMI counts.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :------------------------------------------------------------------------------------------- |
| molecule\_table | pd.DataFrame | A molecule table of cellBC-UMI pairs to be filtered |
| prop | float | Proportion by which to filter integration barcodes. Defaults to 0.5. |
| umi\_count\_thresh | int | Maximum UMI count for which to correct barcodes. Defaults to 10. |
| dist\_thresh | int | Barcode distance threshold, to decide what's similar enough to error correct. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :------------------------- | :----------- | :-------------------------------------------------- |
| corrected\_molecule\_table | pd.DataFrame | Filtered molecule table with error corrected intBCs |
#### Internal Logic
1. Groups the molecule table by cellBC, intBC, and allele.
2. Iterates through pairs of intBCs within each cellBC-allele group.
3. Corrects an intBC to another if:
* They have the same allele.
* The Levenshtein distance between their sequences is less than or equal to `dist_thresh`.
* The number of UMIs of the intBC to be changed is less than or equal to `umi_count_thresh`.
* The proportion of the UMI count in the intBC to be changed out of the total UMI count in both intBCs is less than `prop`.
### `record_stats`
#### Description
Records the number of UMIs per intBC and per cellBC, as well as the read counts for each alignment.
#### Inputs
| Name | Type | Description |
| :-------------- | :----------- | :------------------------ |
| molecule\_table | pd.DataFrame | A DataFrame of alignments |
#### Outputs
| Name | Type | Description |
| :---------------- | :------- | :------------------------------ |
| read\_counts | np.array | Read counts for each alignment |
| umis\_per\_intBC | np.array | Number of unique UMIs per intBC |
| umis\_per\_cellBC | np.array | Number of UMIs per cellBC |
#### Internal Logic
1. Groups the molecule table by cellBC and intBC to count UMIs per intBC.
2. Groups the molecule table by cellBC to count UMIs per cellBC.
3. Extracts the read counts from the 'readCount' column.
### `convert_bam_to_df`
#### Description
Converts a BAM file to a Pandas DataFrame.
#### Inputs
| Name | Type | Description |
| :------- | :--- | :-------------------------------------------------- |
| data\_fp | str | The input filepath for the BAM file to be converted |
#### Outputs
| Name | Type | Description |
| :--- | :----------- | :------------------------------------------------ |
| df | pd.DataFrame | A Pandas dataframe containing the BAM information |
#### Internal Logic
1. Opens the BAM file using `pysam.AlignmentFile`.
2. Iterates through each alignment in the BAM file.
3. Extracts relevant information from the alignment, including cellBC, UMI, readCount, grpFlag, sequence, quality scores, and read name.
4. Creates a Pandas DataFrame from the extracted information.
### `convert_alleletable_to_character_matrix`
#### Description
Converts an AlleleTable into a character matrix, suitable for input into a CassiopeiaSolver object.
#### Inputs
| Name | Type | Description |
| :-------------------- | :---------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| alleletable | pd.DataFrame | Allele Table to be converted into a character matrix |
| ignore\_intbcs | List\[str] | A set of intBCs to ignore |
| allele\_rep\_thresh | float | A threshold for removing target sites that have an allele represented by this proportion |
| missing\_data\_allele | Optional\[str] | Value in the allele table that indicates that the cut-site is missing. This will be converted into `missing_data_state` |
| missing\_data\_state | int | A state to use for missing data. Defaults to -1. |
| mutation\_priors | Optional\[pd.DataFrame] | A table storing the prior probability of a mutation occurring. This table is used to create a character matrix-specific probability dictionary for reconstruction. |
| cut\_sites | Optional\[List\[str]] | Columns in the AlleleTable to treat as cut sites. If None, we assume that the cut-sites are denoted by columns of the form "r" (e.g. "r1") |
| collapse\_duplicates | bool | Whether or not to collapse duplicate character states present for a single cellBC-intBC pair. This option has no effect if there are no allele conflicts. Defaults to True. |
#### Outputs
| Name | Type | Description |
| :------------------- | :---------------------------- | :--------------------------------------------------- |
| character\_matrix | pd.DataFrame | A character matrix |
| prior\_probs | Dict\[int, Dict\[int, float]] | A probability dictionary |
| indel\_to\_charstate | Dict\[int, Dict\[int, str]] | A dictionary mapping states to the original mutation |
#### Internal Logic
1. Filters out samples with intBCs in `ignore_intbcs`.
2. Iterates through each cellBC and intBC-cut\_site combination.
3. Assigns a unique integer state to each unique indel observed at a cut site.
4. Constructs the character matrix by mapping cellBCs to their corresponding character states.
5. Optionally collapses duplicate character states for a single cellBC-intBC pair.
6. Optionally calculates prior probabilities for each character state based on the provided `mutation_priors`.
### `convert_alleletable_to_lineage_profile`
#### Description
Converts an AlleleTable to a lineage profile, which is a pivot table over the cellBC / intBCs.
#### Inputs
| Name | Type | Description |
| :------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| allele\_table | pd.DataFrame | AlleleTable |
| cut\_sites | Optional\[List\[str]] | Columns in the AlleleTable to treat as cut sites. If None, we assume that the cut-sites are denoted by columns of the form "r" (e.g. "r1") |
| collapse\_duplicates | bool | Whether or not to collapse duplicate character states present for a single cellBC-intBC pair. This option has no effect if there are no allele conflicts. Defaults to True. |
#### Outputs
| Name | Type | Description |
| :--------------- | :----------- | :--------------------- |
| lineage\_profile | pd.DataFrame | An NxM lineage profile |
#### Internal Logic
1. Groups the allele table by cellBC and intBC.
2. Creates a multi-index DataFrame with cellBCs as rows and intBC-cut\_site combinations as columns.
3. Fills the DataFrame with the observed indels for each cellBC-intBC-cut\_site combination.
4. Optionally collapses duplicate character states for a single cellBC-intBC pair.
5. Reorders the columns based on the frequency of intBCs.
### `convert_lineage_profile_to_character_matrix`
#### Description
Converts a lineage profile to a character matrix, where indels are abstracted into integers.
#### Inputs
| Name | Type | Description |
| :------------------------- | :---------------------- | :----------------------------------------------------- |
| lineage\_profile | pd.DataFrame | Lineage profile |
| indel\_priors | Optional\[pd.DataFrame] | Dataframe mapping indels to prior probabilities |
| missing\_allele\_indicator | Optional\[str] | An allele that is being used to represent missing data |
| missing\_state\_indicator | int | State to indicate missing data. Defaults to -1. |
#### Outputs
| Name | Type | Description |
| :------------------- | :---------------------------- | :----------------------------------------------------- |
| character\_matrix | pd.DataFrame | A character matrix |
| prior\_probs | Dict\[int, Dict\[int, float]] | Prior probability dictionary |
| indel\_to\_charstate | Dict\[int, Dict\[int, str]] | Mapping from character/state pairs to indel identities |
#### Internal Logic
1. Replaces missing values with "Missing" and optionally replaces a specific missing allele indicator with "Missing".
2. Assigns a unique integer state to each unique indel observed at a cut site.
3. Constructs the character matrix by mapping cellBCs to their corresponding character states.
4. Optionally calculates prior probabilities for each character state based on the provided `indel_priors`.
### `compute_empirical_indel_priors`
#### Description
Computes indel prior probabilities from the input allele table.
#### Inputs
| Name | Type | Description |
| :------------------ | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| allele\_table | pd.DataFrame | AlleleTable |
| grouping\_variables | List\[str] | Variables to stratify data by, to treat as independent groups in counting indel occurrences. These must be columns in the allele table. Defaults to \["intBC"]. |
| cut\_sites | Optional\[List\[str]] | Columns in the AlleleTable to treat as cut sites. If None, we assume that the cut-sites are denoted by columns of the form "r" (e.g. "r1") |
#### Outputs
| Name | Type | Description |
| :------------ | :----------- | :------------------------------------------------------ |
| indel\_priors | pd.DataFrame | A DataFrame mapping indel identities to the probability |
#### Internal Logic
1. Groups the allele table by the specified `grouping_variables`.
2. Counts the number of times each unique indel occurs in each group.
3. Calculates the frequency of each indel by dividing its count by the total number of groups.
### `get_default_cut_site_columns`
#### Description
Retrieves the default cut-sites columns of an AlleleTable, assuming the AlleleTable was created using the Cassiopeia pipeline.
#### Inputs
| Name | Type | Description |
| :------------ | :----------- | :---------- |
| allele\_table | pd.DataFrame | AlleleTable |
#### Outputs
| Name | Type | Description |
| :--------- | :--------- | :-------------------------------------------------------- |
| cut\_sites | List\[str] | Columns in the AlleleTable corresponding to the cut sites |
#### Internal Logic
1. Selects columns from the allele table that match the pattern "r" (e.g., "r1", "r2").
## Dependencies
| Dependency | Purpose |
| :--------- | :--------------------------------- |
| ngs\_tools | Sequence manipulation and analysis |
| pandas | Data manipulation and analysis |
| numpy | Numerical operations |
| pylab | Plotting and visualization |
| pysam | Reading and manipulating BAM files |
| re | Regular expression operations |
| tqdm | Progress bar visualization |
## Error Handling
The
# BirthDeathFitnessSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/BirthDeathFitnessSimulator.py
## High-level description
The `BirthDeathFitnessSimulator` class simulates phylogenetic trees using a forward birth-death process, incorporating fitness variations among lineages. It allows users to define custom birth and death waiting time distributions, mutation rates, and fitness effects, enabling the simulation of diverse evolutionary scenarios. The simulation can be stopped based on either the number of extant lineages or a predefined experiment time.
## Code Structure
The `BirthDeathFitnessSimulator` class inherits from the `TreeSimulator` abstract class. It implements the `simulate_tree` method, which orchestrates the simulation process. The simulation involves initializing a tree, maintaining a priority queue of lineages, and iteratively sampling events (birth or death) for each lineage. The `sample_lineage_event` method handles the event sampling and updates the tree accordingly. The `update_fitness` method calculates the fitness of a lineage based on its mutations and a user-defined fitness distribution.
## References
This class references the `CassiopeiaTree` class for representing the simulated tree and the `TreeSimulatorError` exception for handling errors during simulation.
## Symbols
### `BirthDeathFitnessSimulator`
#### Description
This class simulates phylogenetic trees using a forward birth-death process with fitness variations. It allows users to define custom birth and death waiting time distributions, mutation rates, and fitness effects.
#### Inputs
| Name | Type | Description |
| :--------------------------- | :------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| birth\_waiting\_distribution | Callable\[\[float], float] | A function that samples waiting times from the birth distribution. |
| initial\_birth\_scale | float | The initial scale parameter for the birth distribution. |
| death\_waiting\_distribution | Optional\[Callable\[\[], float]] | A function that samples waiting times from the death distribution. Defaults to no death (infinite waiting time). |
| mutation\_distribution | Optional\[Callable\[\[], int]] | A function that samples the number of mutations occurring at a division event. Defaults to no mutations. |
| fitness\_distribution | Optional\[Callable\[\[], float]] | A function that samples the exponential that the fitness base is raised by, determining the strength of fitness mutations. Required if `mutation_distribution` is provided. |
| fitness\_base | float | The base that is raised by the value given by the fitness distribution, determining the base strength of fitness mutations. Defaults to Euler's constant (e). |
| num\_extant | Optional\[int] | The number of extant lineages as a stopping condition. |
| experiment\_time | Optional\[float] | The total experiment time as a stopping condition. |
| collapse\_unifurcations | bool | Whether to collapse unifurcations in the resulting tree. Defaults to True. |
| random\_seed | int | A seed for reproducibility. |
| initial\_tree | Optional\[CassiopeiaTree] | A tree used for initializing the simulation. |
#### Outputs
The class itself does not have outputs. Its main functionality is to simulate a tree, which is returned by the `simulate_tree` method.
#### Internal Logic
The simulation process involves the following steps:
1. **Initialization:** A tree is initialized with a single root node.
2. **Lineage Queue:** A priority queue is created to store active lineages, sorted by their next event time.
3. **Event Sampling:** For each lineage in the queue, the waiting times for birth and death events are sampled.
4. **Lineage Update:** The lineage with the earliest event is processed. If it's a birth event, a new lineage is created and added to the queue. If it's a death event, the lineage is marked as inactive.
5. **Fitness Update:** At each birth event, the fitness of the new lineage is updated based on the `mutation_distribution` and `fitness_distribution`.
6. **Stopping Condition:** The simulation continues until either the desired number of extant lineages (`num_extant`) is reached or the experiment time (`experiment_time`) is exceeded.
7. **Tree Population:** The simulated tree is populated with metadata, including branch lengths, node times, and character states.
#### Side Effects
The `simulate_tree` method modifies the internal state of the `BirthDeathFitnessSimulator` object by creating and populating a `CassiopeiaTree` object.
### `simulate_tree`
#### Description
This method simulates a phylogenetic tree using the defined birth-death process with fitness variations.
#### Inputs
This method does not take any explicit inputs.
#### Outputs
| Name | Type | Description |
| :--- | :------------- | :--------------------------------------------------------- |
| tree | CassiopeiaTree | A `CassiopeiaTree` object representing the simulated tree. |
#### Internal Logic
The method follows the internal logic described for the `BirthDeathFitnessSimulator` class. It uses helper functions like `initialize_tree`, `make_initial_lineage_dict`, `make_lineage_dict`, and `sample_lineage_event` to manage the simulation process.
#### Error Handling
The method raises a `TreeSimulatorError` if all lineages die before a stopping condition is met.
### `sample_lineage_event`
#### Description
This helper function samples a birth or death event for a given lineage and updates the tree accordingly.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------------------------- | :--------------------------------------------- |
| lineage | Dict\[str, Union\[int, float]] | A dictionary representing the current lineage. |
| current\_lineages | PriorityQueue | The priority queue of active lineages. |
| tree | nx.DiGraph | The NetworkX DiGraph representing the tree. |
| names | Generator | A generator for unique node names. |
| observed\_nodes | List\[str] | A list of observed nodes. |
#### Outputs
This method does not have any explicit outputs. It modifies the `tree` and `current_lineages` objects in place.
#### Internal Logic
The method samples birth and death waiting times for the lineage. If a birth event occurs first, a new lineage is created, its fitness is updated, and it's added to the queue. If a death event occurs first, the lineage is marked as inactive. The method also handles cases where the lineage would live past the experiment time, cutting off the branch length and marking the lineage as inactive.
#### Error Handling
The method raises a `TreeSimulatorError` if a negative waiting time is sampled or a non-active lineage is passed.
### `update_fitness`
#### Description
This method updates the fitness of a lineage by scaling its birth scale parameter based on the number and strength of mutations.
#### Inputs
| Name | Type | Description |
| :----------- | :---- | :------------------------------------------------ |
| birth\_scale | float | The current birth scale parameter of the lineage. |
#### Outputs
| Name | Type | Description |
| :-------------------- | :---- | :--------------------------------- |
| updated\_birth\_scale | float | The updated birth scale parameter. |
#### Internal Logic
The method samples the number of mutations using the `mutation_distribution`. For each mutation, it calculates a multiplicative factor by exponentiating the `fitness_base` by a value sampled from the `fitness_distribution`. The birth scale parameter is then multiplied by the product of these factors.
#### Error Handling
The method raises a `TreeSimulatorError` if a negative number of mutations is sampled.
### `populate_tree_from_simulation`
#### Description
This method populates the simulated tree with metadata, including branch lengths, node times, and character states.
#### Inputs
| Name | Type | Description |
| :-------------- | :--------- | :---------------------------------------------------- |
| tree | nx.DiGraph | The NetworkX DiGraph representing the simulated tree. |
| observed\_nodes | List\[str] | A list of observed nodes. |
#### Outputs
| Name | Type | Description |
| :--------------- | :------------- | :--------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | A `CassiopeiaTree` object representing the populated tree. |
#### Internal Logic
The method creates a `CassiopeiaTree` object from the simulated tree. It then iterates through the nodes, setting the time, birth scale, and character states for each node. Finally, it prunes dead lineages and collapses unifurcations if specified.
#### Error Handling
The method raises a `TreeSimulatorError` if all lineages died before a stopping condition was met.
## Dependencies
This class depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :------------------------------------------------ |
| networkx | Representing and manipulating the tree structure. |
| numpy | Random number generation and array operations. |
| queue | Maintaining the priority queue of lineages. |
## Error Handling
The `BirthDeathFitnessSimulator` class uses the `TreeSimulatorError` exception to handle various error conditions, such as invalid input parameters, negative waiting times, and all lineages dying before a stopping condition is met.
# BrownianSpatialDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/BrownianSpatialDataSimulator.py
## High-level description
The `BrownianSpatialDataSimulator` class simulates spatial data for a lineage tree using a Brownian motion process. This simulator assigns spatial coordinates to each cell in a `CassiopeiaTree`, mimicking the movement of cells in a spatial context.
## Code Structure
The `BrownianSpatialDataSimulator` class inherits from the `SpatialDataSimulator` class. It primarily implements the `overlay_data` method, which is responsible for assigning spatial coordinates to the nodes of a `CassiopeiaTree` based on a Brownian motion model.
## References
* `cassiopeia.data.CassiopeiaTree`: The class representing the lineage tree.
* `cassiopeia.simulator.SpatialDataSimulator`: The parent class for spatial data simulators.
## Symbols
### `BrownianSpatialDataSimulator`
#### Description
This class simulates spatial data for a lineage tree using a Brownian motion process. It takes the dimensionality of the space, a diffusion coefficient, and optional scaling and random seed parameters as input.
#### Inputs
| Name | Type | Description |
| :--------------------- | :------------- | :--------------------------------------------------------------------------- |
| dim | int | The number of spatial dimensions. |
| diffusion\_coefficient | float | The diffusion coefficient for the Brownian motion process. |
| scale\_unit\_area | bool | Whether to scale the spatial coordinates to a unit area. Defaults to `True`. |
| random\_seed | Optional\[int] | A random seed for reproducibility. Defaults to `None`. |
#### Outputs
This class does not directly return any output. It modifies the input `CassiopeiaTree` object in-place.
#### Internal Logic
1. **Initialization:** The constructor initializes the simulator with the specified parameters and performs validation checks.
2. **`overlay_data` Method:**
* Traverses the tree from root to leaves.
* Assigns the root cell to the origin (0, 0, ...) in the spatial space.
* For each edge (parent-child relationship) in the tree:
* Calculates the branch length (time since last cell division).
* Samples a random displacement vector from a normal distribution with mean 0 and variance proportional to the branch length and diffusion coefficient.
* Adds the displacement vector to the parent's coordinates to obtain the child's coordinates.
* Optionally scales the spatial coordinates to a unit area.
* Sets the spatial coordinates as a node attribute and in the `cell_meta` DataFrame of the `CassiopeiaTree`.
## Side Effects
* Modifies the input `CassiopeiaTree` object in-place by adding spatial coordinates to the nodes and cell metadata.
## Dependencies
* `numpy`: Used for numerical operations and random number generation.
* `pandas`: Used for handling the `cell_meta` DataFrame.
### Configuration
| Option | Type | Default | Description |
| :---------------- | :--- | :------ | :------------------------------------------------------- |
| scale\_unit\_area | bool | True | Whether to scale the spatial coordinates to a unit area. |
## Error Handling
Raises `DataSimulatorError` if:
* `dim` is less than or equal to zero.
* `diffusion_coefficient` is negative.
# Cas9LineageTracingDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/Cas9LineageTracingDataSimulator.py
## High-level description
The `Cas9LineageTracingDataSimulator` class simulates Cas9-based lineage tracing data by overlaying mutations onto a CassiopeiaTree. It models Cas9 cutting as an exponential process and incorporates features like cassettes, state distributions, and silencing rates to mimic real-world lineage tracing experiments.
## Code Structure
The `Cas9LineageTracingDataSimulator` class inherits from the `LineageTracingDataSimulator` class. It defines several parameters in its constructor to control the simulation process, such as the number of cassettes, size of each cassette, mutation rate, state distribution, and silencing rates. The main functionality is implemented in the `overlay_data` method, which modifies the character states of the input CassiopeiaTree in place.
## References
* `cassiopeia.data.CassiopeiaTree`: The class representing the tree structure on which the data is overlaid.
* `cassiopeia.simulator.LineageTracingDataSimulator`: The parent class defining the interface for lineage tracing data simulators.
## Symbols
### `Cas9LineageTracingDataSimulator`
#### Description
This class simulates Cas9-based lineage tracing data by overlaying mutations onto a CassiopeiaTree. It models Cas9 cutting as an exponential process and incorporates features like cassettes, state distributions, and silencing rates to mimic real-world lineage tracing experiments.
#### Inputs
| Name | Type | Description |
| :------------------------------- | :--------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| number\_of\_cassettes | int | Number of cassettes (arrays of target sites). Defaults to 10. |
| size\_of\_cassette | int | Number of editable target sites per cassette. Defaults to 3. |
| mutation\_rate | Union\[float, List\[float]] | Exponential parameter for the Cas9 cutting rate. Can be a float (all sites mutate at the same rate), a list of floats of length `size_of_cassette` (each site mutates at the specified rate across all cassettes), or a list of floats of length `number_of_cassettes * size_of_cassette` (each site and cassette mutates at the specified rate). Defaults to 0.01. |
| state\_generating\_distribution | Callable\[\[], float] | Distribution from which to simulate state likelihoods. This is only used if `state_priors` is not specified. Defaults to an exponential distribution with parameter 1e-5. |
| number\_of\_states | int | Number of states to simulate. This is only used if `state_priors` is not specified. Defaults to 100. |
| state\_priors | Optional\[Dict\[int, float]] | An optional dictionary mapping states to their prior probabilities. Can also be a list of dictionaries of length `size_of_cassette` or `number_of_cassettes * size_of_cassette` to specify per-site or per-site-per-cassette priors. If this argument is None, states will be generated using the `state_generating_distribution`. Defaults to None. |
| heritable\_silencing\_rate | float | Silencing rate for the cassettes, per node, simulating heritable missing data events. Defaults to 1e-4. |
| stochastic\_silencing\_rate | float | Rate at which to randomly drop out cassettes, to simulate dropout due to low sensitivity of assays. Defaults to 1e-2. |
| heritable\_missing\_data\_state | int | Integer representing data that has gone missing due to a heritable event (i.e. Cas9 resection or heritable silencing). Defaults to -1. |
| stochastic\_missing\_data\_state | int | Integer representing data that has gone missing due to the stochastic dropout from single-cell assay sensitivity. Defaults to -1. |
| random\_seed | Optional\[int] | Numpy random seed to use for deterministic simulations. Note that the numpy random seed gets set during every call to `overlay_data`, thereby producing deterministic simulations every time this function is called. Defaults to None. |
| collapse\_sites\_on\_cassette | bool | Whether or not to collapse cuts that occur in the same cassette in a single iteration. This option only takes effect when `size_of_cassette` is greater than 1. Defaults to True. |
#### Outputs
None
#### Internal Logic
The constructor initializes the simulator with the provided parameters and performs several checks to ensure the validity of the input. It also pre-computes the mutation rate and state priors for each character in the lineage.
### `overlay_data`
#### Description
This method overlays Cas9-based lineage tracing data onto the provided CassiopeiaTree. It simulates mutations, silencing events, and cassette dropouts based on the parameters specified in the constructor.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :----------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree on which to overlay the data. |
#### Outputs
None
#### Internal Logic
1. **Initialize character states:** Sets all character states to 0 (unmutated) for the root node and -1 (missing) for all other nodes.
2. **Simulate mutations:** For each non-root node, calculate the mutation probability for each open site based on the node's lifetime and the site's mutation rate. Randomly introduce mutations based on these probabilities.
3. **Collapse sites on cassettes:** If `collapse_sites_on_cassette` is True and `size_of_cassette` is greater than 1, collapse cuts that occur within the same cassette in a single iteration.
4. **Introduce states at cut sites:** For each cut site, randomly select a state from the state distribution and assign it to the site.
5. **Silence cassettes:** Randomly silence entire cassettes based on the heritable silencing rate.
6. **Apply stochastic silencing:** Randomly drop out cassettes at the leaves based on the stochastic silencing rate.
7. **Set character states:** Update the character states of all nodes in the tree based on the simulated mutations and silencing events.
### `collapse_sites`
#### Description
This method collapses cassettes by identifying cuts that occur within the same cassette and collapsing the sites between them.
#### Inputs
| Name | Type | Description |
| :--------------- | :--------- | :--------------------------------------------------- |
| character\_array | List\[int] | The character array being modified. |
| cuts | List\[int] | The sites in the character array that are being cut. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :--------- | :---------------------------------------------------- |
| updated\_character\_array | List\[int] | The updated character array with collapsed cassettes. |
| cuts\_remaining | List\[int] | The sites that are not part of a cassette collapse. |
#### Internal Logic
1. **Identify cassettes with multiple cuts:** Group the cuts by the cassette they belong to.
2. **Collapse sites within cassettes:** For each cassette with multiple cuts, identify the leftmost and rightmost cut sites and set all sites between them to the heritable missing data state.
3. **Return updated character array and remaining cuts:** Return the updated character array and the list of cuts that were not part of a cassette collapse.
### `introduce_states`
#### Description
This method introduces new states at the specified cut sites in the character array.
#### Inputs
| Name | Type | Description |
| :--------------- | :--------- | :---------------------------------- |
| character\_array | List\[int] | The character array being modified. |
| cuts | List\[int] | The loci being cut. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :--------- | :------------------------------------------------------------ |
| updated\_character\_array | List\[int] | The updated character array with new states at the cut sites. |
#### Internal Logic
1. **Iterate over cut sites:** For each cut site, randomly select a state from the state distribution based on the site's prior probabilities.
2. **Assign new states:** Assign the selected state to the corresponding site in the character array.
3. **Return updated character array:** Return the updated character array.
### `silence_cassettes`
#### Description
This method randomly silences cassettes based on the specified silencing rate.
#### Inputs
| Name | Type | Description |
| :--------------- | :--------- | :---------------------------------------------------------- |
| character\_array | List\[int] | The character array being modified. |
| silencing\_rate | float | The silencing rate. |
| missing\_state | int | The state to use for encoding missing data. Defaults to -1. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :--------- | :--------------------------------------------------- |
| updated\_character\_array | List\[int] | The updated character array with silenced cassettes. |
#### Internal Logic
1. **Iterate over cassettes:** For each cassette, randomly decide whether to silence it based on the silencing rate.
2. **Silence cassettes:** If a cassette is selected for silencing, set all sites within that cassette to the missing data state.
3. **Return updated character array:** Return the updated character array.
### `get_cassettes`
#### Description
This method returns a list of indices corresponding to the start positions of the cassettes in the character array.
#### Inputs
None
#### Outputs
| Name | Type | Description |
| :-------- | :--------- | :------------------------------------------------------------------------- |
| cassettes | List\[int] | An array of indices corresponding to the start positions of the cassettes. |
#### Internal Logic
Calculates the start positions of each cassette based on the `size_of_cassette` and `number_of_cassettes` parameters.
## Side Effects
The `overlay_data` method modifies the character states of the input CassiopeiaTree in place.
## Dependencies
* `numpy`: Used for random number generation and array operations.
* `pandas`: Used for representing the character matrix.
## Error Handling
The constructor raises `DataSimulatorError` if any of the input parameters are invalid.
# ClonalSpatialDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/ClonalSpatialDataSimulator.py
## High-level description
The `ClonalSpatialDataSimulator` class simulates spatial data with a clonal spatial autocorrelation constraint, meaning that cells within the same subclone are more likely to be located near each other. It achieves this by first randomly sampling points in a defined space and then assigning these points to cells in a given `CassiopeiaTree` in a way that enforces spatial proximity within subclones.
## Code Structure
The `ClonalSpatialDataSimulator` class inherits from the `SpatialDataSimulator` class. It primarily utilizes the `__init__`, `sample_points`, and `overlay_data` methods to simulate and assign spatial coordinates to cells in a tree. The `__triangulation_graph`, `__nearest_neighbors_graph`, `__points_to_graph`, and `__split_graph` methods are helper functions used within `overlay_data` to partition points based on the tree structure.
## References
This class references the `CassiopeiaTree` class for tree manipulation and data storage. It also uses several external libraries like `networkx`, `numpy`, `scipy`, `cv2`, `poisson_disc`, and `sklearn.neighbors`.
## Symbols
### `ClonalSpatialDataSimulator`
#### Description
This class simulates spatial data with a clonal spatial autocorrelation constraint. It assigns spatial coordinates to cells in a `CassiopeiaTree` such that cells within the same subclone are more likely to be spatially clustered.
#### Inputs
| Name | Type | Description |
| :----------- | :-------------------------- | :----------------------------------------------------------------------------------- |
| shape | Optional\[Tuple\[int, ...]] | Shape of the space to place cells on. Defaults to None. |
| space | Optional\[np.ndarray] | Numpy array mask representing the space where cells can be placed. Defaults to None. |
| random\_seed | Optional\[int] | A seed for reproducibility. Defaults to None. |
#### Outputs
This class doesn't directly return any values. It modifies the input `CassiopeiaTree` object in-place.
#### Internal Logic
1. **Initialization (`__init__`)**: The constructor initializes the simulator with either a `shape` or a `space` parameter. If `shape` is provided, it generates a space (e.g., an ellipse in 2D) within a grid of that shape. If `space` is provided, it uses the given space directly.
2. **Point Sampling (`sample_points`)**: This method samples points within the defined space using Poisson-Disc sampling, ensuring an approximately even distribution of points.
3. **Data Overlay (`overlay_data`)**: This method assigns the sampled points to cells in the `CassiopeiaTree`. It starts by assigning all points to the root node. Then, it traverses the tree from root to leaves, splitting the points assigned to each parent node among its children. The splitting process considers spatial proximity, assigning each point to the spatially closest child node. This process ensures that cells within the same subclone are more likely to be located near each other.
#### Side Effects
The `overlay_data` method modifies the input `CassiopeiaTree` object in-place by adding spatial coordinates as node attributes and cell metadata.
## Dependencies
| Dependency | Purpose |
| :---------------- | :--------------------------------------------------------- |
| networkx | Graph manipulation and algorithms. |
| numpy | Numerical operations and array manipulation. |
| scipy | Scientific computing, including spatial algorithms. |
| cv2 | Image processing (used for generating elliptical spaces). |
| poisson\_disc | Poisson-Disc sampling for generating evenly spaced points. |
| sklearn.neighbors | Nearest neighbor search for spatial point assignment. |
## Error Handling
The class raises `DataSimulatorError` for various invalid input scenarios, such as providing both `shape` and `space` or if required dependencies are not installed.
# CompleteBinarySimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/CompleteBinarySimulator.py
## High-level description
The `CompleteBinarySimulator` class simulates a complete binary tree, where each node has exactly two children except for the leaves. It provides a way to generate a balanced tree topology with specified depth or number of cells, which can be used for in silico simulations of single-cell phylogenies.
## Code Structure
The `CompleteBinarySimulator` class inherits from the `TreeSimulator` abstract class and implements the `simulate_tree` method. It uses the `networkx` library to generate a balanced binary tree and then converts it into a `CassiopeiaTree` object.
## References
This code references the following symbols:
* `TreeSimulator`: The abstract base class for tree simulators.
* `CassiopeiaTree`: The data structure representing a tree in Cassiopeia.
* `TreeSimulatorError`: An exception class for tree simulator errors.
* `nx.balanced_tree`: A function from the `networkx` library to generate a balanced binary tree.
## Symbols
### `CompleteBinarySimulator`
#### Description
This class simulates a complete binary tree with a specified depth or number of cells. It uses the `networkx` library to generate a balanced binary tree and then converts it into a `CassiopeiaTree` object. The branch lengths are normalized by the height of the tree, resulting in a tree with a height of 1.
#### Inputs
| Name | Type | Description |
| :--------- | :------------- | :-------------------------------------------------- |
| num\_cells | Optional\[int] | Number of cells to simulate (must be a power of 2). |
| depth | Optional\[int] | Depth of the tree. |
#### Outputs
This class doesn't directly return any output. It simulates a tree and stores it internally.
#### Internal Logic
1. **Initialization:**
* Checks if exactly one of `num_cells` or `depth` is provided.
* If `num_cells` is provided, calculates the `depth` and validates that `num_cells` is a power of 2.
* If `depth` is provided, validates that it is greater than 0.
2. **Tree Simulation (`simulate_tree` method):**
* Generates a balanced binary tree using `nx.balanced_tree` with a branching factor of 2.
* Creates a mapping from the default integer node names to unique string names.
* Adds a root node ("root") and connects it to the root of the generated binary tree.
* Relabels the nodes using the generated mapping.
* Creates a `CassiopeiaTree` object from the generated tree.
* Initializes branch lengths by dividing the time of each node by the total depth of the tree plus 1.
#### Side Effects
* Creates and stores a `CassiopeiaTree` object internally.
## Dependencies
| Dependency | Purpose |
| :--------------------------------- | :----------------------------------------------------------- |
| networkx | Used to generate the balanced binary tree. |
| numpy | Used for logarithmic calculations. |
| cassiopeia.data.CassiopeiaTree | Used to create the `CassiopeiaTree` object. |
| cassiopeia.mixins | Used to raise `TreeSimulatorError` in case of invalid input. |
| cassiopeia.simulator.TreeSimulator | The abstract base class for tree simulators. |
## Error Handling
* Raises `TreeSimulatorError` if:
* Neither or both `num_cells` and `depth` are provided.
* `num_cells` is not a power of 2.
* `depth` is not greater than 0.
# DataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/DataSimulator.py
## High-level description
The `DataSimulator` class in `cassiopeia/simulator/DataSimulator.py` is an abstract base class for simulating various types of data to overlay onto a `CassiopeiaTree`. This class provides a blueprint for creating simulators that can generate data like lineage tracing information, transcriptomes, or proteomes, allowing for in silico exploration of different data types and their impact on phylogenetic reconstruction.
## Code Structure
This code defines a single abstract class `DataSimulator` which serves as a template for concrete data simulator implementations. It doesn't have complex interconnections between symbols.
## Symbols
### Symbol Name: `DataSimulator`
#### Description
This abstract class defines the blueprint for all data simulators. It mandates that any derived class must implement the `overlay_data` method.
#### Inputs
This class doesn't take any input during initialization.
#### Outputs
This class doesn't return any output during initialization.
#### Internal Logic
This class doesn't have any internal logic as it's an abstract class.
### Symbol Name: `overlay_data`
#### Description
This abstract method defines the core functionality of a `DataSimulator`. It's intended to be implemented by derived classes to overlay specific types of data onto a `CassiopeiaTree`.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :------------------------------------------------------------------------------------------------------------------ |
| tree | CassiopeiaTree | The `CassiopeiaTree` object to overlay the simulated data on. The tree is assumed to have its topology initialized. |
#### Outputs
This method doesn't return any value. It modifies the input `CassiopeiaTree` in-place.
#### Internal Logic
This method is abstract and doesn't have an implementation in this base class. Derived classes are responsible for implementing the specific logic for data overlay.
# LeafSubsampler.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/LeafSubsampler.py
## High-level description
The `LeafSubsampler` class in `LeafSubsampler.py` defines an abstract base class for all leaf samplers in the Cassiopeia library. It provides a blueprint for creating derived classes that implement different methods for subsampling the leaves of a CassiopeiaTree.
Leaf subsampling is a common operation in phylogenetic analysis, where a subset of leaves (representing cells or taxa) is selected from a larger tree to perform downstream analyses. This can be useful for reducing computational complexity, focusing on specific lineages, or simulating different sampling scenarios.
## Code Structure
This code defines an abstract base class `LeafSubsampler` with one abstract method `subsample_leaves`. This method is meant to be implemented by concrete leaf subsampler classes that inherit from this base class.
## References
This code references the following symbols from other files:
* `cassiopeia.data.CassiopeiaTree`: This class represents a phylogenetic tree with associated data, such as character states and metadata.
* `cassiopeia.mixins.LeafSubsamplerError`: This exception class is raised for errors specific to leaf subsampling operations.
* `cassiopeia.mixins.LeafSubsamplerWarning`: This warning class is used to issue warnings related to leaf subsampling.
* `abc.ABC` and `abc.abstractmethod`: These are used to define abstract base classes and methods, ensuring that derived classes implement the required functionality.
## Symbols
### `LeafSubsampler`
#### Description
This is an abstract base class for all leaf samplers. It defines the interface that concrete leaf sampler classes must implement.
#### Inputs
This class does not have an `__init__` method and therefore does not take any inputs.
#### Outputs
This class does not directly return any outputs.
#### Internal Logic
This class does not have any internal logic as it is an abstract base class.
## Side Effects
This class does not have any side effects as it is an abstract base class.
# LineageTracingDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/LineageTracingDataSimulator.py
## High-level description
The `LineageTracingDataSimulator` class in `LineageTracingDataSimulator.py` is an abstract base class for simulating lineage tracing data on a `CassiopeiaTree`. It provides a framework for overlaying lineage tracing data onto a tree structure, enabling the exploration of different lineage tracing technologies and their impact on reconstructing cell phylogenies.
## Code Structure
The `LineageTracingDataSimulator` class inherits from the abstract base class `DataSimulator` and defines a single abstract method, `overlay_data`. This method is intended to be implemented by concrete subclasses to perform the actual data overlay based on the specific lineage tracing technology being simulated.
## References
This code references the following symbols:
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.simulator.DataSimulator.DataSimulator`
## Symbols
### `LineageTracingDataSimulator`
#### Description
This is an abstract class that serves as a base for all lineage tracing data simulators. It defines the common interface for simulating lineage tracing data on a `CassiopeiaTree`.
#### Inputs
This class does not have its own constructor and therefore does not accept any direct inputs.
#### Outputs
This class is an abstract class and does not directly produce outputs.
#### Internal Logic
This class does not have any internal logic as it is an abstract class.
## Side Effects
This class does not have any side effects as it is an abstract class.
#### Performance Considerations
This class does not have any performance considerations as it is an abstract class.
***
## Dependencies
This code depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :---------------------------------------------- |
| `abc` | Provides the abstract base class functionality. |
### Configuration
This code does not define any configuration options.
## Error Handling
This code does not implement specific error handling beyond basic exception raising.
## Logging
This code does not implement logging.
## API/Interface Reference
This code does not expose an API.
# SequentialLineageTracingDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/SequentialLineageTracingDataSimulator.py
## High-level description
The `SequentialLineageTracingDataSimulator` class simulates lineage tracing data generated by sequential Cas9-based technologies, such as the DNA Typewriter. It overlays simulated edits onto a `CassiopeiaTree`, mimicking the sequential recording process of these technologies. The simulator considers factors like initiation and continuation rates of Cas9 recording, cassette architecture, state distributions, and silencing rates to generate realistic data.
## Code Structure
The `SequentialLineageTracingDataSimulator` class inherits from the `LineageTracingDataSimulator` class. It primarily implements the `overlay_data` method, which simulates the sequential editing process on the provided `CassiopeiaTree`. The class also includes helper functions like `edit_site` and `silence_cassettes` to manage individual editing events and cassette silencing, respectively.
## References
* `cassiopeia.data.CassiopeiaTree`: The simulator operates on a `CassiopeiaTree` object, modifying its character states to represent the simulated lineage tracing data.
* `cassiopeia.simulator.LineageTracingDataSimulator`: This class inherits from the `LineageTracingDataSimulator` class, providing a framework for simulating lineage tracing data.
## Symbols
### `SequentialLineageTracingDataSimulator`
#### Description
This class simulates sequential Cas9-based lineage tracing data and overlays it onto a `CassiopeiaTree`. It models the sequential editing process on a "DNA tape" or "cassette" where only one site can be edited at a time.
#### Inputs
| Name | Type | Description |
| :------------------------------- | :---------------- | :------------------------------------------------------------------------------------------------- |
| number\_of\_cassettes | int | Number of cassettes in the system. |
| size\_of\_cassette | int | Number of editable target sites per cassette. |
| initiation\_rate | float | Exponential parameter for the Cas9 initiation rate. |
| continuation\_rate | float | Exponential parameter for the Cas9 continuation rate. |
| state\_priors | Dict\[int, float] | Dictionary mapping states to their prior probabilities. |
| heritable\_silencing\_rate | float | Silencing rate for the cassettes, simulating heritable missing data events. |
| stochastic\_silencing\_rate | float | Rate at which to randomly drop out cassettes, simulating dropout due to low sensitivity of assays. |
| heritable\_missing\_data\_state | int | Integer representing data that has gone missing due to a heritable event. |
| stochastic\_missing\_data\_state | int | Integer representing data that has gone missing due to stochastic dropout. |
| random\_seed | Optional\[int] | Numpy random seed for deterministic simulations. |
#### Outputs
This class doesn't directly return any output. It modifies the provided `CassiopeiaTree` in place.
#### Internal Logic
The simulator initializes a character matrix representing the cassettes and their states. It then iterates through each node in the tree, simulating the editing process based on the node's lineage and lifetime. For each cassette, it determines if it's initiated and simulates edits based on the continuation rate. It also applies heritable and stochastic silencing to the cassettes, mimicking real-world data variability. Finally, it updates the `CassiopeiaTree` with the simulated character matrix.
### `overlay_data`
#### Description
This method overlays the simulated Cas9-based lineage tracing data onto the provided `CassiopeiaTree`.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :-------------------------------------------------------------- |
| tree | CassiopeiaTree | The `CassiopeiaTree` object to overlay the simulated data onto. |
#### Outputs
This method doesn't return any output. It modifies the input `CassiopeiaTree` in place.
#### Internal Logic
The method first initializes a character matrix with all sites set to an unedited state. It then traverses the tree in depth-first order. For each node, it simulates the Cas9 editing process based on the node's lifetime and the parent's character state. It then applies heritable and stochastic silencing to the character array. Finally, it updates the `CassiopeiaTree` with the simulated character matrix.
### `edit_site`
#### Description
This helper function edits a specific site in the character array based on the provided state priors.
#### Inputs
| Name | Type | Description |
| :--------------- | :---------------- | :------------------------------------------------------ |
| character\_array | List\[int] | The character array representing the cassette states. |
| site | int | The index of the site to edit. |
| state\_priors | Dict\[int, float] | Dictionary mapping states to their prior probabilities. |
#### Outputs
| Name | Type | Description |
| :--------------- | :--------- | :------------------------------------------------ |
| character\_array | List\[int] | The updated character array with the edited site. |
#### Internal Logic
The function randomly samples a state from the state priors based on their probabilities and updates the character array at the specified site with the chosen state.
### `silence_cassettes`
#### Description
This helper function simulates the silencing of cassettes in the character array based on the provided silencing rate.
#### Inputs
| Name | Type | Description |
| :--------------- | :--------- | :---------------------------------------------------- |
| character\_array | List\[int] | The character array representing the cassette states. |
| silencing\_rate | float | The probability of silencing a cassette. |
| missing\_state | int | The state to use for representing silenced cassettes. |
#### Outputs
| Name | Type | Description |
| :------------------------ | :--------- | :--------------------------------------------------- |
| updated\_character\_array | List\[int] | The updated character array with silenced cassettes. |
#### Internal Logic
The function iterates through each cassette and, based on the silencing rate, randomly determines whether to silence it. If a cassette is silenced, all its sites in the character array are set to the specified missing state.
# SimpleFitSubcloneSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/SimpleFitSubcloneSimulator.py
## High-level description
The `SimpleFitSubcloneSimulator` class simulates the growth of a clonal population where a single subclone develops a fitness advantage after a specified number of generations. This simulator is useful for generating both deterministic and stochastic phylogenies that exhibit non-neutral evolution due to the presence of a fitter subclone.
## Code Structure
The `SimpleFitSubcloneSimulator` class inherits from the `TreeSimulator` class. It overrides the `simulate_tree` method to implement its specific simulation logic. The class uses a queue to manage the growth of the tree, adding new nodes and edges based on the specified branch lengths and the time of the experiment.
## References
This code references the following symbols:
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.simulator.TreeSimulator`
## Symbols
### `SimpleFitSubcloneSimulator`
#### Description
This class simulates a clonal population with a single fit subclone emerging after a specified number of generations. It allows for both deterministic and stochastic simulations by accepting either constant values or callables for branch lengths.
#### Inputs
| Name | Type | Description |
| :-------------------------------- | :----------------------------------- | :-------------------------------------------------------------------------------------------------- |
| branch\_length\_neutral | Union\[float, Callable\[\[], float]] | Branch length of neutrally evolving individuals. Can be a constant or a callable returning a float. |
| branch\_length\_fit | Union\[float, Callable\[\[], float]] | Branch length of the fit subclone. Can be a constant or a callable returning a float. |
| experiment\_duration | float | Total duration of the simulation. |
| generations\_until\_fit\_subclone | int | Generation at which the fit subclone emerges. |
#### Outputs
This class doesn't directly return any output. It simulates a tree and stores it internally.
#### Internal Logic
1. Initializes the branch length callables for neutral and fit individuals.
2. Creates a directed graph to represent the tree.
3. Initializes a queue to manage the growth of the tree, starting with the root node.
4. Iteratively processes nodes from the queue:
* Determines the time until the next division based on the node's fitness.
* If the time exceeds the experiment duration, the node becomes a leaf.
* Otherwise, creates two children nodes, assigns their fitness based on the current generation and the `generations_until_fit_subclone` parameter, and adds them to the queue.
5. Constructs a `CassiopeiaTree` object from the simulated tree graph and returns it.
## Side Effects
The `simulate_tree` method modifies the internal state of the `SimpleFitSubcloneSimulator` object by creating and populating the tree graph.
## Dependencies
| Dependency | Purpose |
| :------------------- | :------------------------------------------------- |
| networkx | Used for creating and manipulating the tree graph. |
| queue | Used for managing the growth of the tree. |
| typing | Used for type hinting. |
| cassiopeia.data | Used for the `CassiopeiaTree` class. |
| cassiopeia.simulator | Used for the `TreeSimulator` class. |
```python theme={null}
class SimpleFitSubcloneSimulator(TreeSimulator):
...
```
### `_create_callable`
#### Description
This private method converts a constant branch length value or a callable into a callable. This allows the simulator to handle both deterministic and stochastic branch lengths.
#### Inputs
| Name | Type | Description |
| :--- | :----------------------------------- | :------------------------------- |
| x | Union\[float, Callable\[\[], float]] | Branch length value or callable. |
#### Outputs
| Name | Type | Description |
| :-------------------------------------- | :-------------------- | :----------------------------------------- |
| constant\_branch\_length\_callable or x | Callable\[\[], float] | A callable that returns the branch length. |
#### Internal Logic
1. If `x` is an integer or float, it defines a new callable that always returns `x`.
2. Otherwise, it returns `x` directly, assuming it's already a callable.
```python theme={null}
def _create_callable(
self, x: Union[float, Callable[[], float]]
) -> Callable[[], float]:
# In case the user provides an int, we still hold their back...
if type(x) in [int, float]:
def constant_branch_length_callable() -> float:
return x
return constant_branch_length_callable
else:
return x
```
### `simulate_tree`
#### Description
This method simulates the growth of a clonal population with a single fit subclone and returns the resulting tree as a `CassiopeiaTree` object.
#### Inputs
This method doesn't take any explicit inputs. It uses the parameters provided during the initialization of the `SimpleFitSubcloneSimulator` object.
#### Outputs
| Name | Type | Description |
| :--- | :------------- | :------------------ |
| res | CassiopeiaTree | The simulated tree. |
#### Internal Logic
Refer to the "Internal Logic" section of the `SimpleFitSubcloneSimulator` class description for details on the simulation process.
```python theme={null}
def simulate_tree(self) -> CassiopeiaTree:
...
```
# SpatialDataSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/SpatialDataSimulator.py
## High-level description
The `SpatialDataSimulator` class in `SpatialDataSimulator.py` defines an abstract base class for simulating spatial data on a `CassiopeiaTree`. It provides a blueprint for overlaying spatial coordinates onto the tree, mimicking the spatial organization of cells in a tissue. This abstract class is intended to be subclassed for specific spatial simulation models.
## Code Structure
This code defines a single abstract class `SpatialDataSimulator` which inherits from the `DataSimulator` class. The `SpatialDataSimulator` class doesn't directly interact with other symbols in the code, but it defines an abstract method `overlay_data` that needs to be implemented by its subclasses.
## References
This code references the following code symbols:
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.simulator.DataSimulator.DataSimulator`
## Symbols
### `SpatialDataSimulator`
#### Description
This abstract class serves as a template for simulating spatial data on a `CassiopeiaTree`. It defines the structure for overlaying spatial coordinates onto the tree nodes, representing the spatial arrangement of cells. Subclasses of `SpatialDataSimulator` implement the `overlay_data` method to apply specific spatial models.
#### Inputs
This class doesn't have an `__init__` method and therefore doesn't take any direct inputs.
#### Outputs
This class doesn't directly return any outputs.
#### Internal Logic
This class doesn't have any internal logic as it's an abstract class.
## Side Effects
This class doesn't have any side effects as it's an abstract class.
## Dependencies
This code depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :---------------------------------------------- |
| abc | Provides the abstract base class functionality. |
### `overlay_data`
#### Description
This abstract method defines the interface for overlaying spatial data onto a `CassiopeiaTree`. Subclasses must implement this method to apply specific spatial simulation models.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :-------------------------------------------------------- |
| tree | CassiopeiaTree | The `CassiopeiaTree` object to overlay spatial data onto. |
#### Outputs
This method doesn't return any value, it modifies the `CassiopeiaTree` object in-place.
#### Internal Logic
This method is abstract and doesn't have an implementation in this class. Subclasses are responsible for providing the implementation based on the specific spatial model.
# SpatialLeafSubsampler.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/SpatialLeafSubsampler.py
## High-level description
The `SpatialLeafSubsampler` class is a subclass of `LeafSubsampler` that subsamples leaves within a specified region of interest in a `CassiopeiaTree` with spatial information. It supports subsampling using a bounding box or a numpy mask, and allows for downsampling by ratio or a fixed number of leaves.
## Code Structure
The `SpatialLeafSubsampler` class has a single method, `subsample_leaves`, which takes a `CassiopeiaTree` as input and returns a new `CassiopeiaTree` containing the subsampled leaves. The class constructor initializes the region of interest and downsampling parameters.
## References
* `CassiopeiaTree`: The `SpatialLeafSubsampler` operates on `CassiopeiaTree` objects.
* `LeafSubsampler`: The `SpatialLeafSubsampler` is a subclass of `LeafSubsampler`.
## Symbols
### `SpatialLeafSubsampler`
#### Description
This class subsamples leaves within a region of interest of a `CassiopeiaTree` with spatial information and produces a new `CassiopeiaTree` containing the sampled leaves.
#### Inputs
| Name | Type | Description |
| :----------------- | :---------------------- | :-------------------------------------------------------------------------------------------------------------------------- |
| bounding\_box | Optional\[List\[tuple]] | A list of tuples of the form (min, max) for each dimension of the bounding box. |
| space | Optional\[np.ndarray] | Numpy array mask representing the space that cells will be sampled from. |
| ratio | Optional\[float] | Specifies the number of leaves to be sampled as a ratio of the total number of leaves in the region of interest. |
| number\_of\_leaves | Optional\[int] | Explicitly specifies the number of leaves to be sampled. |
| attribute\_key | Optional\[str] | The key in the CassiopeiaTree's node attributes that contains the spatial coordinates of the leaves. Defaults to "spatial". |
| merge\_cells | Optional\[bool] | Whether or not to merge cells contained within the same pixel in the space. Defaults to False. |
#### Outputs
None
#### Internal Logic
The constructor checks for valid input parameters and stores them as instance variables.
### `subsample_leaves`
#### Description
This method subsamples the leaves of a `CassiopeiaTree` to those within a region of interest and returns a tree pruned to contain lineages relevant to only leaves in the sample (the "induced subtree" on the sample).
#### Inputs
| Name | Type | Description |
| :------------------------- | :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The `CassiopeiaTree` for which to subsample leaves. |
| keep\_singular\_root\_edge | Optional\[bool] | Whether or not to collapse the single edge leading from the root in the subsample, if it exists. Defaults to True. |
| collapse\_duplicates | bool | Whether or not to collapse duplicated character states, so that only unique character states are present in each ambiguous state. Defaults to True. |
#### Outputs
| Name | Type | Description |
| :--------------- | :------------- | :---------------------------------------------------------------------------------------------- |
| subsampled\_tree | CassiopeiaTree | A new `CassiopeiaTree` that is the induced subtree on a sample of the leaves in the given tree. |
#### Internal Logic
1. **Check for spatial information:** Ensures the tree has spatial information associated with the leaves.
2. **Subset leaves:**
* If `bounding_box` is provided, keeps leaves within the specified bounding box.
* If `space` is provided, keeps leaves within the specified mask.
3. **Determine number of leaves to sample:**
* If `ratio` is provided, calculates the number of leaves to sample based on the ratio and the number of leaves in the region of interest.
* If `number_of_leaves` is provided, uses the specified number.
* If neither is provided, keeps all leaves in the region of interest.
4. **Sample leaves:** Randomly samples the specified number of leaves from the subset of leaves within the region of interest.
5. **Prune lineages:** Removes leaves not in the sample and prunes lineages no longer relevant to the sampled leaves.
6. **Merge cells (optional):** If `merge_cells` is True, merges cells within the same pixel in the space, creating a new cell with ambiguous character states.
7. **Collapse unifurcations:** Collapses unifurcations in the tree, preserving node times.
8. **Copy and annotate branch lengths and times:** Copies branch lengths and times from the original tree to the subsampled tree.
## Side Effects
The `subsample_leaves` method modifies the input `CassiopeiaTree` object in place if `copy` is set to False.
## Error Handling
The constructor and `subsample_leaves` method raise `LeafSubsamplerError` for invalid input parameters or if the tree does not have the required spatial information.
## Dependencies
* `numpy`: Used for array operations and random sampling.
* `pandas`: Used for character matrix manipulation.
* `networkx`: Used for tree manipulation.
* `collections`: Used for creating a defaultdict.
* `copy`: Used for deep copying the tree.
* `warnings`: Used for issuing warnings.
## Configuration
The `SpatialLeafSubsampler` class is configured using the parameters passed to its constructor.
## Logging
The code does not implement any specific logging mechanisms.
## API/Interface Reference
The `SpatialLeafSubsampler` class exposes a single public method, `subsample_leaves`, which can be used to subsample the leaves of a `CassiopeiaTree` object.
# SupercellularSampler.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/SupercellularSampler.py
## High-level description
The `SupercellularSampler` class is a subclass of `LeafSubsampler` that simulates the merging of cells in a lineage tree, creating "supercellular" states. This is achieved by iteratively merging pairs of leaves, chosen with probability inversely proportional to their branch distance, resulting in a tree with ambiguous character states.
## Code Structure
The `SupercellularSampler` class has a single public method, `subsample_leaves`, which takes a `CassiopeiaTree` as input and returns a new `CassiopeiaTree` with merged leaves. The merging process is controlled by either the `ratio` or `number_of_merges` parameter passed to the constructor.
## References
This class references the following code symbols:
* `CassiopeiaTree`
* `LeafSubsampler`
* `LeafSubsamplerError`
## Symbols
### SupercellularSampler
#### Description
This class implements a leaf subsampling strategy where leaves are merged iteratively to create a tree with ambiguous character states, simulating the merging of cells.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :--------------------------------------------------------------------- |
| ratio | Optional\[float] | The number of times to merge as a ratio of the total number of leaves. |
| number\_of\_merges | Optional\[float] | Explicit number of merges to perform. |
#### Outputs
This class does not directly return any output. It modifies the input `CassiopeiaTree` object.
#### Internal Logic
The `subsample_leaves` method performs the following steps:
1. **Determine the number of merges:** Based on the provided `ratio` or `number_of_merges`, the number of merge operations is calculated.
2. **Iterative merging:** For each merge operation:
* Randomly select a leaf node (`leaf1`).
* Calculate the branch distances from `leaf1` to all other leaves.
* Randomly select a second leaf node (`leaf2`) with probability inversely proportional to its distance from `leaf1`.
* Merge `leaf1` and `leaf2` into a new leaf node with a combined name and ambiguous character states.
* Connect the new leaf node to the LCA of the original leaves.
* Set the time of the new leaf node to the mean time of the original leaves.
3. **Tree cleanup:**
* Collapse unifurcations in the tree.
* Optionally collapse duplicated character states in ambiguous nodes.
4. **Return the modified tree:** The modified `CassiopeiaTree` object is returned.
## Side Effects
The `subsample_leaves` method modifies the input `CassiopeiaTree` object in-place.
## Error Handling
The class raises `LeafSubsamplerError` if:
* Both `ratio` and `number_of_merges` are not specified or both are specified.
* The number of merges exceeds the number of leaves in the tree.
* No merges are to be performed.
* The tree does not have a character matrix defined.
## Dependencies
This class depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :--------------------------------------------- |
| numpy | Used for random sampling and array operations. |
| pandas | Used for handling the character matrix. |
| networkx | Used for tree manipulation. |
| copy | Used for deep copying the tree. |
```python theme={null}
import copy
from typing import Optional
import numpy as np
import pandas as pd
import networkx as nx
from cassiopeia.data.CassiopeiaTree import CassiopeiaTree, CassiopeiaTreeError
from cassiopeia.simulator.LeafSubsampler import (
LeafSubsampler,
LeafSubsamplerError,
)
```
# TreeSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/TreeSimulator.py
## High-level description
The `TreeSimulator` class is an abstract base class for all tree simulators in Cassiopeia. It defines the basic interface for simulating single-cell phylogenies, requiring derived classes to implement a `simulate_tree` method that returns a `CassiopeiaTree` object representing the simulated phylogeny. This enables users to generate ground truth phylogenies for evaluating different reconstruction and analysis methods.
## References
This class is referenced in other parts of the codebase, particularly in concrete simulator implementations like `BirthDeathFitnessSimulator`, `CompleteBinarySimulator`, and `SimpleFitSubcloneSimulator`. These classes inherit from `TreeSimulator` and provide specific simulation algorithms.
## Symbols
### `TreeSimulator`
#### Description
This is an abstract base class for all tree simulators. It defines the blueprint for simulating single-cell phylogenies.
#### Inputs
This class does not have an **init** method and therefore does not take any inputs.
#### Outputs
This class does not directly return any outputs.
#### Internal Logic
The class uses the `abc` module to define the `simulate_tree` method as an abstract method, ensuring that all concrete subclasses must implement this method.
### `simulate_tree`
#### Description
An abstract method that needs to be implemented by all derived classes. This method should simulate a `CassiopeiaTree` with at least its tree topology initialized.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :--- | :--------------- | :---------------------------------------------------------------------------------------------------------------------- |
| | `CassiopeiaTree` | A `CassiopeiaTree` object representing the simulated phylogeny. The tree should have at least its topology initialized. |
#### Internal Logic
This method is abstract and therefore does not have any internal logic defined in the base class. Concrete subclasses are responsible for implementing the specific simulation logic.
# UniformLeafSubsampler.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/UniformLeafSubsampler.py
## High-level description
The `UniformLeafSubsampler` class is a subclass of `LeafSubsampler` that performs uniform random sampling of leaves in a `CassiopeiaTree`. It creates a new `CassiopeiaTree` containing only the lineages of the sampled leaves, preserving character states, metadata, and dissimilarity maps for the sampled cells.
## References
This code references the following symbols:
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.simulator.LeafSubsampler.LeafSubsampler`
* `cassiopeia.simulator.LeafSubsampler.LeafSubsamplerError`
## Symbols
### `UniformLeafSubsampler`
#### Description
This class implements the logic for uniformly subsampling leaves from a `CassiopeiaTree`. It provides options to specify the sample size either as a ratio of the total number of leaves or as an explicit number.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :------------------------------------ |
| ratio | Optional\[float] | The proportion of leaves to sample. |
| number\_of\_leaves | Optional\[int] | The exact number of leaves to sample. |
#### Outputs
This class doesn't directly return any values. It modifies the input `CassiopeiaTree` object.
#### Internal Logic
The `__init__` method initializes the `UniformLeafSubsampler` object, ensuring that either `ratio` or `number_of_leaves` is provided, but not both.
The `subsample_leaves` method performs the actual subsampling. It first determines the desired sample size based on the provided `ratio` or `number_of_leaves`. Then, it randomly selects leaves to remove and prunes the tree accordingly. Finally, it optionally collapses any remaining unifurcations (nodes with a single child) to maintain a valid tree structure.
#### Side Effects
* Modifies the input `CassiopeiaTree` object in place.
#### Performance Considerations
The performance of this class depends on the size of the input tree and the desired sample size. The random selection of leaves and tree pruning operations have a time complexity that scales with the number of nodes and edges in the tree.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/__init__.py
## High-level description
This file serves as the top-level module for the `cassiopeia.simulator` package, exposing various tree and data simulators for generating synthetic lineage tracing data. It imports and re-exports specific simulator classes, making them readily available for use in other parts of the codebase.
## Code Structure
This file simply imports classes from submodules within the `cassiopeia.simulator` package and defines a list `__all__` to specify which symbols should be exported when using `from cassiopeia.simulator import *`.
## Symbols
### `__all__`
#### Description
A list of strings representing the names of the simulator classes to be exported when using `from cassiopeia.simulator import *`.
#### Internal Logic
The list includes the names of various tree and data simulators, such as `BirthDeathFitnessSimulator`, `BrownianSpatialDataSimulator`, `Cas9LineageTracingDataSimulator`, etc. These classes are responsible for generating synthetic lineage tracing data with different characteristics and underlying models.
# ecDNABirthDeathSimulator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/ecDNABirthDeathSimulator.py
## High-level description
The `ecDNABirthDeathSimulator` class simulates the evolution of a cell population with extrachromosomal DNA (ecDNA) using a forward birth-death process. It extends the `BirthDeathFitnessSimulator` class to incorporate ecDNA-specific dynamics, including co-segregation and fitness effects based on ecDNA copy number. The simulator generates a phylogenetic tree representing the evolutionary history of the cell population, along with metadata about ecDNA copy numbers and observed ecDNA counts in each cell.
## Code Structure
The `ecDNABirthDeathSimulator` class inherits from the `BirthDeathFitnessSimulator` class. It overrides several methods to incorporate ecDNA-specific logic, including `initialize_tree`, `update_fitness`, `sample_lineage_event`, and `get_ecdna_array`. The `populate_tree_from_simulation` method is used to create a `CassiopeiaTree` object from the simulated tree, including metadata about ecDNA copy numbers and observed counts.
## References
* `BirthDeathFitnessSimulator`: The parent class that provides the basic framework for simulating a birth-death process with fitness.
* `CassiopeiaTree`: The data structure used to represent the simulated phylogenetic tree and associated metadata.
## Symbols
### `ecDNABirthDeathSimulator`
#### Description
This class simulates the evolution of a cell population with ecDNA using a forward birth-death process. It allows for various birth and death waiting time distributions, fitness regimes, and ecDNA co-segregation patterns.
#### Inputs
| Name | Type | Description |
| :--------------------------- | :---------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| birth\_waiting\_distribution | Callable\[\[float], float] | A function that samples waiting times from the birth distribution. |
| initial\_birth\_scale | float | The initial scale parameter for the birth distribution. |
| death\_waiting\_distribution | Optional\[Callable\[\[], float]] | A function that samples waiting times from the death distribution. Defaults to no death. |
| mutation\_distribution | Optional\[Callable\[\[], int]] | A function that samples the number of mutations occurring at a division event. Defaults to no mutations. |
| fitness\_distribution | Optional\[Callable\[\[], float]] | A function that samples the exponential that the fitness base is raised by. Determines the distribution of fitness mutation strengths. |
| fitness\_base | float | The base that is raised by the value given by the fitness distribution. Determines the base strength of fitness mutations. Defaults to Euler's constant. |
| num\_extant | Optional\[int] | The number of extant lineages at which to stop the simulation. |
| experiment\_time | Optional\[float] | The time at which to stop the simulation. |
| collapse\_unifurcations | bool | Whether to collapse unifurcations in the resulting tree. Defaults to True. |
| prune\_dead\_lineages | bool | Whether to prune dead lineages from the resulting tree. Defaults to True. |
| random\_seed | int | A seed for reproducibility. |
| initial\_copy\_number | np.array | The initial copy number of each ecDNA species in the root cell. Defaults to \[1]. |
| cosegregation\_coefficient | float | A coefficient describing the likelihood of co-segregation between ecDNA species. Defaults to 0.0. |
| splitting\_function | Callable\[\[int], int] | A function that describes the segregation of ecDNA at cell division. Defaults to a binomial distribution with p=0.5. |
| fitness\_array | np.array | A matrix defining the fitness of a cell based on the presence or absence of each ecDNA species. |
| fitness\_function | Optional\[Callable\[\[int, int, float], float]] | A function that calculates the fitness of a cell based on ecDNA copy numbers and the fitness array. |
| capture\_efficiency | float | The probability of observing an ecDNA species in a cell. Defaults to 1.0. |
| initial\_tree | Optional\[CassiopeiaTree] | A tree used to initialize the simulation. |
#### Outputs
A `CassiopeiaTree` object representing the simulated phylogenetic tree with ecDNA copy number and observation data.
#### Internal Logic
The simulator works by maintaining a priority queue of lineages, sorted by their next event time. At each step, the lineage with the earliest event time is popped from the queue. If the event is a birth, a new lineage is created with updated fitness and ecDNA copy number, and both lineages are added back to the queue. If the event is a death, the lineage is removed from the queue. The simulation continues until one of the stopping conditions is met.
### `initialize_tree`
#### Description
Initializes the tree with a single root node or from a provided initial tree.
#### Inputs
| Name | Type | Description |
| :---- | :-------- | :--------------------------------- |
| names | Generator | A generator for unique node names. |
#### Outputs
An `nx.DiGraph` object representing the initialized tree.
#### Internal Logic
If an initial tree is provided, the method copies its topology and attributes. Otherwise, it creates a new tree with a single root node and sets its initial attributes.
### `update_fitness`
#### Description
Calculates the fitness of a lineage based on its ecDNA copy number.
#### Inputs
| Name | Type | Description |
| :----------- | :------- | :------------------------------------------ |
| ecdna\_array | np.array | The ecDNA copy number array of the lineage. |
#### Outputs
The updated birth scale parameter representing the lineage's fitness.
#### Internal Logic
The method calculates the fitness based on the provided `fitness_array` and `fitness_function`. If no `fitness_function` is provided, it uses a simple additive model based on the presence or absence of each ecDNA species.
### `sample_lineage_event`
#### Description
Simulates the next event (birth or death) for a given lineage.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------------------------- | :------------------------------------- |
| lineage | Dict\[str, Union\[int, float]] | A dictionary representing the lineage. |
| current\_lineages | PriorityQueue | The priority queue of active lineages. |
| tree | nx.DiGraph | The tree being constructed. |
| names | Generator | A generator for unique node names. |
| observed\_nodes | List\[str] | A list of observed nodes. |
#### Outputs
None. The method modifies the `tree`, `current_lineages`, and `observed_nodes` in place.
#### Internal Logic
The method samples birth and death waiting times for the lineage. If the birth time is less than the death time, a new lineage is created with updated fitness and ecDNA copy number, and both lineages are added to the queue. If the death time is less than the birth time, the lineage is marked as inactive and added to the queue. If both times exceed the experiment time, the lineage is cut off at the experiment time and marked as observed.
### `get_ecdna_array`
#### Description
Generates the ecDNA copy number array for a child cell given its parent's copy number and co-segregation parameters.
#### Inputs
| Name | Type | Description |
| :--------- | :--------- | :------------------------------------- |
| parent\_id | str | The ID of the parent node in the tree. |
| tree | nx.DiGraph | The tree being constructed. |
#### Outputs
A `np.array` representing the ecDNA copy number array for the child cell.
#### Internal Logic
The method first doubles the parent's ecDNA copy number array. If the parent already has a child, it subtracts the child's copy number from the doubled array. Otherwise, it uses the `splitting_function` and `cosegregation_coefficient` to simulate the segregation of ecDNA into the child cell.
## Side Effects
The `ecDNABirthDeathSimulator` modifies the `tree`, `current_lineages`, and `observed_nodes` objects in place during the simulation.
## Dependencies
* `networkx`: Used for representing and manipulating the tree topology.
* `numpy`: Used for numerical operations and random number generation.
* `pandas`: Used for storing and manipulating cell metadata.
* `queue`: Used for the priority queue of active lineages.
## Error Handling
The `ecDNABirthDeathSimulator` raises `TreeSimulatorError` exceptions for various invalid input conditions, such as specifying no stopping conditions or a negative number of mutations. It also raises `ecDNABirthDeathSimulatorError` if a child's ecDNA copy number exceeds its parent's.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/simulator/overview
## High-level description
The `cassiopeia/simulator` directory contains a collection of classes and modules for simulating various aspects of single-cell lineage tracing experiments. These simulators can generate synthetic phylogenetic trees, spatial data, lineage tracing data, and perform leaf subsampling. The main components include:
1. Tree simulators (e.g., BirthDeathFitnessSimulator, CompleteBinarySimulator)
2. Data simulators (e.g., Cas9LineageTracingDataSimulator, BrownianSpatialDataSimulator)
3. Leaf subsamplers (e.g., UniformLeafSubsampler, SpatialLeafSubsampler)
4. Specialized simulators (e.g., ecDNABirthDeathSimulator)
These simulators provide a framework for generating synthetic data to test and validate lineage reconstruction algorithms and analysis methods.
## What does it do?
The `cassiopeia/simulator` directory provides tools to:
1. Generate synthetic phylogenetic trees with various growth models and fitness effects.
2. Simulate lineage tracing data, including Cas9-based editing and sequential recording.
3. Create spatial data for cells in a lineage tree.
4. Subsample leaves from a tree to mimic experimental sampling or create supercellular states.
5. Simulate specialized scenarios like extrachromosomal DNA (ecDNA) evolution.
These simulators allow researchers to create realistic in silico datasets that mimic the complexities of real single-cell lineage tracing experiments. By generating ground truth phylogenies and associated data, researchers can evaluate the performance of reconstruction algorithms, test hypotheses about evolutionary processes, and explore the effects of different experimental designs and parameters.
## Entry points
The main entry points for using the simulators are:
1. `TreeSimulator` abstract base class: This is the starting point for implementing new tree simulation models.
2. `DataSimulator` abstract base class: This is the base class for implementing new data simulation models.
3. `LeafSubsampler` abstract base class: This is the base class for implementing new leaf subsampling strategies.
Concrete implementations of these base classes, such as `BirthDeathFitnessSimulator`, `Cas9LineageTracingDataSimulator`, and `UniformLeafSubsampler`, provide specific simulation and subsampling functionalities.
The `__init__.py` file in this directory serves as the main interface for importing and using the various simulator classes.
## Key Files
1. `BirthDeathFitnessSimulator.py`: Implements a birth-death process with fitness variations for tree simulation.
2. `Cas9LineageTracingDataSimulator.py`: Simulates Cas9-based lineage tracing data.
3. `BrownianSpatialDataSimulator.py`: Generates spatial data for cells using a Brownian motion model.
4. `UniformLeafSubsampler.py`: Implements uniform random subsampling of leaves from a tree.
5. `ecDNABirthDeathSimulator.py`: Simulates the evolution of cell populations with extrachromosomal DNA.
These files contain the core implementations of various simulation and subsampling strategies, each addressing different aspects of single-cell lineage tracing experiments.
## Dependencies
The simulator modules rely on several external libraries:
1. `networkx`: Used for graph operations and tree manipulations.
2. `numpy`: Used for numerical computations and random number generation.
3. `pandas`: Used for handling data structures like character matrices.
4. `scipy`: Used for various scientific computing tasks, including spatial algorithms.
5. `sklearn`: Used for nearest neighbor searches in spatial simulations.
Additionally, the simulators depend on other parts of the Cassiopeia package, particularly the `cassiopeia.data.CassiopeiaTree` class for representing and manipulating phylogenetic trees.
## Configuration
Most simulator classes accept various parameters during initialization to configure their behavior. Common configuration options include:
1. Simulation duration or stopping conditions (e.g., `experiment_time`, `num_extant`)
2. Mutation rates and distributions
3. Fitness parameters
4. Spatial simulation parameters (e.g., `diffusion_coefficient`)
5. Subsampling ratios or target numbers of leaves
These parameters allow users to fine-tune the simulations to match specific experimental scenarios or test different hypotheses about evolutionary processes.
In conclusion, the `cassiopeia/simulator` directory provides a comprehensive set of tools for generating synthetic single-cell lineage tracing data, enabling researchers to test and validate analysis methods, explore experimental designs, and gain insights into the underlying biological processes.
# CassiopeiaSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/CassiopeiaSolver.py
## High-level description
The `CassiopeiaSolver` is an abstract base class for all phylogenetic inference algorithms in the Cassiopeia framework. It defines the basic structure and interface for solvers, requiring subclasses to implement the `solve` method, which performs the actual tree inference. It also handles the transformation of prior probabilities into weights used by some solvers.
## Code Structure
The `CassiopeiaSolver` class is an abstract base class. It doesn't have any internal logic or complex interactions between symbols. The main purpose of this class is to define a common interface for all derived solver classes.
## Symbols
### `CassiopeiaSolver`
#### Description
This is an abstract base class for all phylogenetic inference algorithms. It defines the basic structure and interface for solvers.
#### Inputs
| Name | Type | Description |
| :-------------------- | :--- | :---------------------------------------------------------------------------------------------------------------------- |
| prior\_transformation | str | A string specifying the function to use for transforming prior probabilities into weights. Defaults to "negative\_log". |
#### Outputs
This class does not have any outputs.
#### Internal Logic
This class does not have any internal logic.
### `solve`
#### Description
This is an abstract method that needs to be implemented by all derived solver classes. It performs the actual tree inference.
#### Inputs
| Name | Type | Description |
| :---------------------------- | :------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | A `CassiopeiaTree` object storing character information for phylogenetic inference. |
| layer | Optional\[str] | Layer storing the character matrix for solving. If None, the default character matrix is used in the `CassiopeiaTree`. |
| collapse\_mutationless\_edges | bool | Indicates if the final reconstructed tree should collapse mutationless edges based on internal states inferred by Camin-Sokal parsimony. Defaults to False. |
| logfile | str | File location to log output. Defaults to "stdout.log". |
#### Outputs
This method does not have any outputs. It modifies the input `cassiopeia_tree` object by populating its tree attribute.
#### Internal Logic
This method is abstract and does not have any internal logic. It needs to be implemented by derived classes.
# DistanceSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/DistanceSolver.py
## High-level description
The `DistanceSolver` class is a base class for distance-based phylogenetic tree solvers in Cassiopeia. It provides a general framework for solving trees using a dissimilarity map, iteratively joining samples into cherries and updating the map. Subclasses like `NeighborJoiningSolver` and `UPGMASolver` implement specific cherry-finding and map-updating logic.
## Code Structure
The `DistanceSolver` class defines the general structure and common methods for distance-based solvers. It relies on abstract methods like `find_cherry` and `update_dissimilarity_map` to be implemented by subclasses. The `solve` method orchestrates the tree-solving process, while helper methods like `get_dissimilarity_map` and `setup_dissimilarity_map` manage the dissimilarity matrix.
## References
* `CassiopeiaSolver`: The parent class defining the general solver interface.
* `solver_utilities`: Provides utility functions for tree manipulation and data handling.
* `dissimilarity_functions`: Offers various dissimilarity functions for comparing samples.
## Symbols
### `DistanceSolver`
#### Description
This class provides a general framework for distance-based phylogenetic tree solvers. It defines the overall structure and common methods, leaving the specific cherry-finding and dissimilarity map updating logic to be implemented by subclasses.
#### Inputs
| Name | Type | Description |
| :---------------------- | :------------------------------------------------------------------------------------- | :-------------------------------------------------------------------- |
| dissimilarity\_function | Optional\[Callable\[\[np.array, np.array, int, Dict\[int, Dict\[int, float]]], float]] | Function to compute dissimilarity between samples. |
| add\_root | bool | Whether to add an implicit root to the tree (default: False). |
| prior\_transformation | str | Function to transform priors into weights (default: "negative\_log"). |
| threads | int | Number of threads to use for solver (default: 1). |
#### Outputs
None
#### Internal Logic
The `solve` method orchestrates the tree-solving process:
1. **Setup:** Initializes the dissimilarity map and creates a basic tree with all samples as leaves.
2. **Iteration:** Iteratively finds cherries, joins them in the tree, and updates the dissimilarity map until only two nodes remain.
3. **Rooting:** Roots the tree using the specified or implicit root.
4. **Population:** Populates the `CassiopeiaTree` with the solved tree topology.
### `get_dissimilarity_map`
#### Description
Obtains or generates the dissimilarity map used throughout the solver. This method can be overridden by subclasses to use other types of sample comparison maps.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :-------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object to obtain the dissimilarity map from. |
| layer | Optional\[str] | Layer storing the character matrix for solving (default: None). |
#### Outputs
| Name | Type | Description |
| :----------------- | :----------- | :----------------------------------------- |
| dissimilarity\_map | pd.DataFrame | The dissimilarity matrix used for solving. |
### `solve`
#### Description
Solves the phylogenetic tree using the distance-based approach. It iteratively finds cherries, joins them, and updates the dissimilarity map.
#### Inputs
| Name | Type | Description |
| :---------------------------- | :------------- | :------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The `CassiopeiaTree` object to populate with the solved tree. |
| layer | Optional\[str] | Layer storing the character matrix for solving (default: None). |
| collapse\_mutationless\_edges | bool | Whether to collapse mutationless edges in the final tree (default: False). |
| logfile | str | File location to log output (default: "stdout.log"). |
#### Outputs
None
#### Internal Logic
1. **Initialization:** Obtains the dissimilarity map and creates a basic tree with all samples as leaves.
2. **Cherry Finding:** Iteratively calls the subclass-specific `find_cherry` method to identify a pair of samples to join.
3. **Joining and Updating:** Joins the identified cherry in the tree and updates the dissimilarity map using the subclass-specific `update_dissimilarity_map` method.
4. **Rooting:** Roots the tree using the `root_tree` method, which is also implemented by subclasses.
5. **Population:** Populates the input `CassiopeiaTree` with the solved tree topology.
### `_ccphylo_solve`
#### Description
Solves the tree using fast distance-based algorithms implemented by CCPhylo. Requires CCPhylo to be installed and configured.
#### Inputs
| Name | Type | Description |
| :---------------------------- | :------------- | :------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The `CassiopeiaTree` object to populate with the solved tree. |
| layer | Optional\[str] | Layer storing the character matrix for solving (default: None). |
| collapse\_mutationless\_edges | bool | Whether to collapse mutationless edges in the final tree (default: False). |
| logfile | str | File location to log output (default: "stdout.log"). |
| method | str | The CCPhylo algorithm to use (default: "dnj"). |
#### Outputs
None
#### Internal Logic
1. **Preparation:** Saves the dissimilarity map as a phylip file.
2. **CCPhylo Execution:** Runs the specified CCPhylo algorithm using the saved dissimilarity map.
3. **Conversion and Rooting:** Converts the resulting tree to a NetworkX graph and roots it using `root_tree`.
4. **Population:** Populates the input `CassiopeiaTree` with the solved tree topology.
### `_setup_ccphylo`
#### Description
Sets up the CCPhylo solver by retrieving the path from the configuration file and validating it.
#### Inputs
None
#### Outputs
None
#### Internal Logic
1. **Path Retrieval:** Reads the `ccphylo_path` from the configuration file.
2. **Validation:** Checks if the path exists and is executable. Raises a `DistanceSolverError` if not.
### `setup_dissimilarity_map`
#### Description
Sets up the solver by creating the dissimilarity map if needed and setting up the root sample.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :--------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The `CassiopeiaTree` object to set up the dissimilarity map for. |
| layer | Optional\[str] | Layer storing the character matrix for solving (default: None). |
#### Outputs
None
#### Internal Logic
1. **Root Handling:** If no root sample is specified, either sets up an implicit root or raises a `DistanceSolverError`.
2. **Dissimilarity Map Creation:** If the dissimilarity map is not present, computes it using the provided `dissimilarity_function` or raises a `DistanceSolverError`.
### `root_tree`
#### Description
(Abstract method) Roots the tree at the specified root sample.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------- | :--------------------------------------- |
| tree | nx.Graph | The unrooted tree topology. |
| root\_sample | str | The sample to treat as the root. |
| remaining\_samples | List\[str] | The last two unjoined nodes in the tree. |
#### Outputs
| Name | Type | Description |
| :----------- | :--------- | :--------------- |
| rooted\_tree | nx.DiGraph | The rooted tree. |
### `find_cherry`
#### Description
(Abstract method) Selects two samples to join together as a cherry.
#### Inputs
| Name | Type | Description |
| :----------------- | :-------------- | :-------------------------------------- |
| dissimilarity\_map | np.array(float) | The dissimilarity map relating samples. |
#### Outputs
| Name | Type | Description |
| :----- | :--------------- | :--------------------------------- |
| cherry | Tuple\[int, int] | A tuple of sample indices to join. |
### `update_dissimilarity_map`
#### Description
(Abstract method) Updates the dissimilarity map after joining a cherry.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :----------------------------------- |
| dissimilarity\_map | pd.DataFrame | The dissimilarity map to update. |
| cherry | Tuple\[str, str] | The joined cherry nodes. |
| new\_node | str | The new node name to add to the map. |
#### Outputs
| Name | Type | Description |
| :----------- | :----------- | :----------------------------- |
| updated\_map | pd.DataFrame | The updated dissimilarity map. |
### `setup_root_finder`
#### Description
(Abstract method) Defines how an implicit root is to be added to the tree.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :-------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The `CassiopeiaTree` object to set up the root for. |
#### Outputs
None
## Side Effects
The `solve` method modifies the input `CassiopeiaTree` object by populating its tree attribute with the solved tree topology.
## Dependencies
* `ete3`: Used for tree manipulation and visualization.
* `networkx`: Used for graph representation and algorithms.
* `numpy`: Used for numerical operations.
* `pandas`: Used for data manipulation and representation.
* `configparser`: Used for reading configuration files.
* `subprocess`: Used for running external commands (CCPhylo).
## Configuration
The `_ccphylo_solve` method relies on the `ccphylo_path` option defined in the configuration file to locate the CCPhylo executable.
# GreedySolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/GreedySolver.py
## High-level description
The `GreedySolver` class in Cassiopeia is an abstract base class for implementing top-down greedy algorithms for phylogenetic tree reconstruction. It provides a framework for recursively splitting a set of samples based on a split criterion, building the tree from the root down to the leaves. Subclasses of `GreedySolver` implement the specific logic for performing the split.
## Code Structure
The `GreedySolver` class defines the overall structure of a greedy solver, including methods for performing splits, computing mutation frequencies, and adding duplicate samples to the tree. The `perform_split` method is an abstract method that must be implemented by subclasses to define the specific split criterion.
## References
* `CassiopeiaSolver`: The `GreedySolver` class inherits from the `CassiopeiaSolver` abstract base class.
* `solver_utilities`: The `GreedySolver` class uses utility functions from the `solver_utilities` module, such as `node_name_generator` and `transform_priors`.
* `missing_data_methods`: The `GreedySolver` class uses functions from the `missing_data_methods` module to handle missing data during character splits.
* `find_duplicate_groups`: The `GreedySolver` class uses the `find_duplicate_groups` function from the `cassiopeia.mixins` module to identify duplicate samples.
## Symbols
### `GreedySolver`
#### Description
This class represents a greedy solver for phylogenetic tree reconstruction. It provides a framework for recursively splitting a set of samples based on a split criterion, building the tree from the root down to the leaves.
#### Inputs
| Name | Type | Description |
| :-------------------- | :--- | :---------------------------------------------------------------------------------- |
| prior\_transformation | str | Function to use when transforming priors into weights. Defaults to "negative\_log". |
#### Outputs
None
#### Internal Logic
1. Transform the priors into weights using the specified `prior_transformation`.
2. Recursively split the set of samples using the `perform_split` method:
a. Create an ancestral node for each split.
b. Place the resulting partitions as daughter clades.
3. Continue splitting until each partition contains only a single sample.
4. If a split cannot be performed, create a polytomy.
5. Add duplicate samples to the tree.
6. Collapse mutationless edges if specified.
### `perform_split`
#### Description
This abstract method defines the logic for partitioning a set of samples. It must be implemented by subclasses of `GreedySolver`.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :----------------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | Character matrix |
| samples | List\[int] | A list of samples to partition |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Weighting of each (character, state) pair. Typically a transformation of the priors. |
| missing\_state\_indicator | int | Character representing missing data. |
#### Outputs
| Name | Type | Description |
| :--------------- | :---------------------- | :------------------------ |
| left\_partition | List\[Union\[int, str]] | The left partition group |
| right\_partition | List\[Union\[int, str]] | The right partition group |
### `compute_mutation_frequencies`
#### Description
This method computes the frequencies of each character/state pair in a given set of samples.
#### Inputs
| Name | Type | Description |
| :------------------------ | :----------- | :----------------------------------- |
| samples | List\[str] | The set of samples to consider |
| unique\_character\_matrix | pd.DataFrame | The character matrix |
| missing\_state\_indicator | int | Character representing missing data. |
#### Outputs
| Name | Type | Description |
| :--------- | :-------------------------- | :---------------------------------------------------------------------------------- |
| freq\_dict | Dict\[int, Dict\[int, int]] | A dictionary mapping each character to a dictionary of state/sample frequency pairs |
#### Internal Logic
1. Iterate over each character in the character matrix.
2. For each character, count the occurrences of each state for the given samples.
3. Store the frequencies in a dictionary:
* Keys are characters
* Values are dictionaries mapping states to their frequencies
### `__add_duplicates_to_tree`
#### Description
This private method adds duplicate samples to the tree.
#### Inputs
| Name | Type | Description |
| :-------------------- | :-------------------------- | :--------------------------------------- |
| tree | nx.DiGraph | The tree to add duplicates to |
| character\_matrix | pd.DataFrame | Character matrix |
| node\_name\_generator | Generator\[str, None, None] | Generator for creating unique node names |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :----------------------------- |
| tree | nx.DiGraph | The tree with duplicates added |
#### Internal Logic
1. Identify duplicate samples using the `find_duplicate_groups` function.
2. For each group of duplicates:
a. Create a new internal node.
b. Connect the new node to the original sample.
c. Connect the new node to all duplicates of the original sample.
## Side Effects
The `solve` method modifies the input `CassiopeiaTree` object by populating its tree attribute.
## Dependencies
* `networkx`: Used for representing and manipulating the tree.
* `numpy`: Used for numerical operations.
* `pandas`: Used for representing the character matrix.
# HybridSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/HybridSolver.py
## High-level description
The `HybridSolver` class in Cassiopeia implements a hybrid approach to phylogenetic tree reconstruction. It combines a top-down greedy solver with a more precise bottom-up solver to efficiently infer relationships in large datasets. The greedy solver is applied initially to cluster cells until a specified cutoff is reached (either based on the number of cells or the distance to the latest common ancestor). Then, the bottom solver is used to reconstruct subtrees for each cluster, resulting in a complete tree.
## Code Structure
The `HybridSolver` class inherits from the `CassiopeiaSolver` class and utilizes two solver instances: `top_solver` (a subclass of `GreedySolver`) and `bottom_solver` (any subclass of `CassiopeiaSolver`). The `solve` method orchestrates the hybrid solving process, while `apply_top_solver` and `apply_bottom_solver` handle the respective solver applications. `assess_cutoff` determines when to switch from the top solver to the bottom solver.
## References
* `GreedySolver`: The `HybridSolver` relies on a `GreedySolver` instance for the initial top-down clustering.
* `CassiopeiaSolver`: Both `top_solver` and `bottom_solver` are subclasses of `CassiopeiaSolver`, providing a common interface for tree reconstruction.
* `CassiopeiaTree`: The `HybridSolver` operates on a `CassiopeiaTree` object, which stores the character matrix, priors, and the resulting tree.
## Symbols
### `HybridSolver`
#### Description
This class implements the hybrid phylogenetic tree reconstruction algorithm. It combines a greedy top-down solver with a more precise bottom-up solver to handle large datasets efficiently.
#### Inputs
| Name | Type | Description |
| :-------------------- | :-------------------------------- | :------------------------------------------------------------------------------------------------------------------------- |
| top\_solver | GreedySolver.GreedySolver | An algorithm to be applied at the top of the tree. Must be a subclass of GreedySolver. |
| bottom\_solver | CassiopeiaSolver.CassiopeiaSolver | An algorithm to be applied at the bottom of the tree. Must be a subclass of CassiopeiaSolver. |
| lca\_cutoff | float | Distance to the latest-common-ancestor (LCA) of a subclade to be used as a cutoff for transitioning to the bottom solver. |
| cell\_cutoff | int | Number of cells in a subclade to be used as a cutoff for transitioning to the bottom solver. |
| threads | int | Number of threads to be used. This corresponds to the number of subproblems to be run concurrently with the bottom solver. |
| prior\_transformation | str | Function to use when transforming priors into weights. |
| progress\_bar | bool | Indicates if a progress bar should be shown when `solve` is called. |
#### Outputs
The `HybridSolver` does not directly return any outputs. It modifies the input `CassiopeiaTree` object by populating it with the reconstructed tree.
#### Internal Logic
1. **Apply Top Solver:** The `apply_top_solver` method is used to recursively cluster cells using the `top_solver` until the specified cutoff is reached (either `lca_cutoff` or `cell_cutoff`).
2. **Generate Subproblems:** As the top solver clusters cells, it generates subproblems, each consisting of a subtree root and a list of samples within that subtree.
3. **Apply Bottom Solver:** The `apply_bottom_solver` method is applied to each subproblem, using the `bottom_solver` to reconstruct the subtree for the corresponding samples.
4. **Combine Subtrees:** The resulting subtrees are combined into a single tree, forming the complete reconstructed phylogeny.
5. **Add Duplicates and Prune:** Duplicate samples are added to the tree, and any spurious leaves (not present in the original character matrix) are pruned.
6. **Populate CassiopeiaTree:** The final reconstructed tree is used to populate the input `CassiopeiaTree` object.
#### Side Effects
* Modifies the input `CassiopeiaTree` object by populating it with the reconstructed tree.
* Creates log files for each subproblem solved by the bottom solver.
### `apply_top_solver`
#### Description
This method recursively applies the top solver to cluster cells until the specified cutoff is reached.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :----------------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | Character matrix |
| samples | List\[str] | Samples in the subclade of interest. |
| tree | nx.DiGraph | In progress tree for the HybridSolver. |
| node\_name\_generator | Generator\[str, None, None] | Generator for creating unique node names while applying the top-solver. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Weights of character-state combinations, derived from priors if these are available. |
| missing\_state\_indicator | int | Indicator for missing data |
| root | Optional\[int] | Node ID of the root in the subtree containing the samples. |
#### Outputs
| Name | Type | Description |
| :---------- | :----------------------------- | :------------------------------------------------------------------------- |
| root | int | The ID of the node serving as the root of the tree containing the samples. |
| subproblems | List\[Tuple\[int, List\[str]]] | A list of subproblems in the form \[subtree-root, subtree-samples]. |
| tree | nx.DiGraph | The updated tree with the top solver applied. |
#### Internal Logic
1. **Base Case:** If there is only one sample, return the sample ID, an empty list of subproblems, and the unmodified tree.
2. **Perform Split:** Use the `top_solver` to partition the samples based on the character matrix and weights.
3. **Create Root:** Generate a unique node name for the root of the current subtree.
4. **Handle Single Clade:** If the split results in a single clade, connect all samples in the clade to the root and return.
5. **Recursive Application:** For each clade resulting from the split:
* If the cutoff is reached, add the clade as a subproblem.
* Otherwise, recursively apply `apply_top_solver` to the clade, connect the resulting subtree to the current root, and add any new subproblems.
6. **Return Results:** Return the root ID, the list of subproblems, and the updated tree.
### `apply_bottom_solver`
#### Description
This method applies the bottom solver to a given subproblem, reconstructing the subtree for the corresponding samples.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :------------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | CassiopeiaTree for the entire dataset. This will be subsetted with respect to the samples specified. |
| root | int | Identifier of the root in the master tree |
| samples | List\[str] | A list of samples for which to infer a tree. |
| logfile | str | Base location for logging output. A specific logfile will be created from this base logfile name. |
| layer | Optional\[str] | Layer storing the character matrix for solving. If None, the default character matrix is used in the CassiopeiaTree. |
#### Outputs
| Name | Type | Description |
| :--------------- | :--------- | :--------------------------------------------------------- |
| subproblem\_tree | nx.DiGraph | A tree in the form of a Networkx graph for the subproblem. |
| root | int | The original root identifier. |
#### Internal Logic
1. **Base Case:** If there is only one sample, create a simple tree with the root connected to the sample and return.
2. **Subset Character Matrix:** Extract the relevant portion of the character matrix corresponding to the input samples.
3. **Create Subtree:** Create a new `CassiopeiaTree` object for the subproblem, using the subsetted character matrix and priors from the original tree.
4. **Solve Subproblem:** Apply the `bottom_solver` to the subtree, reconstructing the phylogeny for the subproblem samples.
5. **Connect to Root:** Connect the root of the reconstructed subtree to the input root ID.
6. **Return Results:** Return the reconstructed subtree and the original root ID.
### `assess_cutoff`
#### Description
This method determines whether the specified cutoff has been reached for a given set of samples.
#### Inputs
| Name | Type | Description |
| :------------------------ | :----------- | :---------------------------- |
| samples | List\[str] | A list of samples in a clade. |
| character\_matrix | pd.DataFrame | Character matrix |
| missing\_state\_indicator | int | Indicator for missing data. |
#### Outputs
| Name | Type | Description |
| :-------------- | :--- | :------------------------------------------- |
| cutoff\_reached | bool | True if the cutoff is reached, False if not. |
#### Internal Logic
1. **Cell Cutoff:** If `cell_cutoff` is specified, check if the number of samples is less than or equal to the cutoff. If so, return True.
2. **LCA Cutoff:** If `lca_cutoff` is specified:
* Calculate the LCA character states for the samples.
* Calculate the Hamming distance between each sample's character states and the LCA character states.
* If the maximum distance is less than or equal to the `lca_cutoff`, return True.
3. **Return False:** If neither cutoff is reached, return False.
### `__add_duplicates_to_tree_and_remove_spurious_leaves`
#### Description
This private method adds duplicate samples to the tree and removes any spurious leaves that are not present in the original character matrix.
#### Inputs
| Name | Type | Description |
| :-------------------- | :-------------------------- | :---------------------------------------- |
| tree | nx.DiGraph | The tree after solving. |
| character\_matrix | pd.DataFrame | Character matrix |
| node\_name\_generator | Generator\[str, None, None] | Generator for creating unique node names. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :--------------------------------------------------------- |
| tree | nx.DiGraph | The tree with duplicates added and spurious leaves pruned. |
#### Internal Logic
1. **Add Duplicates:**
* Identify groups of duplicate samples based on their character states.
* For each group, create a new internal node and connect it to the original sample and all its duplicates.
2. **Remove Spurious Leaves:**
* Identify leaves in the tree that are not present in the character matrix.
* For each spurious leaf, remove it and prune its lineage until a node with at least two children is reached.
3. **Return Tree:** Return the updated tree.
## Dependencies
* `networkx`: Used for representing and manipulating the tree structure.
* `pandas`: Used for handling the character matrix and dissimilarity map.
* `multiprocessing`: Used for parallelizing the bottom solver application.
* `tqdm`: Used for displaying progress bars.
## Error Handling
* `HybridSolverError` is raised if no cutoff is specified (either `lca_cutoff` or `cell_cutoff`).
## Logging
The `solve` method creates log files for each subproblem solved by the bottom solver. The base file name for the log files is specified by the `logfile` argument, and a unique identifier is appended to each file name.
## TODOs
None.
# ILPSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/ILPSolver.py
## High-level description
The `ILPSolver` class implements the Cassiopeia-ILP algorithm for inferring a maximum parsimony phylogenetic tree. It constructs a potential graph of possible evolutionary states and uses Gurobi, an ILP solver, to find the maximum parsimony Steiner Tree within this graph. This tree represents the proposed phylogenetic relationship between the input samples.
## Code Structure
The `ILPSolver` class inherits from the `CassiopeiaSolver` abstract class. It overrides the `solve` method to implement the Cassiopeia-ILP algorithm. The main steps involve inferring a potential graph, generating a Steiner Tree ILP model, solving the ILP problem, post-processing the solution, and populating the resulting tree in a `CassiopeiaTree` object.
## References
* `ilp_solver_utilities`: This module provides utility functions for inferring the potential graph and other ILP-related operations.
* `dissimilarity_functions`: This module provides functions for calculating dissimilarity between character states, including weighted Hamming distance.
* `solver_utilities`: This module provides general utility functions for tree solvers, such as transforming priors into weights.
* `gurobipy`: This module provides the interface to the Gurobi optimizer.
## Symbols
### `ILPSolver`
#### Description
This class implements the Cassiopeia-ILP algorithm for phylogenetic tree inference.
#### Inputs
| Name | Type | Description |
| :--------------------------------------- | :------------- | :------------------------------------------------------------------------------------------------------------------------------------- |
| convergence\_time\_limit | int | Amount of time allotted to the ILP for convergence (default: 12600 seconds). |
| convergence\_iteration\_limit | int | Number of iterations allowed for ILP convergence (default: 0, meaning no limit). |
| maximum\_potential\_graph\_layer\_size | int | Maximum size allowed for an iteration of the potential graph inference procedure (default: 10000). |
| maximum\_potential\_graph\_lca\_distance | Optional\[int] | Maximum height of LCA to add to the potential graph (default: None, meaning the maximum distance between any pair of samples is used). |
| weighted | bool | Whether to weight edges on the potential graph by the negative log likelihood of the mutations (default: False). |
| seed | Optional\[int] | Random seed to use during ILP optimization (default: None). |
| mip\_gap | float | Objective gap for mixed integer linear programming problem (default: 0.01). |
| prior\_transformation | str | Function to use when transforming priors into weights (default: "negative\_log"). |
#### Outputs
The `solve` method does not return any value but populates the provided `CassiopeiaTree` object with the inferred tree.
#### Internal Logic
The `solve` method performs the following steps:
1. **Preprocessing:** Checks for ambiguous states, prepares the character matrix, and transforms priors into weights if necessary.
2. **Root Identification:** Identifies the root of the tree based on the least common ancestor of all samples.
3. **Potential Graph Inference:** Infers the potential graph using the `infer_potential_graph` method.
4. **Steiner Tree Model Generation:** Generates a Gurobi ILP model for the Steiner Tree problem using the `generate_steiner_model` method.
5. **ILP Solution:** Solves the ILP problem using the `solve_steiner_instance` method.
6. **Solution Post-processing:** Post-processes the solution to remove self-loops and enforce a single parent for each node using the `post_process_steiner_solution` method.
7. **Tree Population:** Appends sample names to the solution, removes spurious leaves, and populates the `CassiopeiaTree` object with the resulting tree.
### `infer_potential_graph`
#### Description
This method infers the potential graph, a network of possible evolutionary intermediates, from the character matrix.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------------------------------- | :----------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | Character matrix of mutation observations. |
| pid | int | Process ID for future reference. |
| lca\_height | int | Maximum LCA height to consider for connecting nodes to an LCA. |
| weights | Optional\[Dict\[int, Dict\[int, str]]] | Weights for character-state pairs, derived from the priors if available. |
| missing\_state\_indicator | int | Indicator for missing data (default: -1). |
#### Outputs
| Name | Type | Description |
| :--------------- | :--------- | :------------------------------------------------- |
| potential\_graph | nx.DiGraph | A potential graph represented by a directed graph. |
#### Internal Logic
1. **Edge Inference:** Calls the `ilp_solver_utilities.infer_potential_graph_cython` function to obtain the edges of the potential graph in character string format.
2. **Edge Decoding:** Decodes the character strings into tuples representing character states.
3. **Graph Construction:** Creates a Networkx directed graph and adds the decoded edges.
4. **Edge Weighting:** Adds edge weights based on the provided weights or the number of mutations along each edge using the `add_edge_weights` method.
### `add_edge_weights`
#### Description
This method annotates edges in the potential graph with weights.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------------------------------- | :--------------------------------------------- |
| potential\_graph | nx.DiGraph | Potential graph to annotate with edge weights. |
| weights | Optional\[Dict\[int, Dict\[int, str]]] | Weights for character-state pairs. |
| missing\_state\_indicator | int | Indicator for missing data (default: -1). |
#### Outputs
| Name | Type | Description |
| :-------------- | :--------- | :----------------------------------------------------------------------------- |
| weighted\_graph | nx.DiGraph | The potential graph with edge weights added, stored in the `weight` attribute. |
#### Internal Logic
Iterates through the edges of the potential graph and calculates the weight for each edge using either the provided weights or the weighted Hamming distance between the character states of the connected nodes.
### `generate_steiner_model`
#### Description
This method generates a Gurobi ILP model for the Steiner Tree problem.
#### Inputs
| Name | Type | Description |
| :--------------- | :---------------- | :--------------------------------------------------- |
| potential\_graph | nx.DiGraph | Potential graph representing the evolutionary space. |
| root | List\[int] | A node in the graph to treat as the source. |
| targets | List\[List\[int]] | A list of nodes in the tree that serve as targets. |
#### Outputs
| Name | Type | Description |
| :-------------- | :------------------------------------ | :-------------------------------- |
| model | gurobipy.Model | A Gurobipy Model instance. |
| edge\_variables | Dict\[Tuple\[int, int], gurobipy.Var] | Edge variables used in the model. |
#### Internal Logic
1. **Model Initialization:** Creates a Gurobi model instance.
2. **Variable Addition:** Adds flow variables for each edge and binary variables indicating edge usage.
3. **Constraint Addition:** Adds constraints for flow conservation and edge usage.
4. **Objective Definition:** Sets the objective function to minimize the total weight of the used edges.
### `solve_steiner_instance`
#### Description
This method solves the Steiner Tree problem using the generated Gurobi model.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------------------------------ | :--------------------------------------------------------- |
| model | gurobipy.Model | A Gurobi model instance. |
| edge\_variables | Dict\[Tuple\[int, int], gurobipy.Var] | Edge variables used in the model. |
| potential\_graph | nx.DiGraph | Potential graph used as input to the Steiner Tree problem. |
| pid | int | Process ID. |
| logfile | str | Location to store standard out. |
#### Outputs
| Name | Type | Description |
| :-------- | :---------------- | :--------------------------------------------------------- |
| solutions | List\[nx.DiGraph] | A list of solutions, each represented as a directed graph. |
#### Internal Logic
1. **Parameter Configuration:** Sets Gurobi parameters, including convergence criteria, MIP gap, and logging options.
2. **Model Optimization:** Calls the `model.optimize()` method to solve the ILP problem.
3. **Solution Recovery:** Extracts the solutions from the Gurobi model and converts them into Networkx directed graphs.
### `post_process_steiner_solution`
#### Description
This method post-processes the Steiner Tree solution to remove self-loops and enforce a single parent for each node.
#### Inputs
| Name | Type | Description |
| :------- | :--------- | :------------------- |
| solution | nx.DiGraph | The Gurobi solution. |
| root | List\[int] | The root node. |
#### Outputs
| Name | Type | Description |
| :------------------ | :--------- | :------------------------------ |
| processed\_solution | nx.DiGraph | A cleaned up Networkx solution. |
#### Internal Logic
1. **Self-loop Removal:** Removes any self-loops from the solution.
2. **Spurious Root Removal:** Removes any spurious roots, ensuring only one root remains.
3. **Single Parent Enforcement:** For nodes with multiple parents, selects the parent with the highest edge weight and removes edges to other parents.
### `__append_sample_names_and_remove_spurious_leaves`
#### Description
This private method appends sample names to the character states in the tree and prunes spurious leaves.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------- | :----------------------------------------- |
| solution | nx.DiGraph | A Steiner Tree solution. |
| character\_matrix | pd.DataFrame | Character matrix of mutation observations. |
#### Outputs
| Name | Type | Description |
| :------- | :--------- | :---------------------------------------------------------- |
| solution | nx.DiGraph | A solution with extra leaves corresponding to sample names. |
#### Internal Logic
1. **Sample Appending:** Appends sample names to the deepest nodes in the tree that correspond to their character states.
2. **Spurious Leaf Removal:** Removes any extant nodes that do not have samples appended to them and prunes their lineages.
## Side Effects
The `solve` method modifies the provided `CassiopeiaTree` object by populating it with the inferred tree.
## Dependencies
| Dependency | Purpose |
| :--------- | :--------------------------------------------------------------- |
| gurobipy | Used for solving the ILP problem. |
| networkx | Used for representing the potential graph and the inferred tree. |
| pandas | Used for handling the character matrix. |
| numpy | Used for numerical operations. |
## Error Handling
The `ILPSolver` class raises `ILPSolverError` exceptions in the following cases:
* If the input character matrix contains ambiguous states.
* If the root node is not found in the potential graph.
* If any target node is not found in the potential graph.
* If Gurobi is not installed.
* If the potential graph cannot be inferred with the specified parameters.
## Logging
The `solve` method logs progress and information about the potential graph inference and Steiner Tree optimization to the specified log file.
# MaxCutGreedySolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/MaxCutGreedySolver.py
## High-level description
The `MaxCutGreedySolver` class, inheriting from `GreedySolver`, implements a phylogenetic tree reconstruction algorithm that combines a greedy approach with a max-cut optimization. It recursively partitions samples based on the most frequent mutation and then refines the split using a hill-climbing method to maximize the weight of edges cut in a connectivity graph built from shared mutations.
## Code Structure
The `MaxCutGreedySolver` class overrides the `perform_split` method of its parent class, `GreedySolver`. It leverages the `compute_mutation_frequencies` method from the parent class and utilizes functions like `construct_connectivity_graph` and `max_cut_improve_cut` from the `graph_utilities` module.
## References
This code references the following symbols from other files:
* `GreedySolver` (from cassiopeia/solver/GreedySolver.py)
* `missing_data_methods` (from cassiopeia/solver/missing\_data\_methods.py)
* `solver_utilities` (from cassiopeia/solver/solver\_utilities.py)
* `graph_utilities` (from cassiopeia/solver/graph\_utilities.py)
## Symbols
### `MaxCutGreedySolver`
#### Description
This class implements the MaxCut Greedy algorithm for phylogenetic tree reconstruction. It extends the basic GreedySolver by adding a max-cut optimization step to each split.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------- | :---------------------------------------------------------------------------------------------------------------- |
| missing\_data\_classifier | Callable | Function to handle missing data during character splits (default: `missing_data_methods.assign_missing_average`). |
| prior\_transformation | str | Transformation function for converting priors to weights (default: `"negative_log"`). |
#### Outputs
This class doesn't directly return any output. It populates a `CassiopeiaTree` object during the `solve` method (inherited from `GreedySolver`).
#### Internal Logic
The core logic resides in the `perform_split` method:
1. **Identify Splitting Character/State:** It identifies the most frequent character/state pair, considering potential weights derived from priors.
2. **Initial Split:** Samples are initially partitioned based on the chosen character/state.
3. **Handle Missing Data:** The `missing_data_classifier` function is used to assign samples with missing data to either side of the partition.
4. **Construct Connectivity Graph:** A connectivity graph is built where nodes represent samples and edge weights reflect the similarity based on shared mutations.
5. **Max-Cut Optimization:** The `max_cut_improve_cut` function refines the initial split by iteratively moving nodes between partitions to maximize the weight of edges cut, effectively grouping samples with similar mutations.
6. **Return Refined Split:** The optimized left and right partitions are returned.
### `perform_split`
#### Description
This method partitions a set of samples based on the most frequent mutation and optimizes the split using the max-cut criterion.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :--------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | Character matrix containing character states for each sample. |
| samples | List\[int] | List of samples to be partitioned. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Weights for each (character, state) pair, usually derived from priors. |
| missing\_state\_indicator | int | Character representing missing data (default: -1). |
#### Outputs
| Name | Type | Description |
| :------------------- | :--------- | :--------------------------------------------------------- |
| improved\_left\_set | List\[str] | List of samples in the left partition after optimization. |
| improved\_right\_set | List\[str] | List of samples in the right partition after optimization. |
#### Internal Logic
See the 'Internal Logic' section of the `MaxCutGreedySolver` class description.
## Side Effects
The `perform_split` method modifies the input `left_set` and `right_set` lists in-place during the missing data imputation and max-cut optimization steps.
## Dependencies
This code depends on the following external libraries:
| Dependency | Purpose |
| :--------- | :----------------------------------------------------------------------------------- |
| pandas | Handling and manipulating character matrices. |
| networkx | Constructing and manipulating graphs for representing relationships between samples. |
| numpy | Numerical operations, particularly in calculating scores and frequencies. |
## Error Handling
This code does not implement specific error handling beyond what is inherited from the parent class `GreedySolver`.
# MaxCutSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/MaxCutSolver.py
## High-level description
The `MaxCutSolver` class is a subclass of `GreedySolver` that implements a graph-based algorithm for inferring phylogenetic trees. It aims to find an optimal partition of samples based on their shared and differing mutations, represented as a connectivity graph, by approximating the max-cut problem.
## Code Structure
The `MaxCutSolver` class inherits from `GreedySolver` and overrides the `perform_split` method. It utilizes the `construct_connectivity_graph`, `evaluate_cut`, and `max_cut_improve_cut` functions from the `graph_utilities` module to construct and manipulate the connectivity graph.
## References
This code references the following symbols from other files:
* `GreedySolver` (cassiopeia/solver/GreedySolver.py)
* `graph_utilities` (cassiopeia/solver/graph\_utilities.py)
## Symbols
### MaxCutSolver
#### Description
This class implements the MaxCut algorithm for phylogenetic tree inference. It inherits from the `GreedySolver` class and provides a concrete implementation for the `perform_split` method.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------- | :------------------------------------------------------------------------ |
| sdimension | Optional\[int] | The number of dimensions for the embedding space (default: 3) |
| iterations | Optional\[int] | The number of iterations for updating the embeddings (default: 50) |
| prior\_transformation | str | The transformation function to apply to priors (default: "negative\_log") |
#### Outputs
This class doesn't directly return any output. It modifies the input `CassiopeiaTree` object during the `solve` method.
#### Internal Logic
The `perform_split` method performs the following steps:
1. **Construct Connectivity Graph:** It builds a connectivity graph where nodes represent samples and edge weights represent the difference between the number of triplets separating and grouping the samples based on shared mutations.
2. **Embed Samples:** Randomly embed the samples in a d-dimensional sphere.
3. **Update Embeddings:** Iteratively update the embeddings based on edge weights, pulling together nodes with strong negative connections and pushing apart nodes with positive connections.
4. **Find Max Cut:** Generate random hyperplanes to bisect the d-sphere and select the one that maximizes the cut weight, effectively partitioning the samples.
5. **Improve Cut:** Apply a greedy hill-climbing procedure to further optimize the cut by iteratively moving nodes between partitions to maximize the cut weight.
#### Performance Considerations
The performance of the algorithm is influenced by the `sdimension` and `iterations` parameters. Higher values may lead to better solutions but increase computational cost.
### evaluate\_cut
#### Description
This function calculates the weight of a cut in a graph.
#### Inputs
| Name | Type | Description |
| :--- | :--------- | :----------------------------------------------- |
| cut | List\[str] | A list of nodes representing one side of the cut |
| G | nx.DiGraph | The graph on which the cut is evaluated |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :----------------------------------------- |
| cut\_score | float | The total weight of edges crossing the cut |
#### Internal Logic
The function iterates through each edge in the graph and checks if it crosses the cut (i.e., if one node is in the `cut` list and the other is not). If an edge crosses the cut, its weight is added to the `cut_score`.
### Side Effects
This function has no side effects.
***
# NeighborJoiningSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/NeighborJoiningSolver.py
## High-level description
The `NeighborJoiningSolver` class implements the Neighbor-Joining algorithm for phylogenetic tree reconstruction. It iteratively joins pairs of samples (or cherries) that minimize the Q-criterion based on a dissimilarity map, ultimately building an unrooted tree. The class provides options for rooting the tree and using fast implementations from the CCPhylo library.
## Code Structure
The `NeighborJoiningSolver` class inherits from the `DistanceSolver` class and overrides several methods to implement the Neighbor-Joining algorithm. The main methods are `find_cherry`, `update_dissimilarity_map`, and `root_tree`. The `find_cherry` method identifies the pair of samples to join based on the Q-criterion. The `update_dissimilarity_map` method updates the dissimilarity map after joining a cherry. The `root_tree` method roots the tree at a specified sample.
## References
* `DistanceSolver`: The parent class that defines the general structure for distance-based solvers.
* `dissimilarity_functions`: A module containing various dissimilarity functions, including the weighted Hamming distance used by default.
* `solver_utilities`: A module containing utility functions for tree solvers.
## Symbols
### `NeighborJoiningSolver`
#### Description
This class implements the Neighbor-Joining algorithm for phylogenetic tree reconstruction. It inherits from the `DistanceSolver` class and overrides several methods to implement the specific logic of the Neighbor-Joining algorithm.
#### Inputs
| Name | Type | Description |
| :---------------------- | :------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------- |
| dissimilarity\_function | Optional\[Callable\[\[np.array, np.array, int, Dict\[int, Dict\[int, float]]], float]] | A function to compute the dissimilarity between samples. Defaults to `weighted_hamming_distance`. |
| add\_root | bool | Whether to add an implicit root to the tree. Defaults to `False`. |
| prior\_transformation | str | The transformation to apply to priors when calculating weights. Defaults to `"negative_log"`. |
| fast | bool | Whether to use a fast implementation of Neighbor-Joining. Defaults to `False`. |
| implementation | str | The fast implementation to use. Options are: `"ccphylo_dnj"`, `"ccphylo_hnj"`, `"ccphylo_nj"`. Defaults to `"ccphylo_dnj"`. |
| threads | int | The number of threads to use for the solver. Defaults to `1`. |
#### Outputs
The `solve` method populates the `tree` attribute of the input `CassiopeiaTree` object with the reconstructed tree.
#### Internal Logic
The `solve` method iteratively joins pairs of samples (cherries) that minimize the Q-criterion, updating the dissimilarity map at each step. The process continues until only two samples remain, which are then joined to form the final unrooted tree. The tree can be rooted using the `root_tree` method.
### `root_tree`
#### Description
This method roots the tree produced by the Neighbor-Joining algorithm at the specified root sample.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------- | :---------------------------------------------------------- |
| tree | nx.Graph | The unrooted tree topology represented as a NetworkX graph. |
| root\_sample | str | The sample to treat as the root. |
| remaining\_samples | List\[str] | The last two unjoined nodes in the tree. |
#### Outputs
| Name | Type | Description |
| :----------- | :--------- | :----------------------------------------------------------------- |
| rooted\_tree | nx.DiGraph | The rooted tree topology represented as a NetworkX directed graph. |
#### Internal Logic
The method first joins the last two remaining samples in the tree. Then, it performs a depth-first search from the specified root sample and creates a directed graph based on the edges traversed.
### `find_cherry`
#### Description
This method finds a pair of samples to join into a cherry by minimizing the Q-criterion.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------- | :-------------------------------------- |
| dissimilarity\_matrix | np.array | A sample x sample dissimilarity matrix. |
#### Outputs
| Name | Type | Description |
| :----- | :--------------- | :--------------------------------------------------------------------------------------------------- |
| cherry | Tuple\[int, int] | A tuple of integers representing the indices of the two samples to join in the dissimilarity matrix. |
#### Internal Logic
The method calculates the Q-criterion for every pair of samples using the `compute_q` method. The pair with the minimum Q-criterion is selected as the cherry.
### `compute_q`
#### Description
This static method computes the Q-criterion for every pair of samples in the dissimilarity map.
#### Inputs
| Name | Type | Description |
| :----------------- | :------------ | :----------------------------------- |
| dissimilarity\_map | np.array(int) | A sample x sample dissimilarity map. |
#### Outputs
| Name | Type | Description |
| :--- | :------- | :---------------------------------------------------------- |
| q | np.array | A matrix storing the Q-criterion for every pair of samples. |
#### Internal Logic
The method iterates through all pairs of samples and calculates the Q-criterion using the formula: Q(i,j) = d(i, j) - 1/(n-2) (sum(d(i, :)) + sum(d(j,:))), where d(i,j) is the dissimilarity between samples i and j, and n is the number of samples.
### `update_dissimilarity_map`
#### Description
This method updates the dissimilarity map after joining two nodes into a cherry.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :--------------------------------------------------------------------------------- |
| dissimilarity\_map | pd.DataFrame | The dissimilarity map to update. |
| cherry | Tuple\[str, str] | A tuple of strings representing the names of the two samples joined in the cherry. |
| new\_node | str | The name of the new node representing the cherry. |
#### Outputs
| Name | Type | Description |
| :----------------- | :----------- | :----------------------------- |
| dissimilarity\_map | pd.DataFrame | The updated dissimilarity map. |
#### Internal Logic
The method calculates the new dissimilarities between the new node and all other nodes using the formula: d'(m, v) = 0.5 \* (d(v, m1) + d(v, m2) - d(m1, m2)), where m is the new node, v is any other node, and m1 and m2 are the two samples joined in the cherry. The original cherry nodes are then removed from the dissimilarity map.
### `__update_dissimilarity_map_numba`
#### Description
This private static method provides a faster, Numba-optimized implementation for updating the dissimilarity map.
#### Inputs
| Name | Type | Description |
| :----------------- | :------- | :------------------------------------------------------------- |
| dissimilarity\_map | np.array | The dissimilarity map to update, represented as a NumPy array. |
| cherry\_i | int | The index of the first sample in the cherry. |
| cherry\_j | int | The index of the second sample in the cherry. |
#### Outputs
| Name | Type | Description |
| :----------- | :------- | :----------------------------- |
| updated\_map | np.array | The updated dissimilarity map. |
#### Internal Logic
The method adds a new row and column to the dissimilarity map for the new node and calculates the new dissimilarities using the same formula as `update_dissimilarity_map`.
### `setup_root_finder`
#### Description
This method defines the implicit rooting strategy for the NeighborJoiningSolver.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The input CassiopeiaTree object. |
#### Outputs
This method modifies the `cassiopeia_tree` object in place.
#### Internal Logic
The method adds an implicit root with all-zero character states to the character matrix and recalculates the dissimilarity map. If a dissimilarity map already exists, only the new dissimilarities involving the root are calculated.
## Dependencies
* `networkx`: For representing and manipulating tree topologies.
* `numba`: For optimizing performance-critical code sections.
* `pandas`: For handling data in tabular format.
* `ete3`: For parsing and manipulating trees in Newick format (used in the `_ccphylo_solve` method).
* `configparser`: For reading configuration settings (used in the `_setup_ccphylo` method).
* `subprocess`: For running external commands (used in the `_ccphylo_solve` method).
### Configuration
The `_ccphylo_solve` method requires the path to the CCPhylo executable to be specified in the `config.ini` file under the `Paths` section with the key `ccphylo_path`.
## Error Handling
The `NeighborJoiningSolver` class raises `DistanceSolverError` exceptions for various error conditions, such as missing dissimilarity function, invalid rooting parameters, or errors encountered during the execution of CCPhylo.
# PercolationSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/PercolationSolver.py
## High-level description
The `PercolationSolver` class implements a top-down, percolation-based algorithm for reconstructing phylogenetic trees. It recursively partitions a set of samples based on their similarity in observed mutations, creating a tree structure where similar samples are grouped closer together.
## Code Structure
The `PercolationSolver` class is initialized with a joining solver, prior transformation function, similarity function, and a threshold. The `solve` method orchestrates the tree reconstruction process by recursively partitioning the sample set using the `percolate` method. The `percolate` method constructs a similarity graph, removes edges based on similarity, and partitions the graph into clades. If more than two clades are formed, the `joining_solver` is used to cluster them. Finally, the `__add_duplicates_to_tree` method adds duplicate samples to the tree.
## References
This code references the following symbols:
* `CassiopeiaSolver` (from `cassiopeia.solver`)
* `dissimilarity_functions` (from `cassiopeia.solver`)
* `solver_utilities` (from `cassiopeia.solver`)
* `CassiopeiaTree` (from `cassiopeia.data`)
* `data_utilities` (from `cassiopeia.data.utilities`)
## Symbols
### `PercolationSolver`
#### Description
This class implements the Percolation algorithm for phylogenetic tree reconstruction. It takes a character matrix and optional priors as input and builds a tree by recursively partitioning the sample set based on their similarity in observed mutations.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------- |
| joining\_solver | CassiopeiaSolver.CassiopeiaSolver | The CassiopeiaSolver used to cluster groups of samples if percolation produces more than two groups. |
| prior\_transformation | str | A string specifying the transformation to apply to the priors. Defaults to "negative\_log". |
| similarity\_function | Optional\[Callable\[\[np.array, np.array, int, Optional\[Dict\[int, Dict\[int, float]]]], float]] | A function that calculates the similarity score between two samples. Defaults to `hamming_similarity_without_missing`. |
| threshold | Optional\[int] | The minimum similarity threshold for adding an edge in the similarity graph. Defaults to 0. |
#### Outputs
This class doesn't directly return any output. It modifies the input `CassiopeiaTree` object to store the reconstructed tree.
#### Internal Logic
1. **Initialization:** The class is initialized with a joining solver, prior transformation function, similarity function, and a threshold.
2. **Solve:** The `solve` method orchestrates the tree reconstruction process.
* It first extracts the character matrix and transforms the priors into weights if provided.
* It then creates a NetworkX graph and recursively partitions the sample set using the `percolate` method.
* Finally, it adds duplicate samples to the tree and populates the input `CassiopeiaTree` object.
3. **Percolate:** The `percolate` method partitions a set of samples into two groups based on their similarity.
* It constructs a similarity graph with samples as nodes and edges weighted by their similarity.
* It then iteratively removes the minimum weighted edges until the graph is disconnected into multiple components.
* If more than two components are formed, it clusters them using the `joining_solver` and returns the two resulting groups.
4. **Add Duplicates:** The `__add_duplicates_to_tree` method adds duplicate samples to the tree by finding groups of samples with identical character states and placing them as sister nodes under a new internal node.
#### Side Effects
* Modifies the input `CassiopeiaTree` object to store the reconstructed tree.
### `percolate`
#### Description
Partitions a set of samples into two groups based on their similarity in the character matrix.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | The character matrix representing the mutation profiles of the samples. |
| samples | List\[str] | A list of sample names to be partitioned. |
| priors | Optional\[Dict\[int, Dict\[int, float]]] | A dictionary of prior probabilities for each character/state pair. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | A dictionary of weights for each character/state pair, typically derived from the priors. |
| missing\_state\_indicator | int | The character representing missing data in the character matrix. Defaults to -1. |
#### Outputs
| Name | Type | Description |
| :--------------- | :----------------------------- | :---------------------------------------------------------------------------------------- |
| partition\_named | Tuple\[List\[str], List\[str]] | A tuple containing two lists of sample names, representing the left and right partitions. |
#### Internal Logic
1. **Similarity Graph:** Constructs a weighted graph with samples as nodes and edges weighted by their similarity calculated using the provided `similarity_function`.
2. **Percolation:** Iteratively removes the minimum weighted edges until the graph is disconnected into multiple connected components.
3. **Component Merging (if necessary):** If more than two connected components are formed, it clusters them using the `joining_solver` based on the LCA of each component and returns the two resulting groups.
4. **Return Partitions:** Returns a tuple containing two lists of sample names, representing the left and right partitions.
### `__add_duplicates_to_tree`
#### Description
Adds duplicate samples to the tree.
#### Inputs
| Name | Type | Description |
| :-------------------- | :-------------------------- | :---------------------------------------------------------------------- |
| tree | nx.DiGraph | The tree to which duplicate samples need to be added. |
| character\_matrix | pd.DataFrame | The character matrix representing the mutation profiles of the samples. |
| node\_name\_generator | Generator\[str, None, None] | A generator object that produces unique node names. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :------------------------------------- |
| tree | nx.DiGraph | The tree with duplicate samples added. |
#### Internal Logic
1. **Identify Duplicates:** Identifies groups of samples with identical character states in the character matrix.
2. **Add Duplicates:** For each group of duplicates:
* Creates a new internal node as the parent of the duplicates.
* Connects the original sample and all its duplicates as children of this new internal node.
3. **Return Tree:** Returns the modified tree with the duplicate samples added.
# SharedMutationJoiningSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/SharedMutationJoiningSolver.py
## High-level description
The `SharedMutationJoiningSolver` class implements an agglomerative clustering algorithm for phylogenetic tree reconstruction. It iteratively joins samples with the highest similarity based on shared mutations in their character data, building the tree bottom-up. This solver is particularly suitable for lineage tracing data with a large number of characters and limited mutation rates.
## Code Structure
The `SharedMutationJoiningSolver` class is structured around the `solve` method, which orchestrates the tree reconstruction process. It utilizes helper functions like `find_cherry` to identify the most similar samples and `update_similarity_map_and_character_matrix` to merge nodes and update the similarity matrix. The class also leverages Numba for performance optimization.
## References
This solver references the following key components:
* `dissimilarity_functions`: Provides various similarity/dissimilarity functions for comparing character vectors.
* `solver_utilities`: Offers utility functions for tree manipulation and prior transformation.
* `data_utilities`: Includes functions for computing dissimilarity maps and handling character data.
## Symbols
### `SharedMutationJoiningSolver`
#### Description
This class implements the Shared-Mutation-Joining algorithm for phylogenetic tree reconstruction. It takes a similarity function and a prior transformation method as inputs and constructs a tree by iteratively joining the most similar samples based on shared mutations.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------- |
| similarity\_function | Optional\[Callable\[\[np.array, np.array, int, Optional\[Dict\[int, Dict\[int, float]]]], float]] | Function to compute similarity between samples. Defaults to Hamming similarity without considering missing data. |
| prior\_transformation | str | Method for transforming priors into weights. Defaults to "negative\_log". |
#### Outputs
This class doesn't directly return any output. It modifies the input `CassiopeiaTree` object to store the reconstructed tree.
#### Internal Logic
1. Initializes the solver with the provided similarity function and prior transformation method.
2. Attempts to optimize the similarity function and internal methods using Numba for performance improvement.
3. In the `solve` method:
* Retrieves the character matrix and priors (if available) from the input `CassiopeiaTree`.
* Computes the initial pairwise similarity matrix.
* Iteratively identifies the most similar pair (cherry) using `find_cherry`.
* Merges the cherry into a new internal node and updates the similarity matrix using `update_similarity_map_and_character_matrix`.
* Repeats the process until all samples are joined into a single tree.
* Populates the `CassiopeiaTree` with the reconstructed tree.
* Optionally collapses mutationless edges based on inferred ancestral states.
#### Performance Considerations
* Utilizes Numba to optimize the similarity function and key internal methods for improved performance.
* Computes the dissimilarity map only once and updates it iteratively, reducing redundant calculations.
### `find_cherry`
#### Description
Identifies the pair of samples (cherry) with the highest similarity in the given similarity matrix.
#### Inputs
| Name | Type | Description |
| :----------------- | :------- | :------------------------------------------------------------------ |
| similarity\_matrix | np.array | A square matrix representing pairwise similarities between samples. |
#### Outputs
| Name | Type | Description |
| :----- | :--------------- | :----------------------------------------------------------------------------------------------- |
| cherry | Tuple\[int, int] | A tuple containing the row and column indices of the most similar pair in the similarity matrix. |
#### Internal Logic
1. Sets the diagonal elements of the similarity matrix to negative infinity to avoid self-comparison.
2. Finds the indices of the maximum element in the matrix, which represents the most similar pair.
### `update_similarity_map_and_character_matrix`
#### Description
Updates the similarity matrix and character matrix after merging a cherry into a new internal node.
#### Inputs
| Name | Type | Description |
| :------------------------ | :-------------------------------------------------------------------------- | :------------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | The character matrix storing mutation information for all nodes. |
| similarity\_function | Callable\[\[np.array, np.array, int, Dict\[int, Dict\[int, float]]], float] | The function used to compute pairwise similarity. |
| similarity\_map | pd.DataFrame | The current similarity matrix to be updated. |
| cherry | Tuple\[str, str] | A tuple containing the names of the two samples being merged. |
| new\_node | str | The name of the new internal node created by merging the cherry. |
| missing\_state\_indicator | int | The character representing missing data in the character matrix. Defaults to -1. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Weights for each character/state pair, derived from priors. Defaults to None. |
#### Outputs
| Name | Type | Description |
| :----------------------- | :----------- | :----------------------------------------------------------------- |
| updated\_similarity\_map | pd.DataFrame | The updated similarity matrix with the new internal node included. |
#### Internal Logic
1. Identifies the character vectors of the two samples in the cherry.
2. Computes the character vector of the new internal node (LCA) by taking shared mutations.
3. Adds the new node and its character vector to the character matrix.
4. Calculates the pairwise similarities between the new node and all other nodes using the provided similarity function and weights.
5. Updates the similarity matrix with the new similarities and removes the merged cherry nodes.
#### Performance Considerations
* Employs Numba to optimize the internal `__update_similarity_map` function for faster similarity updates.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------------------------------------------- |
| numba | Used for optimizing code performance. |
| numpy | Used for numerical operations and array manipulation. |
| pandas | Used for handling data in DataFrames. |
| scipy | Used for scientific computing, including distance calculations. |
| networkx | Used for creating and manipulating graph structures representing trees. |
## Error Handling
* Raises `SharedMutationJoiningSolverError` for specific issues encountered during tree reconstruction, such as invalid input parameters or inconsistencies in data.
* Issues a `SharedMutationJoiningSolverWarning` if Numba optimization fails, falling back to Python implementation.
# SpectralGreedySolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/SpectralGreedySolver.py
## High-level description
The `SpectralGreedySolver` class, inheriting from `GreedySolver`, implements a phylogenetic tree solver that combines a greedy splitting strategy with a spectral heuristic for optimizing partitions. It aims to construct a more accurate tree by refining the initial greedy splits using spectral methods to maximize the similarity within partitions based on a mutation similarity graph.
## Code Structure
The `SpectralGreedySolver` class inherits from `GreedySolver` and overrides the `perform_split` method. It utilizes functions from `graph_utilities` for graph construction and optimization. The key logic lies in how it first performs a greedy split based on mutation frequencies and then refines this split using spectral methods on a similarity graph.
## References
This code references the following symbols from other files:
* `GreedySolver` (from `cassiopeia.solver.GreedySolver`)
* `missing_data_methods` (from `cassiopeia.solver.missing_data_methods`)
* `dissimilarity_functions` (from `cassiopeia.solver.dissimilarity_functions`)
* `graph_utilities` (from `cassiopeia.solver.graph_utilities`)
* `solver_utilities` (from `cassiopeia.solver.solver_utilities`)
## Symbols
### `SpectralGreedySolver`
#### Description
This class implements a phylogenetic tree solver that combines a greedy splitting strategy with a spectral heuristic for optimizing partitions.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------------ | :---------------------------------------------------------------------------------------------------- |
| missing\_data\_classifier | Callable | A function to handle missing data during partition. Defaults to `assign_missing_average`. |
| similarity\_function | Optional\[Callable] | A function to calculate similarity between samples. Defaults to `hamming_similarity_without_missing`. |
| threshold | Optional\[int] | Minimum similarity threshold for edge inclusion in the similarity graph. Defaults to 0. |
| prior\_transformation | str | Transformation method for converting priors to weights. Defaults to "negative\_log". |
#### Outputs
The class itself doesn't have direct outputs. It's meant to be used to solve a `CassiopeiaTree` object.
#### Internal Logic
The solver first identifies the most frequent mutation to perform an initial greedy split. Then, it constructs a similarity graph based on the provided `similarity_function` and refines the partition using the `spectral_improve_cut` function from `graph_utilities`. This iterative optimization aims to minimize a modified normalized cut, effectively grouping samples with similar mutations.
### `perform_split`
#### Description
This method overrides the parent class's method and performs a partition of samples using both greedy and spectral criteria.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :--------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | Character matrix representing the mutation profiles of samples. |
| samples | List\[int] | List of samples to be partitioned. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Weights for each (character, state) pair, often derived from priors. |
| missing\_state\_indicator | int | Character representing missing data in the character matrix. Defaults to -1. |
#### Outputs
| Name | Type | Description |
| :------------------- | :--------- | :----------------------------------------------------------------- |
| improved\_left\_set | List\[str] | List of samples in the left partition after spectral improvement. |
| improved\_right\_set | List\[str] | List of samples in the right partition after spectral improvement. |
#### Internal Logic
1. **Identify Splitting Point:** The method first identifies the most frequent (character, state) pair based on the input `character_matrix` and `weights`. This pair will be used to perform the initial split.
2. **Initial Split:** Samples are divided into `left_set` and `right_set` based on the presence or absence of the chosen character state. Samples with missing data for this character are placed in the `missing` list.
3. **Handle Missing Data:** The `missing_data_classifier` function is applied to impute the missing data and assign the `missing` samples to either `left_set` or `right_set`.
4. **Construct Similarity Graph:** A similarity graph is constructed using the `construct_similarity_graph` function from `graph_utilities`. This graph represents the similarity between samples based on their mutation profiles.
5. **Spectral Improvement:** The `spectral_improve_cut` function from `graph_utilities` is applied to optimize the initial split by minimizing a modified normalized cut on the similarity graph. This step refines the partition by moving samples between `left_set` and `right_set` to maximize the similarity within each partition.
6. **Return Improved Partitions:** The method returns the optimized `improved_left_set` and `improved_right_set` lists.
# SpectralNeighborJoiningSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/SpectralNeighborJoiningSolver.py
## High-level description
The `SpectralNeighborJoiningSolver` class in Cassiopeia implements a phylogenetic tree reconstruction algorithm based on the spectral neighbor-joining method. It iteratively joins pairs of nodes (or groups of nodes) based on the second singular value of a specific matrix derived from the pairwise similarities between nodes. This process continues until a complete tree is formed.
## Code Structure
The `SpectralNeighborJoiningSolver` class inherits from the `DistanceSolver` class. It overrides the `find_cherry` and `update_dissimilarity_map` methods to implement the spectral neighbor-joining logic. The `_compute_svd2` method is a helper function used to calculate the second singular value of the relevant matrix.
## References
This solver references the `dissimilarity_functions` module for calculating pairwise similarities between nodes and the `solver_utilities` module for helper functions like generating unique node names.
## Symbols
### `SpectralNeighborJoiningSolver`
#### Description
This class implements the spectral neighbor-joining algorithm for phylogenetic tree reconstruction. It uses a spectral method to identify pairs of nodes to join based on their pairwise similarities.
#### Inputs
| Name | Type | Description |
| :-------------------- | :----------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------- |
| similarity\_function | Optional\[Callable\[\[np.ndarray, np.ndarray, int, Dict\[int, Dict\[int, float]]], float]] | A function to compute the pairwise similarity between nodes. Defaults to `exponential_negative_hamming_distance`. |
| add\_root | bool | Whether to add an implicit root to the tree. Defaults to `False`. |
| prior\_transformation | str | The transformation to apply to priors when calculating weights. Defaults to `"negative_log"`. |
#### Outputs
The solver populates the `tree` attribute of the input `CassiopeiaTree` object with the reconstructed tree.
#### Internal Logic
The solver iteratively performs the following steps:
1. **Calculate the dissimilarity map:** This map stores the pairwise dissimilarities between all nodes.
2. **Find a cherry:** Using the `find_cherry` method, the solver identifies the pair of nodes with the smallest second singular value of their corresponding RA matrix (see `_compute_svd2` for details).
3. **Update the dissimilarity map:** The `update_dissimilarity_map` method updates the dissimilarity map to reflect the newly joined cherry.
4. **Repeat steps 2-3 until only two nodes remain.**
5. **Root the tree:** If `add_root` is `True`, the solver adds an implicit root to the tree.
### `get_dissimilarity_map`
#### Description
This method obtains or generates the initial dissimilarity map used by the solver. It calls the parent class's `get_dissimilarity_map` method and then constructs the initial lambda matrix, where each subset is a singleton leaf node.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :--------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The CassiopeiaTree object passed into `solve`. |
| layer | Optional\[str] | Layer storing the character matrix for solving. If None, the default character matrix is used. |
#### Outputs
| Name | Type | Description |
| :----------------- | :----------- | :---------------------------------------------------------------------------------------------- |
| lambda\_matrix\_df | pd.DataFrame | The initial lambda matrix, structured similarly to the `dissimilarity_map` in `DistanceSolver`. |
#### Internal Logic
1. **Obtain the similarity map:** Calls the parent class's `get_dissimilarity_map` method to get the pairwise similarities between nodes.
2. **Prepare for the lambda matrix:** Initializes the `_lambda_indices` attribute, which stores the subsets of nodes being considered. Initially, each subset is a singleton leaf node.
3. **Generate the lambda matrix:** Iterates through all pairs of subsets and calculates the second singular value of their corresponding RA matrix using the `_compute_svd2` method.
4. **Convert to DataFrame:** Converts the lambda matrix from a NumPy array to a Pandas DataFrame.
### `find_cherry`
#### Description
This method finds the pair of nodes (or subsets of nodes) to join into a cherry based on the lambda matrix. The cherry is identified as the pair with the smallest second singular value in the lambda matrix.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------- | :----------------- |
| dissimilarity\_map | np.ndarray | The lambda matrix. |
#### Outputs
| Name | Type | Description |
| :----- | :--------------- | :----------------------------------------------------------------------------------------------------- |
| (i, j) | Tuple\[int, int] | A tuple of indices representing the rows in the lambda matrix corresponding to the nodes to be joined. |
### `_compute_svd2`
#### Description
This method calculates the second largest singular value of the RA matrix for a given pair of subsets of nodes. The RA matrix is constructed based on the pairwise similarities between nodes in the subsets and their complements.
#### Inputs
| Name | Type | Description |
| :-------------- | :---------------- | :------------------------------------------------------------------------------------------------------- |
| pair | Tuple\[int, int] | A tuple of indices representing the rows in the lambda matrix corresponding to the two subsets of nodes. |
| lambda\_indices | List\[List\[int]] | The list of subsets of nodes being considered. |
#### Outputs
| Name | Type | Description |
| :-------- | :---- | :-------------------------------------------------------------------------------- |
| svd2\_val | float | The second largest singular value of the RA matrix for the given pair of subsets. |
#### Internal Logic
1. **Get the combined subset:** Combines the two subsets of nodes specified by the `pair` argument.
2. **Get the complement:** Identifies the nodes that are not in the combined subset.
3. **Reconstruct the RA matrix:** Constructs the RA matrix based on the pairwise similarities between nodes in the combined subset and its complement.
4. **Calculate the second largest singular value:** Computes the singular value decomposition of the RA matrix and returns the second largest singular value.
### `update_dissimilarity_map`
#### Description
This method updates the lambda matrix after a cherry is formed. It removes the rows and columns corresponding to the joined nodes and adds a new row and column for the newly formed cherry. The entries in the new row and column are calculated using the `_compute_svd2` method.
#### Inputs
| Name | Type | Description |
| :-------------- | :--------------- | :--------------------------------------------------------- |
| similarity\_map | pd.DataFrame | The lambda matrix to update. |
| cherry | Tuple\[str, str] | The names of the two nodes that were joined into a cherry. |
| new\_node | str | The name of the newly formed cherry node. |
#### Outputs
| Name | Type | Description |
| :----------------- | :----------- | :------------------------- |
| lambda\_matrix\_df | pd.DataFrame | The updated lambda matrix. |
#### Internal Logic
1. **Identify cherry nodes:** Finds the indices of the cherry nodes in the lambda matrix.
2. **Modify names and indices:** Updates the node names and `_lambda_indices` attribute to reflect the newly formed cherry.
3. **Extract the old lambda matrix:** Removes the rows and columns corresponding to the joined nodes from the lambda matrix.
4. **Add new row and column:** Adds a new row and column to the lambda matrix for the newly formed cherry.
5. **Compute new singular values:** Calculates the second singular values for the new row and column using the `_compute_svd2` method.
6. **Regenerate lambda matrix:** Constructs a new DataFrame from the updated lambda matrix.
### `setup_root_finder`
#### Description
This method defines the implicit rooting strategy for the solver. It adds an implicit root with all-zero character states to the character matrix and recomputes the dissimilarity map.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :--------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The CassiopeiaTree object passed into `solve`. |
#### Internal Logic
1. **Add implicit root:** Adds a new row to the character matrix representing the implicit root with all-zero character states.
2. **Recompute dissimilarity map:** Computes the dissimilarity map using the updated character matrix.
### `root_tree`
#### Description
This method roots the tree by adding an edge between the last two remaining nodes and converting the undirected graph to a directed graph with respect to the specified root sample.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------- | :------------------------------------------- |
| tree | nx.Graph | The undirected tree topology. |
| root\_sample | str | The name of the sample to treat as the root. |
| remaining\_samples | List\[str] | The names of the last two unjoined nodes. |
#### Outputs
| Name | Type | Description |
| :----------- | :--------- | :--------------- |
| rooted\_tree | nx.DiGraph | The rooted tree. |
## Dependencies
| Dependency | Purpose |
| :----------- | :------------------------------------------------------------------- |
| networkx | Representing and manipulating the tree topology. |
| numpy | Numerical operations and array manipulation. |
| pandas | Data manipulation and representation of the dissimilarity map. |
| scipy | Singular value decomposition calculation. |
| ete3 | Tree manipulation and visualization (used in `_ccphylo_solve`). |
| configparser | Reading configuration settings (used in `_setup_ccphylo`). |
| subprocess | Running external commands (used in `_ccphylo_solve`). |
| tempfile | Creating temporary files and directories (used in `_ccphylo_solve`). |
### Configuration
The `_ccphylo_solve` method uses the `ccphylo_path` setting from the `config.ini` file to locate the CCPhylo executable.
## Error Handling
The `DistanceSolverError` exception is raised for various error conditions, such as missing rooting parameters or invalid dissimilarity functions.
## Logging
Currently, no logging mechanisms are implemented in this code.
# SpectralSolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/SpectralSolver.py
## High-level description
The `SpectralSolver` class in `SpectralSolver.py` provides a method for reconstructing phylogenetic trees using a spectral algorithm. It leverages the similarity between samples based on shared mutations to partition them recursively, ultimately building a tree structure. This approach aims to minimize the separation of samples with shared mutations while accounting for the size of each partition.
## Code Structure
The `SpectralSolver` class inherits from the `GreedySolver` class. It primarily overrides the `perform_split` method to implement its specific partitioning logic using a spectral algorithm. The `SpectralSolver` utilizes functions from `graph_utilities` to construct and manipulate similarity graphs.
## References
This code references the following symbols:
* `GreedySolver` (from `cassiopeia.solver.GreedySolver`)
* `dissimilarity_functions` (from `cassiopeia.solver`)
* `graph_utilities` (from `cassiopeia.solver`)
## Symbols
### `SpectralSolver`
#### Description
This class implements a spectral algorithm for phylogenetic tree reconstruction. It inherits from `GreedySolver` and overrides the `perform_split` method to partition samples based on a similarity graph.
#### Inputs
| Name | Type | Description |
| :-------------------- | :---------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- |
| similarity\_function | Optional\[Callable\[\[int, int, pd.DataFrame, int, Optional\[Dict\[int, Dict\[str, float]]]], float]] | A function to calculate similarity between samples (default: `hamming_similarity_without_missing`). |
| threshold | Optional\[int] | Minimum similarity threshold for graph edges (default: 0). |
| prior\_transformation | str | Transformation function for priors (default: "negative\_log"). |
#### Outputs
This class doesn't directly return any output. It modifies the input `CassiopeiaTree` object during the `solve` method.
#### Internal Logic
1. **Initialization:** Sets the similarity threshold, similarity function, and prior transformation.
2. **`perform_split` Method:**
* Constructs a similarity graph using `graph_utilities.construct_similarity_graph`.
* Computes the normalized Laplacian matrix of the graph.
* Calculates the second eigenvector of the Laplacian matrix (Fiedler vector).
* Orders nodes based on their corresponding values in the Fiedler vector.
* Finds the optimal partition index that minimizes the normalized cut.
* Improves the initial partition using `graph_utilities.spectral_improve_cut`.
* Returns the left and right partitions.
3. **`solve` Method (inherited from `GreedySolver`):**
* Prepares data and weights.
* Recursively partitions the samples using `perform_split` until individual samples are reached.
* Constructs the tree by connecting the partitions with ancestral nodes.
* Populates the input `CassiopeiaTree` with the reconstructed tree.
#### Side Effects
The `solve` method modifies the input `CassiopeiaTree` object by populating its tree structure.
#### Performance Considerations
The spectral algorithm's performance depends on the size and structure of the similarity graph. Larger datasets might require more computational resources. The `spectral_improve_cut` function performs a greedy hill-climbing optimization, which could be computationally intensive for complex graphs.
# UPGMASolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/UPGMASolver.py
## High-level description
The `UPGMASolver` class implements the Unweighted Pair Group Method with Arithmetic Mean (UPGMA) algorithm, a hierarchical clustering method used for phylogenetic tree reconstruction. It iteratively joins the most similar samples (or clusters) based on a dissimilarity matrix, updating the matrix after each join until a single cluster remains.
## Code Structure
The `UPGMASolver` class inherits from the `DistanceSolver` class and overrides methods for finding cherries (`find_cherry`), updating the dissimilarity map (`update_dissimilarity_map`), and rooting the tree (`root_tree`). It also uses a helper function `__update_dissimilarity_map_numba` for optimized dissimilarity map updates.
## References
* `DistanceSolver`: The parent class providing the general framework for distance-based solvers.
* `dissimilarity_functions`: A module containing various dissimilarity functions, including `weighted_hamming_distance` used by default.
## Symbols
### `UPGMASolver`
#### Description
This class implements the UPGMA algorithm for phylogenetic tree reconstruction. It uses a dissimilarity matrix to iteratively join the most similar samples or clusters, updating the matrix after each join.
#### Inputs
| Name | Type | Description |
| :---------------------- | :------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------ |
| dissimilarity\_function | Optional\[Callable\[\[np.array, np.array, int, Dict\[int, Dict\[int, float]]], float]] | A function to compute the dissimilarity between samples. Defaults to `weighted_hamming_distance`. |
| prior\_transformation | str | Function to transform priors into weights. Defaults to "negative\_log". |
| fast | bool | Whether to use a fast implementation of UPGMA. Defaults to False. |
| implementation | str | Which fast implementation to use. Defaults to "ccphylo\_upgma". |
| threads | int | Number of threads to use for dissimilarity map computation. Defaults to 1. |
#### Outputs
The `UPGMASolver` object itself, which can be used to solve a `CassiopeiaTree` object.
#### Internal Logic
The `solve` method performs the following steps:
1. Initializes a dissimilarity map and a tree with all samples as leaves.
2. Iteratively finds the most similar pair of samples or clusters (cherry) using `find_cherry`.
3. Joins the cherry into a new node in the tree.
4. Updates the dissimilarity map using `update_dissimilarity_map`.
5. Repeats steps 2-4 until only two clusters remain.
6. Roots the tree using `root_tree`.
7. Populates the input `CassiopeiaTree` with the resulting tree.
### `root_tree`
#### Description
Roots the tree produced by UPGMA by adding a root node as the parent of the last two unjoined nodes.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------- | :------------------------------------------ |
| tree | nx.Graph | The unrooted tree topology. |
| root\_sample | str | Ignored, as the root is implicitly defined. |
| remaining\_samples | List\[str] | The last two unjoined nodes in the tree. |
#### Outputs
| Name | Type | Description |
| :----------- | :--------- | :--------------- |
| rooted\_tree | nx.DiGraph | The rooted tree. |
#### Internal Logic
1. Adds a "root" node to the tree.
2. Connects the "root" node to the two remaining samples.
3. Converts the undirected graph to a directed graph rooted at the "root" node.
### `find_cherry`
#### Description
Finds the pair of samples or clusters with the minimum dissimilarity in the dissimilarity matrix.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------- | :------------------------ |
| dissimilarity\_matrix | np.array | The dissimilarity matrix. |
#### Outputs
| Name | Type | Description |
| :----- | :--------------- | :---------------------------------------------------------------------- |
| (i, j) | Tuple\[int, int] | A tuple of indices in the dissimilarity matrix representing the cherry. |
#### Internal Logic
1. Sets the diagonal of the dissimilarity matrix to infinity to avoid self-comparisons.
2. Finds the indices of the minimum value in the matrix.
### `update_dissimilarity_map`
#### Description
Updates the dissimilarity map after joining two nodes into a cherry. The new dissimilarity between the cherry and other nodes is calculated as a weighted average of the dissimilarities between the cherry's children and the other nodes.
#### Inputs
| Name | Type | Description |
| :----------------- | :--------------- | :------------------------------------------------ |
| dissimilarity\_map | pd.DataFrame | The dissimilarity map to update. |
| cherry | Tuple\[str, str] | A tuple of node names representing the cherry. |
| new\_node | str | The name of the new node representing the cherry. |
#### Outputs
| Name | Type | Description |
| :----------------- | :----------- | :----------------------------- |
| dissimilarity\_map | pd.DataFrame | The updated dissimilarity map. |
#### Internal Logic
1. Calculates the size of each child cluster in the cherry.
2. Updates the cluster size dictionary with the new node's size.
3. Finds the indices of the cherry nodes in the dissimilarity map.
4. Calls the optimized `__update_dissimilarity_map_numba` function to update the dissimilarity map.
5. Creates a new DataFrame with the updated dissimilarity map and new node.
6. Removes the cherry nodes from the dissimilarity map.
### `__update_dissimilarity_map_numba`
#### Description
A private, optimized function for updating the dissimilarity map using Numba for performance improvement.
#### Inputs
| Name | Type | Description |
| :----------------- | :------- | :--------------------------------------------- |
| dissimilarity\_map | np.array | The dissimilarity matrix to update. |
| cherry\_i | int | Index of the first cherry node in the matrix. |
| cherry\_j | int | Index of the second cherry node in the matrix. |
| size\_i | int | Size of the first cherry cluster. |
| size\_j | int | Size of the second cherry cluster. |
#### Outputs
| Name | Type | Description |
| :----------- | :------- | :-------------------------------- |
| updated\_map | np.array | The updated dissimilarity matrix. |
#### Internal Logic
1. Adds a new row and column for the new node.
2. Calculates the new dissimilarities between the new node and all other nodes using a weighted average.
3. Sets the self-dissimilarity of the new node to 0.
### `setup_root_finder`
#### Description
Defines the implicit rooting strategy for UPGMA. Since UPGMA produces a rooted tree, this method simply sets the root sample name to "root".
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The input CassiopeiaTree object. |
#### Outputs
None. The method modifies the `root_sample_name` attribute of the input `CassiopeiaTree` object.
## Side Effects
The `solve` method modifies the input `CassiopeiaTree` object by populating its `tree` attribute with the reconstructed tree.
## Dependencies
* `numba`: Used for optimizing the `__update_dissimilarity_map_numba` function.
* `networkx`: Used for representing and manipulating the tree topology.
* `pandas`: Used for representing the dissimilarity matrix.
* `numpy`: Used for numerical operations.
## Error Handling
The `UPGMASolver` class raises `DistanceSolverError` if an invalid fast implementation is specified or if rooting parameters are incorrect.
# VanillaGreedySolver.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/VanillaGreedySolver.py
## High-level description
The `VanillaGreedySolver` class in `VanillaGreedySolver.py` implements a top-down greedy algorithm for reconstructing phylogenetic trees. It recursively splits a set of samples into mutually exclusive groups based on the presence or absence of the most frequently occurring mutation, optimizing for parsimony. The solver handles missing data using various imputation methods and allows for weighting character-state pairs based on provided priors.
## Code Structure
The `VanillaGreedySolver` class inherits from the `GreedySolver` class. It primarily implements the `perform_split` method, which determines the optimal split based on mutation frequencies. The `solve` method, inherited from `GreedySolver`, orchestrates the tree building process by recursively calling `perform_split`. The `compute_mutation_frequencies` method calculates the frequencies of each character-state pair, considering ambiguous states. The `__add_duplicates_to_tree` method, also inherited, handles the reinsertion of duplicate samples into the tree after the main solving process.
## References
* `GreedySolver`: The parent class defining the general structure of greedy solvers.
* `missing_data_methods`: A module containing various missing data imputation methods.
* `solver_utilities`: A module providing utility functions for tree solvers.
## Symbols
### `VanillaGreedySolver`
#### Description
This class implements the vanilla Cassiopeia-Greedy algorithm for phylogenetic tree reconstruction. It identifies the most frequent mutation and splits the sample set based on the presence or absence of that mutation.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------- | :-------------------------------------------------------------------------------------------------------------------- |
| missing\_data\_classifier | Callable | A function to handle missing data during character splits. Defaults to `missing_data_methods.assign_missing_average`. |
| prior\_transformation | str | The method for transforming priors into weights. Defaults to "negative\_log". |
#### Outputs
None
#### Internal Logic
The `perform_split` method calculates mutation frequencies and identifies the most frequent mutation. It then partitions the samples based on the presence or absence of this mutation, using the specified `missing_data_classifier` to handle missing data.
### `perform_split`
#### Description
This method partitions a set of samples based on the most frequent (character, state) pair in the character matrix.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :------------------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | The character matrix representing the mutation profiles of the samples. |
| samples | List\[int] | A list of sample indices to be partitioned. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | A dictionary of weights for each (character, state) pair, derived from priors. |
| missing\_state\_indicator | int | The character representing missing data in the character matrix. Defaults to -1. |
#### Outputs
| Name | Type | Description |
| :--------- | :--------- | :------------------------------------------------------------------------------------------------ |
| left\_set | List\[str] | The list of samples belonging to the left partition (possessing the most frequent mutation). |
| right\_set | List\[str] | The list of samples belonging to the right partition (not possessing the most frequent mutation). |
#### Internal Logic
1. Calculate mutation frequencies using `compute_mutation_frequencies`.
2. Identify the (character, state) pair with the highest frequency, considering weights if provided.
3. Partition the samples into `left_set` and `right_set` based on the presence or absence of the chosen mutation.
4. Use the `missing_data_classifier` to assign samples with missing data at the chosen character to either partition.
## Error Handling
The code raises a `GreedySolverError` if the input character matrix contains ambiguous states and the solver is not configured to handle them (`self.allow_ambiguous` is False).
## Dependencies
* pandas
* networkx
* ete3
* numpy
## TODOs
None specified in the code.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/__init__.py
## High-level description
This file defines the top-level module for the Cassiopeia tree solver package. It imports and exposes various solver classes and dissimilarity functions for phylogenetic tree reconstruction.
## Code Structure
This file simply imports various solver classes and a module containing dissimilarity functions. These classes and functions are used by other modules in the Cassiopeia solver package.
## Symbols
### `HybridSolver`
#### Description
A hybrid solver that combines a top-down greedy algorithm with a more complex bottom-up solver. The greedy solver is applied until a certain criteria is reached (e.g., maximum LCA distance or number of cells), after which the bottom solver is used to reconstruct subproblems.
### `ILPSolver`
#### Description
An ILP-based solver that infers the maximum parsimony tree by solving for a Steiner Tree over a potential graph of possible evolutionary states.
### `MaxCutGreedySolver`
#### Description
A greedy solver that uses the max-cut criterion to optimize partitions generated by the vanilla greedy algorithm.
### `MaxCutSolver`
#### Description
A solver that uses a max-cut heuristic to partition samples based on a connectivity graph representing a supertree of phylogenetic trees on each individual character.
### `NeighborJoiningSolver`
#### Description
A solver that implements the Neighbor-Joining algorithm to iteratively join samples that minimize the Q-criterion on the dissimilarity map.
### `PercolationSolver`
#### Description
A top-down percolation-based solver that recursively partitions the sample set based on similarity in the observed mutations.
### `SharedMutationJoiningSolver`
#### Description
An agglomerative clustering solver that joins samples that share the most identical character/state mutations.
### `SpectralGreedySolver`
#### Description
A greedy solver that uses a spectral heuristic to optimize partitions generated by the vanilla greedy algorithm.
### `SpectralSolver`
#### Description
A solver that uses a spectral method to partition samples based on a similarity graph representing shared mutations.
### `UPGMASolver`
#### Description
A solver that implements the UPGMA algorithm to iteratively join samples with the minimum dissimilarity.
### `VanillaGreedySolver`
#### Description
A solver that implements the basic Cassiopeia-Greedy algorithm, recursively splitting samples based on the most frequent mutation.
### `SpectralNeighborJoiningSolver`
#### Description
A solver that implements a variation of the Spectral Neighbor-Joining algorithm to iteratively join subsets of leaves into a cherry based on the second singular value of the RA matrix.
### `dissimilarity`
#### Description
A module containing various dissimilarity functions used to compare phylogenetic samples.
# dissimilarity_functions.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/dissimilarity_functions.py
## High-level description
This file provides a collection of dissimilarity and similarity functions used for comparing phylogenetic samples. These functions calculate the distance or similarity between pairs of samples based on their character states, considering factors like missing data, state weights, and linkage methods.
## Code Structure
This code defines several functions for calculating dissimilarity and similarity between phylogenetic samples. Some functions are optimized using the `numba` library. The `cluster_dissimilarity` function utilizes other dissimilarity functions and linkage functions to handle ambiguous character strings.
## Symbols
### `weighted_hamming_distance`
#### Description
Calculates the weighted Hamming distance between two phylogenetic samples. It considers shared indel states and their probabilities, incrementing dissimilarity for differing states and decrementing for identical states based on their weights.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :------------------------------------------------------------------------------------------------------ |
| s1 | List\[int] | Character states of the first sample |
| s2 | List\[int] | Character states of the second sample |
| missing\_state\_indicator | int | The character representing missing values (default: -1) |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | A nested dictionary storing state weights for each character, derived from state priors (default: None) |
#### Outputs
| Name | Type | Description |
| :------- | :---- | :---------------------------------------------------- |
| distance | float | The weighted Hamming distance between the two samples |
#### Internal Logic
The function iterates through each character of the samples. If both states are identical and not missing, it decrements the dissimilarity by the probability of them occurring independently. If the states disagree, it increments the dissimilarity by the probability of those states occurring. The final dissimilarity is normalized by the number of non-missing characters shared by the samples.
### `hamming_similarity_without_missing`
#### Description
Calculates the number of shared (non-missing) character/state mutations between two samples.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------------------------------------------------- |
| s1 | List\[int] | Character states of the first sample |
| s2 | List\[int] | Character states of the second sample |
| missing\_state\_indicator | int | The character representing missing values |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights to weight the similarity of a mutation (default: None) |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :----------------------------------------------------- |
| similarity | float | The number of shared mutations, weighted or unweighted |
#### Internal Logic
The function iterates through each character, ignoring missing states and uncut states (state 0). For each matching state, it increments the similarity by 1 or by the corresponding weight if provided.
### `hamming_similarity_normalized_over_missing`
#### Description
Calculates the number of shared (non-missing) character/state mutations between two samples, normalized by the number of missing data events.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------------------------------------------------- |
| s1 | List\[int] | Character states of the first sample |
| s2 | List\[int] | Character states of the second sample |
| missing\_state\_indicator | int | The character representing missing values |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights to weight the similarity of a mutation (default: None) |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :---------------------------------------------------------------- |
| similarity | float | The normalized number of shared mutations, weighted or unweighted |
#### Internal Logic
Similar to `hamming_similarity_without_missing`, but it also tracks the number of non-missing characters (`num_present`). The final similarity score is divided by `num_present` to normalize for missing data.
### `hamming_distance`
#### Description
Calculates the vanilla Hamming distance between two samples, optionally ignoring missing data.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------ | :------------------------------------------------------------------ |
| s1 | np.array(int) | The first sample |
| s2 | np.array(int) | The second sample |
| ignore\_missing\_state | bool | Whether to ignore missing states in the comparison (default: False) |
| missing\_state\_indicator | int | The character representing missing values (default: -1) |
#### Outputs
| Name | Type | Description |
| :------- | :--- | :------------------------------------------- |
| distance | int | The Hamming distance between the two samples |
#### Internal Logic
The function iterates through each character, incrementing the distance for each mismatch unless both states are missing and `ignore_missing_state` is True.
### `weighted_hamming_similarity`
#### Description
Calculates the weighted number of shared (non-missing) character/state mutations between two samples.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------------------------------------------------- |
| s1 | List\[int] | Character states of the first sample |
| s2 | List\[int] | Character states of the second sample |
| missing\_state\_indicator | int | The character representing missing values |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights to weight the similarity of a mutation (default: None) |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :-------------------------------------- |
| similarity | float | The weighted number of shared mutations |
#### Internal Logic
The function iterates through each character, ignoring missing states. For each matching state, it increments the similarity by 2 times the corresponding weight (or 2 if no weights are provided) if the state is not 0, and by 1 if the state is 0 and no weights are provided. The final similarity is normalized by the number of non-missing characters.
### `exponential_negative_hamming_distance`
#### Description
Calculates a similarity score based on the inverse of the weighted Hamming distance. It uses the formula exp(-d(i,j)), where d is the weighted Hamming distance.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :--------------------------------------------------------------------------- |
| s1 | List\[int] | Character states of the first sample |
| s2 | List\[int] | Character states of the second sample |
| missing\_state\_indicator | int | The character representing missing values (default: -1) |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | A nested dictionary storing state weights for each character (default: None) |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :------------------- |
| similarity | float | The similarity score |
#### Internal Logic
The function first calculates the weighted Hamming distance (`weighted_hamm_dist`) using the same logic as `weighted_hamming_distance`. Then, it returns the exponential of the negative of this distance.
### `cluster_dissimilarity`
#### Description
Calculates the dissimilarity between two potentially ambiguous character strings. An ambiguous character string can have multiple possible states for each character.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------- |
| dissimilarity\_function | Callable\[\[List\[int], List\[int], int, Dict\[int, Dict\[int, float]]], float] | The dissimilarity function to use for pairwise comparisons |
| s1 | Union\[List\[int], List\[Tuple\[int, ...]]] | The first (possibly ambiguous) sample |
| s2 | Union\[List\[int], List\[Tuple\[int, ...]]] | The second (possibly ambiguous) sample |
| missing\_state\_indicator | int | The character representing missing values |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for the dissimilarity function (default: None) |
| linkage\_function | Callable\[\[Union\[np.array, List\[float]]], float] | The linkage function to aggregate dissimilarities (default: np.mean) |
| normalize | bool | Whether to normalize the dissimilarity by the proportion of shared non-missing positions (default: True) |
#### Outputs
| Name | Type | Description |
| :------------ | :---- | :---------------------------------------- |
| dissimilarity | float | The dissimilarity between the two samples |
#### Internal Logic
The function iterates through each character of the input strings. If a character is ambiguous (represented as a tuple), it calculates the dissimilarity for all possible combinations of states using the provided `dissimilarity_function` and aggregates them using the `linkage_function`. The final dissimilarity is the sum of individual character dissimilarities, optionally normalized by the number of shared non-missing positions.
### `cluster_dissimilarity_weighted_hamming_distance_min_linkage`
#### Description
Calculates the dissimilarity between two potentially ambiguous character strings using the weighted Hamming distance and minimum linkage.
#### Inputs
| Name | Type | Description |
| :------------------------ | :------------------------------------------ | :----------------------------------------------------------------- |
| s1 | Union\[List\[int], List\[Tuple\[int, ...]]] | The first (possibly ambiguous) sample |
| s2 | Union\[List\[int], List\[Tuple\[int, ...]]] | The second (possibly ambiguous) sample |
| missing\_state\_indicator | int | The character representing missing values |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for the weighted Hamming distance (default: None) |
#### Outputs
| Name | Type | Description |
| :------------ | :---- | :---------------------------------------- |
| dissimilarity | float | The dissimilarity between the two samples |
#### Internal Logic
This function is similar to `cluster_dissimilarity` but specifically uses the weighted Hamming distance as the dissimilarity function and minimum linkage (np.min) to aggregate dissimilarities. It iterates through each character, calculates the weighted Hamming distance for all possible state combinations if the character is ambiguous, and takes the minimum distance. The final dissimilarity is normalized by the number of shared non-missing positions.
## TODOs
* `hamming_similarity_without_missing`: Optimize this using masks
* `hamming_similarity_normalized_over_missing`: Optimize this using masks
# graph_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/graph_utilities.py
## High-level description
This code defines functions for constructing and manipulating graphs used in phylogenetic tree reconstruction. It includes functions for building connectivity graphs based on shared mutations, similarity graphs based on a given similarity function, and optimizing graph partitions using hill-climbing procedures.
## Code Structure
The code consists of several functions that are used to construct and manipulate graphs for phylogenetic tree reconstruction. The `construct_connectivity_graph` and `construct_similarity_graph` functions are used to build different types of graphs, while `max_cut_improve_cut` and `spectral_improve_cut` are used to optimize partitions on these graphs. The `check_if_cut` function is a utility function used by the optimization functions.
## References
This code references the `solver_utilities` module for converting sample names to indices.
## Symbols
### `check_if_cut`
#### Description
This function checks if two nodes are on opposite sides of a graph partition.
#### Inputs
| Name | Type | Description |
| :--- | :--------- | :------------------------------------------------------------- |
| u | int | The index of the first node. |
| v | int | The index of the second node. |
| cut | List\[int] | A list of node indices representing one side of the partition. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :----------------------------------------------------------------------------- |
| | bool | True if nodes u and v are on opposite sides of the partition, False otherwise. |
#### Internal Logic
The function simply checks if one node is in the `cut` list and the other is not.
### `construct_connectivity_graph`
#### Description
This function constructs a connectivity graph for a set of samples based on their observed mutations. The edge weights represent the tendency of two samples to be grouped together or separated based on their shared and differing mutations.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :-------------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | The character matrix of observed character states for all samples. |
| mutation\_frequencies | Dict\[int, Dict\[int, int]] | A dictionary containing the frequencies of each character/state pair. |
| missing\_state\_indicator | int | The character representing missing values. |
| samples | List\[str] | A list of sample names to include in the graph. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for character/state pairs. |
#### Outputs
| Name | Type | Description |
| :------- | :------- | :------------------------------------------------------------- |
| G\_names | nx.Graph | The constructed connectivity graph with sample names as nodes. |
#### Internal Logic
The function iterates through all pairs of samples and calculates a score based on their shared and differing mutations. The score is positive if the samples share mutations and negative if they have different mutations. The absolute value of the score is higher for mutations that are less frequent in the dataset. The scores are then used as edge weights in the graph.
### `max_cut_improve_cut`
#### Description
This function implements a greedy hill-climbing procedure to optimize a partition for the max-cut criterion on a graph.
#### Inputs
| Name | Type | Description |
| :--- | :--------- | :-------------------------------------------------------------- |
| G | nx.Graph | The graph to optimize the partition on. |
| cut | List\[str] | A list of nodes representing one side of the initial partition. |
#### Outputs
| Name | Type | Description |
| :------- | :--------- | :---------------------------------------------------------------- |
| new\_cut | List\[str] | A new partition that is a local maximum to the max-cut criterion. |
#### Internal Logic
The function iteratively moves nodes between the two sides of the partition, choosing the move that maximizes the weight of edges across the cut. The procedure terminates when no further improvement can be made or a maximum number of iterations is reached.
### `construct_similarity_graph`
#### Description
This function constructs a similarity graph for a set of samples based on a given similarity function.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :----------------------------------------------------------------- |
| character\_matrix | pd.DataFrame | The character matrix of observed character states for all samples. |
| missing\_state\_indicator | int | The character representing missing values. |
| samples | List\[str] | A list of sample names to include in the graph. |
| similarity\_function | Callable | A function that calculates a similarity score between two samples. |
| threshold | int | A minimum similarity threshold to include an edge in the graph. |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for character/state pairs. |
#### Outputs
| Name | Type | Description |
| :------- | :------- | :----------------------------------------------------------- |
| G\_names | nx.Graph | The constructed similarity graph with sample names as nodes. |
#### Internal Logic
The function iterates through all pairs of samples and calculates their similarity using the provided `similarity_function`. If the similarity score is above the given `threshold`, an edge is added between the corresponding nodes in the graph with the weight equal to the similarity score.
### `spectral_improve_cut`
#### Description
This function implements a greedy hill-climbing procedure to optimize a partition for a modified normalized cut criterion on a graph.
#### Inputs
| Name | Type | Description |
| :--- | :--------- | :-------------------------------------------------------------- |
| G | nx.Graph | The graph to optimize the partition on. |
| cut | List\[str] | A list of nodes representing one side of the initial partition. |
#### Outputs
| Name | Type | Description |
| :------- | :--------- | :----------------------------------------------------------------- |
| new\_cut | List\[str] | A new partition that is a local minimum to the objective function. |
#### Internal Logic
The function iteratively moves nodes between the two sides of the partition, choosing the move that minimizes the ratio of the weight of edges across the cut to the minimum of the sum of edge weights within each side of the cut. The procedure terminates when no further improvement can be made or a maximum number of iterations is reached.
# missing_data_methods.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/missing_data_methods.py
## High-level description
The `assign_missing_average` function is a missing data imputation method used in the Cassiopeia-Greedy algorithm. It aims to assign samples with missing data at a particular character to the most similar partition (left or right) based on the average number of shared mutations. This function is particularly useful for handling ambiguous character states, such as those arising from uncertainties in single-cell sequencing data.
## Symbols
### `assign_missing_average`
#### Description
This function takes a character matrix, partition information, and a list of samples with missing data. It calculates a score for each missing sample against both the left and right partitions, based on the average number of shared mutations. The sample is then assigned to the partition with the higher score.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------------------------------------------------------------- |
| `character_matrix` | pandas.DataFrame | The character matrix containing observed character states for all samples. |
| `missing_state_indicator` | int | The character representing missing values in the character matrix. |
| `left_set` | List\[str] | A list of sample names currently assigned to the left partition. |
| `right_set` | List\[str] | A list of sample names currently assigned to the right partition. |
| `missing` | List\[str] | A list of sample names with missing data at the current character being considered. |
| `weights` | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for character/state mutation pairs, derived from priors. |
#### Outputs
| Name | Type | Description |
| :---------- | :--------- | :-------------------------------------------------------- |
| `left_set` | List\[str] | The updated left partition with imputed missing samples. |
| `right_set` | List\[str] | The updated right partition with imputed missing samples. |
#### Internal Logic
1. **Convert sample names to indices:** For efficient indexing, the function first converts the sample names in `left_set`, `right_set`, and `missing` to their corresponding integer indices in the `character_matrix`.
2. **Define `score_side` helper function:** This function calculates the similarity score between a missing sample and a given partition (left or right). It iterates through each character, comparing the states of the missing sample to the states present in the partition. The score is calculated based on the number of shared states, optionally weighted by the provided `weights`.
3. **Calculate scores for each missing sample:** For each sample in `missing`, the function calculates its similarity score to both the left and right partitions using the `score_side` function.
4. **Assign missing samples based on scores:** The missing sample is assigned to the partition with the higher average similarity score. The average score is calculated by dividing the total score by the number of samples in the respective partition.
5. **Return updated partitions:** The function returns the updated `left_set` and `right_set` lists, now including the imputed missing samples.
## References
* `solver_utilities.convert_sample_names_to_indices`: Used to convert sample names to their corresponding indices in the character matrix.
* `unravel_ambiguous_states`: Imported from `cassiopeia.mixins` and used to handle ambiguous character states, expanding them into a list of all possible states.
## Side Effects
This function modifies the input `left_set` and `right_set` lists by adding the imputed missing samples to the appropriate partition.
#### Performance Considerations
The function first converts sample names to indices to improve the efficiency of accessing elements in the character matrix. The use of NumPy arrays for storing and comparing character states further enhances performance.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/overview
## High-level description
The `solver` directory contains implementations of various phylogenetic tree reconstruction algorithms used in the Cassiopeia framework. It includes different solver classes that employ diverse strategies for inferring evolutionary relationships between samples based on their mutation profiles.
## What does it do?
This directory provides a collection of algorithms for reconstructing phylogenetic trees from character matrices representing mutation data. The solvers use different approaches, including:
1. Greedy algorithms (e.g., VanillaGreedySolver, MaxCutGreedySolver)
2. Distance-based methods (e.g., NeighborJoiningSolver, UPGMASolver)
3. Spectral methods (e.g., SpectralSolver, SpectralNeighborJoiningSolver)
4. Integer Linear Programming (ILPSolver)
5. Hybrid approaches (HybridSolver)
These solvers take as input a character matrix representing mutation profiles of samples and optional prior probabilities. They output a reconstructed phylogenetic tree showing the evolutionary relationships between the samples.
## Entry points
The main entry point for using these solvers is through the `solve` method implemented by each solver class. This method takes a `CassiopeiaTree` object containing the character matrix and other relevant data, and populates it with the reconstructed tree.
The `__init__.py` file serves as the main interface for importing the various solver classes and utility functions.
## Key Files
1. `CassiopeiaSolver.py`: Defines the abstract base class for all solvers.
2. `DistanceSolver.py`: Implements the base class for distance-based solvers.
3. `GreedySolver.py`: Provides the framework for greedy top-down solvers.
4. `HybridSolver.py`: Implements a hybrid approach combining top-down and bottom-up strategies.
5. `ILPSolver.py`: Implements the Integer Linear Programming-based solver.
6. `NeighborJoiningSolver.py` and `UPGMASolver.py`: Implement specific distance-based algorithms.
7. `SpectralSolver.py` and related files: Implement spectral methods for tree reconstruction.
8. `VanillaGreedySolver.py` and `MaxCutGreedySolver.py`: Implement specific greedy algorithms.
9. `dissimilarity_functions.py`: Provides various functions for calculating dissimilarity between samples.
10. `graph_utilities.py`: Offers utility functions for graph construction and manipulation used in some solvers.
11. `solver_utilities.py`: Contains general utility functions used across different solvers.
## Dependencies
The solver module relies on several external libraries:
* networkx: For graph representation and manipulation
* numpy and pandas: For numerical operations and data handling
* scipy: For scientific computing and optimization
* ete3: For tree manipulation and visualization
* gurobipy: Used in the ILPSolver for solving optimization problems
## Configuration
Some solvers, particularly the `ILPSolver` and fast implementations of distance-based solvers, may require additional configuration:
* The `ILPSolver` uses Gurobi, which requires a separate license and installation.
* Fast implementations of some distance-based solvers (e.g., NeighborJoiningSolver) use CCPhylo, which needs to be installed and configured separately.
These configurations are typically specified in a `config.ini` file, which is read by the relevant solver classes.
The solvers also support various parameters for customization, such as dissimilarity functions, prior transformations, and algorithm-specific thresholds. These can be set when initializing the solver objects.
In summary, the `solver` directory provides a comprehensive toolkit for phylogenetic tree reconstruction, offering multiple algorithms and utilities to handle various types of mutation data and reconstruction scenarios.
# solver_utilities.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/solver/solver_utilities.py
## High-level description
This file provides utility functions for the solver module in Cassiopeia. These functions are used for tasks such as generating unique node names, collapsing unifurcations in a tree, transforming prior probabilities into weights, and converting sample names to indices.
## Code Structure
The file contains several independent functions that are used by other modules in the solver package. These functions are not directly interconnected but serve as helper functions for various solver algorithms.
## References
This file references the `ete3`, `numpy`, `pandas`, and `time` libraries. It also references the `PriorTransformationError` exception from the `cassiopeia.mixins` module.
## Symbols
### `node_name_generator`
#### Description
This function creates a generator that yields unique node names by hashing timestamps. This is used to ensure that each node in the reconstructed tree has a unique identifier.
#### Inputs
None
#### Outputs
| Name | Type | Description |
| :-------- | :-------------------------- | :----------------------------------------------- |
| generator | Generator\[str, None, None] | A generator object that yields unique node names |
#### Internal Logic
The function uses the `blake2b` hash function from the `hashlib` library to generate a 12-byte hash of the current timestamp. This hash is then converted to a hexadecimal string and appended to the prefix "cassiopeia\_internal\_node" to create a unique node name.
### `collapse_unifurcations`
#### Description
This function collapses all unifurcations in a given `ete3.Tree`. Unifurcations are nodes with only one child. Collapsing them simplifies the tree structure by removing unnecessary nodes.
#### Inputs
| Name | Type | Description |
| :--- | :-------- | :------------------------------------ |
| tree | ete3.Tree | The tree to collapse unifurcations in |
#### Outputs
| Name | Type | Description |
| :-------------- | :-------- | :-------------------------------------------------------- |
| collapsed\_tree | ete3.Tree | A copy of the input tree with all unifurcations collapsed |
#### Internal Logic
The function iterates through all nodes in the tree and identifies those with only one child. For each unifurcation, it removes the node and connects its child directly to its parent.
### `transform_priors`
#### Description
This function transforms prior probabilities into weights for use in greedy solver algorithms. It supports three transformations: negative log, inverse, and square root inverse.
#### Inputs
| Name | Type | Description |
| :-------------------- | :--------------------------------------- | :--------------------------------------------------------------------- |
| priors | Optional\[Dict\[int, Dict\[int, float]]] | A dictionary of prior probabilities for each character/state pair |
| prior\_transformation | str | The transformation to apply to the priors. Defaults to "negative\_log" |
#### Outputs
| Name | Type | Description |
| :------ | :---------------------------- | :---------------------------------------------------- |
| weights | Dict\[int, Dict\[int, float]] | A dictionary of weights for each character/state pair |
#### Internal Logic
The function first checks if the specified transformation is supported. Then, it applies the chosen transformation to each prior probability and stores the resulting weight in a new dictionary.
### `convert_sample_names_to_indices`
#### Description
This function maps sample names to their corresponding integer indices in a given list of names. This is used for efficient indexing operations in the solver algorithms.
#### Inputs
| Name | Type | Description |
| :------ | :--------- | :----------------------------------------------- |
| names | List\[str] | A list of sample names |
| samples | List\[str] | A subset of sample names to be mapped to indices |
#### Outputs
| Name | Type | Description |
| :------ | :--------- | :----------------------------------------------------------- |
| indices | List\[int] | A list of integer indices corresponding to the input samples |
#### Internal Logic
The function creates a dictionary mapping each sample name to its index in the `names` list. Then, it uses this dictionary to map the input `samples` to their corresponding indices.
### `save_dissimilarity_as_phylip`
#### Description
This function saves a dissimilarity map as a PHYLIP file. The PHYLIP format is a standard format for representing phylogenetic trees and distance matrices.
#### Inputs
| Name | Type | Description |
| :----------------- | :----------- | :------------------------------- |
| dissimilarity\_map | pd.DataFrame | A dissimilarity map |
| path | str | The path to save the PHYLIP file |
#### Outputs
None
#### Internal Logic
The function converts the dissimilarity map to a NumPy array and writes it to a file in the PHYLIP format. The file includes the number of samples and a lower triangular matrix of dissimilarities.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/__init__.py
## High-level description
This file serves as the top level for the `tools` module in the Cassiopeia library. It imports and exposes various functions and classes related to tree analysis, parameter estimation, and phylogenetic reconstruction. This module provides tools for tasks such as computing tree metrics, estimating mutation rates, and performing small parsimony analysis.
## References
This file imports and re-exposes symbols from other modules within the `cassiopeia.tools` package. These modules are:
* `cassiopeia.tools.autocorrelation`
* `cassiopeia.tools.branch_length_estimator`
* `cassiopeia.tools.coupling`
* `cassiopeia.tools.fitness_estimator`
* `cassiopeia.tools.parameter_estimators`
* `cassiopeia.tools.small_parsimony`
* `cassiopeia.tools.topology`
* `cassiopeia.tools.tree_metrics`
## Symbols
This file does not define any new symbols, but it re-exports several symbols imported from other modules. The re-exported symbols are listed in the `__all__` list. For detailed documentation of each symbol, please refer to the respective module where it is defined.
# autocorrelation.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/autocorrelation.py
Here's a documentation for the target file `cassiopeia/tools/autocorrelation.py`:
## High-level description
This file provides utility functions for computing autocorrelation statistics on trees, specifically the Moran's I statistic. It is designed to work with CassiopeiaTree objects and can handle both single and multiple variables for autocorrelation analysis.
## Symbols
### `compute_morans_i`
#### Description
Computes Moran's I statistic for a given CassiopeiaTree. This statistic measures the spatial autocorrelation of numerical data associated with the leaves of the tree.
#### Inputs
| Name | Type | Description |
| :------------------ | :-------------------------------------- | :------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree structure on which to compute the statistic |
| meta\_columns | Optional\[List] | Columns in the tree's cell\_meta object to use for computation |
| X | Optional\[pd.DataFrame] | Extra data matrix for computing autocorrelations |
| W | Optional\[pd.DataFrame] | Phylogenetic weight matrix |
| inverse\_weight\_fn | Callable\[\[Union\[int, float]], float] | Function to apply to weights if W is not provided |
#### Outputs
| Name | Type | Description |
| :--- | :-------------------------- | :--------------------- |
| I | Union\[float, pd.DataFrame] | Moran's I statistic(s) |
#### Internal Logic
1. Validates input data and combines meta\_columns and X if both are provided.
2. Ensures all data is numerical.
3. Computes or uses the provided weight matrix W.
4. Normalizes W and standardizes X.
5. Computes Moran's I using the formula: I = X' \* Wn \* X.
## Dependencies
| Dependency | Purpose |
| :---------------- | :--------------------------------- |
| numpy | Numerical computations |
| pandas | Data manipulation |
| cassiopeia.data | CassiopeiaTree class and utilities |
| cassiopeia.mixins | Custom error classes |
## Error Handling
The function raises `AutocorrelationError` in the following cases:
* No data is specified for computing autocorrelations.
* The provided X dataframe doesn't match the tree's leaves.
* Non-numeric data is found in the specified columns.
* The weight matrix doesn't match the tree's leaves.
## Performance Considerations
The computation of Moran's I involves matrix multiplications, which can be computationally expensive for large trees or many variables. The function uses numpy and pandas for efficient calculations, but performance may degrade for very large datasets.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# BranchLengthEstimator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/BranchLengthEstimator.py
## High-level description
This code defines an abstract base class `BranchLengthEstimator` for branch length estimation algorithms in the Cassiopeia framework. It serves as a template for concrete branch length estimation algorithms, requiring them to implement the `estimate_branch_lengths` method.
## Code Structure
The `BranchLengthEstimator` class is an abstract base class defined using the `abc` module. It has a single abstract method, `estimate_branch_lengths`, which needs to be implemented by concrete subclasses.
## Symbols
### Symbol Name: `BranchLengthEstimator`
#### Description
This is an abstract base class for all branch length estimation algorithms. It defines a common interface for these algorithms, ensuring that they all have an `estimate_branch_lengths` method.
#### Inputs
The class itself does not take any inputs during initialization.
#### Outputs
The class itself does not return any outputs.
#### Internal Logic
This is an abstract base class and does not have any internal logic.
### Symbol Name: `estimate_branch_lengths`
#### Description
This is an abstract method that needs to be implemented by subclasses. It should estimate the branch lengths of a given tree.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :------------------------------------------------------------------------------------ |
| tree | CassiopeiaTree | CassiopeiaTree object storing character information and an initialized tree topology. |
#### Outputs
This method does not return any values. It should modify the input `tree` object in-place, updating the branch lengths.
#### Internal Logic
The internal logic of this method is specific to the concrete subclass that implements it.
# IIDExponentialBayesian.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/IIDExponentialBayesian.py
## High-level description
The `IIDExponentialBayesian` class in `cassiopeia/tools/branch_length_estimator/IIDExponentialBayesian.py` implements a Bayesian method for estimating branch lengths in a phylogenetic tree, assuming a subsampled Birth Process for the phylogeny and independent, identically distributed (IID) exponential waiting times for mutations at each site. The estimator calculates the posterior mean branch lengths conditional on the observed tree topology and character data.
## Code Structure
The `IIDExponentialBayesian` class inherits from the abstract base class `BranchLengthEstimator` and implements the `estimate_branch_lengths` method. The core logic resides in `_populate_attributes_with_cpp_implementation`, which leverages a C++ implementation (`_PyInferPosteriorTimes`) for efficient computation of posterior node times and related attributes.
## References
* `cassiopeia.data.CassiopeiaTree`: Used to represent and manipulate phylogenetic trees.
* `._iid_exponential_bayesian._PyInferPosteriorTimes`: C++ module for efficient posterior time inference.
* `.BranchLengthEstimator.BranchLengthEstimator`: Abstract base class for branch length estimators.
## Symbols
### `IIDExponentialBayesian`
#### Description
This class implements the IID Exponential Bayesian branch length estimation method. It assumes a subsampled Birth Process for the phylogeny and IID exponential waiting times for mutations.
#### Inputs
| Name | Type | Description |
| :-------------------- | :---- | :----------------------------------------------------------------------------------- |
| mutation\_rate | float | The CRISPR/Cas9 mutation rate. |
| birth\_rate | float | The phylogeny birth rate. |
| sampling\_probability | float | The probability that a leaf in the ground truth tree was sampled. Must be in (0, 1]. |
| discretization\_level | int | How many timesteps are used to discretize time. Defaults to 600. |
#### Outputs
The class doesn't directly return outputs. Instead, it modifies the input `CassiopeiaTree` object in place, populating its branch lengths based on the estimated posterior mean times.
#### Internal Logic
1. **Input Validation:** Ensures the input tree is valid (binary except for the root) and the sampling probability is within the acceptable range.
2. **Data Preprocessing:** Imputes unambiguous missing states in the character matrix for easier dynamic programming.
3. **C++ Implementation:** Calls the `_PyInferPosteriorTimes` C++ module to efficiently compute the log joint probabilities, posterior means, and posterior distributions of node times.
4. **Branch Length Population:** Uses the computed posterior mean times to populate the branch lengths of the input `CassiopeiaTree`.
#### Side Effects
* Modifies the input `CassiopeiaTree` object in place by populating its branch lengths.
#### Performance Considerations
* The computational complexity of the branch length estimation is O(discretization\_level \* tree.n\_cell \* tree.n\_character).
* The C++ implementation (`_PyInferPosteriorTimes`) is used for performance optimization.
### `estimate_branch_lengths`
#### Description
This method estimates the branch lengths of the provided `CassiopeiaTree` using the IID Exponential Bayesian model.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :---------------------------------------------------------------------- |
| tree | CassiopeiaTree | The input `CassiopeiaTree` object for which to estimate branch lengths. |
#### Outputs
The method doesn't directly return outputs. Instead, it modifies the input `CassiopeiaTree` object in place, populating its branch lengths.
#### Internal Logic
1. **Input Validation:** Validates the input tree topology.
2. **Data Preprocessing:** Creates a deepcopy of the input tree and imputes deducible missing states.
3. **Posterior Time Inference:** Calls `_populate_attributes_with_cpp_implementation` to infer posterior node times using the C++ implementation.
4. **Branch Length Population:** Calls `_populate_branch_lengths` to populate the branch lengths of the original input tree using the inferred posterior means.
#### Side Effects
* Modifies the input `CassiopeiaTree` object in place by populating its branch lengths.
### `log_joints`
#### Description
This method returns the log joint probability density of the observed tree topology, state vectors, and all possible times for a given node.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------------------------------------------------ |
| node | str | The internal node for which to compute the log joint probabilities. |
#### Outputs
| Name | Type | Description |
| :---------- | :------- | :------------------------------------------------------------------- |
| log\_joints | np.array | An array of log joint probabilities for each discretized time point. |
### `posterior_time`
#### Description
This method returns the posterior distribution of the time for a given node, conditional on the observed character states and tree topology.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :---------------------------------------------------------------------- |
| node | str | The internal node for which to compute the posterior time distribution. |
#### Outputs
| Name | Type | Description |
| :-------------- | :------- | :------------------------------------------------------------------------ |
| posterior\_time | np.array | An array representing the posterior time distribution for the given node. |
## Dependencies
* numpy
* copy
* typing
* scipy (used in related code snippets)
* networkx (used in related code snippets)
* parameterized (used in related code snippets)
* pytest (used in related code snippets)
* unittest (used in related code snippets)
## Error Handling
* Raises `ValueError` for invalid input parameters or tree topology.
## TODOs
None found.
# IIDExponentialMLE.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/IIDExponentialMLE.py
## High-level description
The `IIDExponentialMLE` class implements a Maximum Likelihood Estimator (MLE) for inferring branch lengths of a tree under the assumption of independent and identically distributed (IID) memoryless CRISPR/Cas9 mutations with an exponential waiting time. This estimator assumes a tree depth of 1 and utilizes convex optimization to find the mutation rate and branch lengths that maximize the likelihood of the observed character data.
## Code Structure
The `IIDExponentialMLE` class inherits from the abstract base class `BranchLengthEstimator` and implements the `estimate_branch_lengths` method. This method constructs and solves a convex optimization problem to estimate the branch lengths and mutation rate.
## References
This code references the following key symbols:
* `CassiopeiaTree`: Used to represent the tree structure and character data.
* `IIDExponentialMLEError`: A custom exception class for errors specific to this estimator.
## Symbols
### `IIDExponentialMLE`
#### Description
This class implements the IID Exponential MLE algorithm for estimating branch lengths in a phylogenetic tree. It assumes that mutations occur independently and identically at each site with an exponential waiting time.
#### Inputs
| Name | Type | Description |
| :------------------------ | :---------------------- | :----------------------------------------------------------------------------------------------------------------- |
| minimum\_branch\_length | float | Minimum allowed branch length (default: 0.01) |
| relative\_mutation\_rates | Optional\[List\[float]] | List of relative mutation rates for each character site (default: None, meaning all sites evolve at the same rate) |
| verbose | bool | Whether to print verbose output during optimization (default: False) |
| solver | str | Convex optimization solver to use (default: "SCS", options: "ECOS", "SCS", "MOSEK") |
#### Outputs
This class doesn't directly return any outputs. Instead, it modifies the input `CassiopeiaTree` object in place, updating the branch lengths and storing the estimated mutation rate and log-likelihood as attributes.
#### Internal Logic
1. **Data Validation:** Checks for degenerate cases like no mutations or complete saturation in the character matrix.
2. **Problem Setup:**
* Creates optimization variables for each node's time.
* Defines constraints:
* Root node has time 0.
* Minimum branch length constraint.
* Ultrametric constraint for all leaves (except the reference leaf).
* Formulates the log-likelihood function based on the IID exponential mutation model.
3. **Optimization:**
* Defines the objective function as maximizing the log-likelihood.
* Constructs and solves the convex optimization problem using the specified solver.
4. **Result Extraction:**
* Extracts the estimated mutation rate from the scaling factor of the reference leaf's time.
* Retrieves the log-likelihood value.
5. **Tree Update:**
* Computes and sets the branch lengths based on the optimized node times.
* Updates the `CassiopeiaTree` object with the estimated branch lengths.
#### Side Effects
* Modifies the input `CassiopeiaTree` object in place by updating its branch lengths.
* Sets the `mutation_rate` and `log_likelihood` attributes of the `IIDExponentialMLE` object.
#### Performance Considerations
* The choice of solver can impact the performance of the optimization.
* The `minimum_branch_length` parameter can influence the convergence and runtime of the solver.
* Specifying `relative_mutation_rates` can increase the complexity of the optimization problem.
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------------------- |
| cvxpy | Used for defining and solving the convex optimization problem. |
| numpy | Used for numerical operations. |
## Error Handling
* Raises `ValueError` for invalid input parameters or degenerate character matrices.
* Raises `IIDExponentialMLEError` if the solver fails or produces unexpected results.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/__init__.py
## High-level description
This file defines the top-level module for branch length estimation in the Cassiopeia library. It imports two specific estimators: `IIDExponentialMLE` and `IIDExponentialBayesian`.
## Code Structure
This module simply imports two classes from submodules, making them directly accessible through the `cassiopeia.tools.branch_length_estimator` namespace.
## Symbols
### Symbol Name: `IIDExponentialMLE`
#### Description
This class implements a Maximum Likelihood Estimator (MLE) for branch lengths in a phylogenetic tree, assuming that CRISPR/Cas9 mutations occur independently and identically at each site with an exponential waiting time. It uses convex optimization to find the branch lengths that maximize the likelihood of the observed character data.
#### Inputs
This class doesn't have any methods with explicit inputs documented in the code.
#### Outputs
This class doesn't have any methods with explicit outputs documented in the code.
#### Internal Logic
The class utilizes the `cvxpy` library to formulate and solve the optimization problem. It constructs constraints based on the tree topology, minimum branch length, and ultrametricity (if applicable). The objective function is the log-likelihood of the observed character data given the branch lengths and mutation rates.
### Symbol Name: `IIDExponentialBayesian`
#### Description
This class implements a Bayesian approach to estimate branch lengths in a phylogenetic tree. It assumes a subsampled birth process for the phylogeny and independent, identically distributed exponential waiting times for mutations at each site. The estimator calculates the posterior mean branch lengths conditional on the observed tree topology and character data.
#### Inputs
This class doesn't have any methods with explicit inputs documented in the code.
#### Outputs
This class doesn't have any methods with explicit outputs documented in the code.
#### Internal Logic
The class leverages a C++ implementation (`_PyInferPosteriorTimes`) to efficiently compute the posterior node times. It preprocesses the tree data, mapping nodes to integer IDs, and extracts relevant information like the number of mutated characters at each node. After running the C++ implementation, it retrieves the log-likelihood, log joint probabilities, posterior probabilities, and posterior mean times for each node.
# _iid_exponential_bayesian_cpp.cpp
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/_iid_exponential_bayesian_cpp.cpp
## High-level description
This code implements a Bayesian inference algorithm to estimate branch lengths in a phylogenetic tree, assuming an IID exponential model of character evolution. It uses dynamic programming to efficiently calculate the posterior probabilities of branch lengths given observed character data.
## Code Structure
The code defines a class `_InferPosteriorTimes` that encapsulates the branch length estimation logic. The main components of this class are:
* `run()`: This method orchestrates the entire inference process, taking input data and model parameters, performing calculations, and storing the results.
* `down()`: This recursive function calculates the "down" probability, representing the likelihood of observing the data at and below a given node in the tree.
* `up()`: This recursive function calculates the "up" probability, representing the likelihood of observing the data above a given node and the node dividing at a specific time.
* `compute_log_joint()`: This function calculates the log joint probability of the data and a specific branch length.
* `populate_*_res()`: These functions extract and format the calculated results for later access.
The `main()` function is currently empty and serves no purpose in this file.
## References
This code does not reference other code symbols that require further explanation.
## Symbols
### `_InferPosteriorTimes`
#### Description
This class implements the Bayesian inference algorithm for estimating branch lengths in a phylogenetic tree.
#### Inputs
Refer to the `run()` method's parameters in the `_iid_exponential_bayesian_cpp.h` header file for a detailed description of the input data and model parameters.
#### Outputs
The class provides several getter methods to access the calculated results:
* `get_posterior_means_res()`: Returns a vector of pairs, where each pair contains a node index and its estimated posterior mean branch length.
* `get_posteriors_res()`: Returns a vector of pairs, where each pair contains a node index and a vector of its posterior probabilities for each time point.
* `get_log_joints_res()`: Returns a vector of pairs, where each pair contains a node index and a vector of its log joint probabilities for each time point.
* `get_log_likelihood_res()`: Returns the overall log-likelihood of the data given the model.
#### Internal Logic
The core logic of the algorithm lies within the `down()` and `up()` functions, which recursively calculate the "down" and "up" probabilities using dynamic programming. These probabilities are then combined to compute the log joint probabilities and subsequently the posterior probabilities of branch lengths. The algorithm also precomputes the probability of lineages not being sampled (`p_unsampled`) to account for incomplete lineage information.
#### Performance Considerations
The code utilizes dynamic programming with memoization (`down_cache` and `up_cache`) to avoid redundant computations, significantly improving performance. However, the memory usage scales with the size of the tree, the number of discretization time points (`T`), and the number of characters (`K`).
## Side Effects
This code has no notable side effects.
## Dependencies
This code uses the standard C++ library.
## Error Handling
The code includes basic error handling for memory allocation failures and invalid input parameters. It throws `std::invalid_argument` exceptions in such cases.
## Logging
This code does not implement any logging mechanisms.
# _iid_exponential_bayesian_cpp.h
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/_iid_exponential_bayesian_cpp.h
## High-level description
The code defines a class `_InferPosteriorTimes` in C++ that estimates the posterior distribution of branch lengths in a phylogenetic tree under an infinite-sites model of evolution with an exponential prior on branch lengths. This is done using dynamic programming to efficiently compute the likelihoods of different branch length configurations.
## Code Structure
The code defines a single class `_InferPosteriorTimes` which encapsulates all the logic for inferring posterior times. The class has a public `run` method that takes the tree data and model parameters as input and computes the posterior distributions. The results can then be accessed through various getter methods. Internally, the class uses dynamic programming tables (`down_cache`, `up_cache`) to store intermediate results and avoid redundant computations.
## Symbols
### `_InferPosteriorTimes`
#### Description
This class implements a dynamic programming algorithm to infer the posterior distribution of branch lengths in a phylogenetic tree under an infinite-sites model of evolution with an exponential prior on branch lengths.
#### Inputs
| Name | Type | Description |
| :--------------------------------------------- | :-------------------- | :---------------------------------------------------------------------------------------------- |
| N | int | Number of nodes in the tree |
| children | vector\> | Adjacency list representing the tree structure. `children[i]` contains the children of node `i` |
| root | int | The index of the root node |
| is\_internal\_node | vector\ | Boolean vector indicating whether each node is an internal node (1) or a leaf (0) |
| get\_number\_of\_mutated\_characters\_in\_node | vector\ | Number of mutations observed on the branch leading to each node |
| non\_root\_internal\_nodes | vector\ | List of indices of internal nodes excluding the root |
| leaves | vector\ | List of indices of leaf nodes |
| parent | vector\ | `parent[i]` is the parent of node `i` |
| K | int | Total number of characters in the data |
| K\_non\_missing | vector\ | `K_non_missing[i]` is the number of characters that are not missing at node `i` |
| T | int | Discretization level for time. Larger T leads to more accurate but slower computation |
| r | double | Mutation rate per character per unit time |
| lam | double | Cell division rate per unit time |
| sampling\_probability | double | Probability of sampling a lineage at the present time |
| is\_leaf | vector\ | Boolean vector indicating whether each node is a leaf |
#### Outputs
The class does not directly return any output. Instead, the results are stored internally and can be accessed through the following getter methods:
* `get_posterior_means_res()`: Returns a vector of pairs, where each pair represents a node and its posterior mean branch length.
* `get_posteriors_res()`: Returns a vector of pairs, where each pair represents a node and its posterior distribution of branch lengths.
* `get_log_joints_res()`: Returns a vector of pairs, where each pair represents a node and its log joint probability for each time point.
* `get_log_likelihood_res()`: Returns the log-likelihood of the data given the tree and model parameters.
#### Internal Logic
The class uses a dynamic programming algorithm to compute the likelihood of the data given the tree and model parameters. The algorithm works by recursively computing the likelihood of subtrees, starting from the leaves and moving towards the root. Two dynamic programming tables are used:
* `down_cache`: `down_cache[v][t][x]` stores the log-probability of generating all data at and below node `v`, starting from time `t` and with `x` mutations already accumulated on the branch leading to `v`.
* `up_cache`: `up_cache[v][t][x]` stores the log-probability of generating all data above node `v`, with node `v` dividing at time `t` and having accumulated `x` mutations on the branch leading to it.
The algorithm considers four cases at each step of the recursion: no event, mutation, cell division with both lineages sampled, and cell division with one lineage unsampled. The likelihoods of these cases are computed and combined using the log-sum-exp trick to avoid numerical underflow.
Once the dynamic programming tables are populated, the posterior distributions of branch lengths are computed by marginalizing over all possible event times for each internal node.
#### Performance Considerations
The time complexity of the algorithm is O(N*T*K), where N is the number of nodes, T is the discretization level for time, and K is the number of characters. The space complexity is also O(N*T*K). The algorithm can be sped up by using a smaller value of T, but this will result in a less accurate approximation of the posterior distribution.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/branch_length_estimator/overview
## High-level description
This directory contains implementations of branch length estimation algorithms for phylogenetic trees in the Cassiopeia framework. It includes two main estimators: IIDExponentialMLE (Maximum Likelihood Estimator) and IIDExponentialBayesian (Bayesian estimator). These estimators are designed to infer branch lengths in phylogenetic trees under the assumption of independent and identically distributed (IID) exponential waiting times for mutations.
## What does it do?
The branch length estimators in this directory perform the following tasks:
1. Estimate branch lengths of a given phylogenetic tree based on observed character data.
2. Implement different statistical approaches (Maximum Likelihood and Bayesian) to infer branch lengths.
3. Assume an IID exponential model for mutation events in the tree.
4. Handle various input parameters such as mutation rates, birth rates, and sampling probabilities.
5. Provide methods to access estimated branch lengths, posterior distributions, and log-likelihood values.
These estimators are crucial for understanding the evolutionary relationships and timings in phylogenetic trees, particularly in the context of CRISPR/Cas9-induced mutations.
## Entry points
The main entry points for this directory are:
1. `IIDExponentialMLE`: Implements a Maximum Likelihood Estimator for branch lengths.
2. `IIDExponentialBayesian`: Implements a Bayesian estimator for branch lengths.
Both classes inherit from the abstract base class `BranchLengthEstimator` and implement the `estimate_branch_lengths` method. This method takes a `CassiopeiaTree` object as input and modifies it in-place by updating its branch lengths.
The workflow typically involves:
1. Creating an instance of either `IIDExponentialMLE` or `IIDExponentialBayesian` with appropriate parameters.
2. Calling the `estimate_branch_lengths` method on a `CassiopeiaTree` object.
3. Accessing the estimated branch lengths and other relevant information from the modified tree or the estimator object.
## Key Files
1. `BranchLengthEstimator.py`: Defines the abstract base class `BranchLengthEstimator`.
2. `IIDExponentialMLE.py`: Implements the Maximum Likelihood Estimator.
3. `IIDExponentialBayesian.py`: Implements the Bayesian estimator.
4. `_iid_exponential_bayesian_cpp.cpp` and `_iid_exponential_bayesian_cpp.h`: C++ implementation of the core Bayesian inference algorithm for improved performance.
## Dependencies
The main dependencies for this directory include:
1. numpy: Used for numerical operations and array manipulations.
2. cvxpy: Used in `IIDExponentialMLE` for defining and solving the convex optimization problem.
3. scipy: Used in related code snippets for scientific computing.
4. networkx: Used in related code snippets for graph operations.
The C++ implementation also relies on standard C++ libraries for efficient computations.
## Configuration
The estimators can be configured through various parameters:
For `IIDExponentialMLE`:
* `minimum_branch_length`: Minimum allowed branch length (default: 0.01)
* `relative_mutation_rates`: List of relative mutation rates for each character site (optional)
* `verbose`: Whether to print verbose output during optimization (default: False)
* `solver`: Convex optimization solver to use (default: "SCS", options: "ECOS", "SCS", "MOSEK")
For `IIDExponentialBayesian`:
* `mutation_rate`: The CRISPR/Cas9 mutation rate
* `birth_rate`: The phylogeny birth rate
* `sampling_probability`: The probability that a leaf in the ground truth tree was sampled (must be in (0, 1])
* `discretization_level`: Number of timesteps used to discretize time (default: 600)
These parameters allow users to fine-tune the estimators based on their specific phylogenetic analysis requirements and the characteristics of their data.
# coupling.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/coupling.py
## High-level description
The `compute_evolutionary_coupling` function calculates the evolutionary coupling between different categories of a given meta variable in a CassiopeiaTree. This coupling statistic, a Z-normalized mean distance between categories, reflects the phylogenetic relatedness of these categories within the tree.
## References
This function references the `compute_phylogenetic_weight_matrix`, `compute_inter_cluster_distances`, and `net_relatedness_index` functions from the `cassiopeia.data.utilities` module.
## Symbols
### `compute_evolutionary_coupling`
#### Description
This function calculates the evolutionary coupling between categories of a specified meta variable in a CassiopeiaTree. It first computes the phylogenetic weight matrix or uses a precomputed one. Then, it filters categories based on a minimum proportion threshold. It calculates inter-cluster distances between categories using a specified distance function (defaulting to Net Relatedness Index). To generate a null distribution, it shuffles the meta variable assignments and recomputes the inter-cluster distances multiple times. Finally, it calculates Z-scores for the observed inter-cluster distances based on the null distribution, representing the evolutionary coupling between categories.
#### Inputs
| Name | Type | Description |
| :---------------------------- | :------------------------------- | :------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the tree and meta data. |
| meta\_variable | str | The name of the column in `tree.cell_meta` representing the categorical variable. |
| minimum\_proportion | float | Minimum proportion of cells a category must appear in to be considered (default 0.05). |
| number\_of\_shuffles | int | Number of shuffles for generating the null distribution (default 500). |
| random\_state | Optional\[np.random.RandomState] | Numpy random state for shuffling (default None). |
| dissimilarity\_map | Optional\[pd.DataFrame] | Precomputed dissimilarity map between leaves (default None). |
| cluster\_comparison\_function | Callable | Function to compare mean distance between groups (default `net_relatedness_index`). |
| \*\*comparison\_kwargs | | Additional arguments for the cluster comparison function. |
#### Outputs
| Name | Type | Description |
| :-------- | :----------- | :---------------------------------------------------------------------------------- |
| Z\_scores | pd.DataFrame | A K x K DataFrame containing the evolutionary coupling scores between K categories. |
#### Internal Logic
1. **Compute/Retrieve Dissimilarity:** Calculate the phylogenetic weight matrix using `compute_phylogenetic_weight_matrix` if no `dissimilarity_map` is provided, otherwise use the provided map.
2. **Filter Categories:** If `minimum_proportion` is greater than 0, filter out categories with frequencies below the specified proportion.
3. **Calculate Inter-cluster Distances:** Compute the distances between categories using the `compute_inter_cluster_distances` function with the specified `cluster_comparison_function` and additional arguments.
4. **Generate Null Distribution:** Shuffle the meta variable assignments `number_of_shuffles` times and recompute inter-cluster distances for each shuffle, storing the results.
5. **Calculate Z-scores:** For each pair of categories, calculate the Z-score of the observed inter-cluster distance based on the mean and standard deviation of the corresponding distances in the null distribution.
#### Performance Considerations
The computational complexity is O(n^2 log n + (B+1)(K^2 \* O(distance\_function))), where n is the number of leaves, K is the number of categories, and B is the number of shuffles. Precomputing the dissimilarity map can significantly reduce runtime.
# _FitnessEstimator.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_FitnessEstimator.py
## High-level description
The `FitnessEstimator` class in `cassiopeia/tools/fitness_estimator/_FitnessEstimator.py` defines an abstract base class for all fitness estimation algorithms in the Cassiopeia library. It mandates that all derived classes implement the `estimate_fitness` method, which calculates and assigns a 'fitness' attribute to each node in a given `CassiopeiaTree`.
## Code Structure
The code defines an abstract base class `FitnessEstimator` and a custom exception class `FitnessEstimatorError`. The `FitnessEstimator` class has a single abstract method, `estimate_fitness`, which needs to be implemented by concrete fitness estimation algorithm classes.
## References
This code references the `CassiopeiaTree` class from `cassiopeia.data`.
## Symbols
### `FitnessEstimatorError`
#### Description
This class represents an exception specific to the `FitnessEstimator` class and its subclasses. It is raised when errors occur during fitness estimation.
#### Inputs
*This class does not have any inputs.*
#### Outputs
*This class does not have any outputs.*
### `FitnessEstimator`
#### Description
This abstract base class defines the interface for all fitness estimation algorithms. It requires derived classes to implement the `estimate_fitness` method.
#### Inputs
*This class does not have any inputs.*
#### Outputs
*This class does not have any outputs.*
#### Internal Logic
*This class does not have any internal logic.*
### `FitnessEstimator.estimate_fitness`
#### Description
This abstract method defines the interface for estimating the fitness of nodes in a `CassiopeiaTree`. Concrete fitness estimation algorithms must implement this method.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :----------------------------------------------------------------------------- |
| tree | CassiopeiaTree | A `CassiopeiaTree` object representing the tree for which to estimate fitness. |
#### Outputs
*This method does not return any values. It modifies the input `CassiopeiaTree` object directly.*
#### Internal Logic
*This method is abstract and therefore has no implementation in the base class.*
## Side Effects
This code modifies the input `CassiopeiaTree` object in the `estimate_fitness` method by adding a 'fitness' attribute to each node.
## TODOs
*This code does not contain any TODOs.*
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/__init__.py
## High-level description
This file serves as the top-level entry point for the fitness estimator module in Cassiopeia. It exposes the main components, including the abstract `FitnessEstimator` class and the concrete `LBIJungle` implementation.
## References
* `cassiopeia.tools.fitness_estimator._FitnessEstimator`: Defines the abstract `FitnessEstimator` class and the `FitnessEstimatorError` exception.
* `cassiopeia.tools.fitness_estimator._lbi_jungle`: Implements the `LBIJungle` class, a concrete fitness estimator based on the LBI algorithm.
## Symbols
### Symbol Name
`FitnessEstimator`
#### Description
An abstract class defining the interface for fitness estimation algorithms in Cassiopeia. It mandates the implementation of the `estimate_fitness` method by its subclasses.
#### Inputs
N/A (abstract class)
#### Outputs
N/A (abstract class)
### Symbol Name
`FitnessEstimatorError`
#### Description
An exception class specific to the `FitnessEstimator` class, used to signal errors during fitness estimation.
#### Inputs
N/A (exception class)
#### Outputs
N/A (exception class)
### Symbol Name
`LBIJungle`
#### Description
A concrete implementation of the `FitnessEstimator` class that utilizes the LBI (lineage birth and immigration) algorithm for estimating fitness. It leverages the external `jungle` package for the core LBI calculations.
#### Inputs
N/A (class instantiation)
#### Outputs
N/A (class instantiation)
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/__init__.py
## High-level description
This file serves as a package initializer for the `jungle` module, which seems to be related to phylogenetic tree analysis, specifically focusing on fitness estimation. It imports and exposes functionality from submodules dealing with "forest", "sfs" (likely Site Frequency Spectrum), "size\_matched\_model", and "tree" data structures and algorithms.
## Code Structure
This file uses relative imports to include symbols from submodules within the `jungle` package. These submodules likely contain classes and functions related to different aspects of phylogenetic tree analysis.
## Symbols
### `*` (from `.forest`, `.sfs`, `.size_matched_model`, `.tree`)
#### Description
The `*` symbol indicates that all public symbols (functions, classes, etc.) are imported from the specified submodules (`forest`, `sfs`, `size_matched_model`, and `tree`) and made available directly through the `jungle` package namespace.
#### Inputs
N/A - This is a module-level import, not a function or class.
#### Outputs
N/A - This is a module-level import, not a function or class.
#### Internal Logic
N/A - This is a module-level import, not a function or class.
# forest.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/forest.py
## High-level description
The code defines a `Forest` class in Python, representing a collection of phylogenetic trees. It provides functionalities for loading, generating, manipulating, analyzing, and saving these tree collections. This includes methods for calculating various phylogenetic statistics, annotating tree nodes with features, and visualizing the trees.
## Code Structure
The `Forest` class is the main component, containing methods for manipulating collections of `Tree` objects. The `Tree` class (defined in a separate file) represents a single phylogenetic tree and provides methods for manipulating and analyzing individual trees. The `Forest` class leverages the functionalities of the `Tree` class to operate on entire collections of trees.
## References
* `jungle.tree.Tree`: The `Forest` class heavily relies on the `Tree` class for individual tree operations.
## Symbols
### Symbol Name: `Forest`
#### Description:
This class represents a collection of phylogenetic trees and provides methods for their manipulation and analysis.
#### Inputs:
| Name | Type | Description |
| :----- | :--- | :--------------------------------------------------------------------------------------------- |
| trees | list | A list of `Tree` objects representing the phylogenetic trees. |
| name | str | Optional name for the forest. |
| params | dict | Optional parameters for generating trees if the forest is created using the `generate` method. |
#### Outputs:
The class itself is the output, representing a `Forest` object.
#### Internal Logic:
The class stores the trees in a list and provides methods for:
* Loading trees from various formats (Newick, pickle, tar.gz).
* Generating trees.
* Concatenating forests.
* Annotating nodes with features.
* Calculating phylogenetic statistics.
* Fitting models to distributions of metrics.
* Visualizing trees.
* Saving the forest to disk.
### Symbol Name: `from_newick`
#### Description:
Class method to construct a `Forest` object by loading trees from Newick files.
#### Inputs:
| Name | Type | Description |
| :-------- | :--- | :---------------------------------- |
| filenames | list | A list of paths to Newick files. |
| name | str | Optional name for the forest. |
| params | dict | Optional parameters for the forest. |
#### Outputs:
| Name | Type | Description |
| :----- | :----- | :--------------------------------------------- |
| forest | Forest | A `Forest` object containing the loaded trees. |
#### Internal Logic:
The method iterates through the provided filenames, creates a `Tree` object for each file using `Tree.from_newick`, and appends it to the `trees` list of the `Forest` object.
### Symbol Name: `from_pickle`
#### Description:
Class method to load a `Forest` object from a pickle file.
#### Inputs:
| Name | Type | Description |
| :------- | :--- | :----------------------------------------------------------------- |
| filename | str | Path to the pickle file. |
| gzip | bool | Whether the file is gzipped. If None, it infers from the filename. |
#### Outputs:
| Name | Type | Description |
| :----- | :----- | :-------------------------- |
| forest | Forest | The loaded `Forest` object. |
#### Internal Logic:
The method opens the pickle file, optionally gzipped, and loads the `Forest` object using `pickle.load`.
### Symbol Name: `generate`
#### Description:
Class method to generate a `Forest` object by simulating trees.
#### Inputs:
| Name | Type | Description |
| :------- | :--- | :-------------------------------------------- |
| n\_trees | int | The number of trees to generate. |
| name | str | Optional name for the forest. |
| params | dict | Optional parameters for generating the trees. |
#### Outputs:
| Name | Type | Description |
| :----- | :----- | :------------------------------------------------ |
| forest | Forest | A `Forest` object containing the generated trees. |
#### Internal Logic:
The method generates `n_trees` number of trees using `Tree.generate` and appends them to the `trees` list of the `Forest` object.
### Symbol Name: `from_newick_tar_gz`
#### Description:
Class method to load a `Forest` object from a gzipped tar archive of Newick files.
#### Inputs:
| Name | Type | Description |
| :------- | :--- | :---------------------------------- |
| filename | str | Path to the tar.gz file. |
| name | str | Optional name for the forest. |
| params | dict | Optional parameters for the forest. |
#### Outputs:
| Name | Type | Description |
| :----- | :----- | :--------------------------------------------- |
| forest | Forest | A `Forest` object containing the loaded trees. |
#### Internal Logic:
The method opens the tar.gz file, iterates through its members, extracts the content of each Newick file, creates a `Tree` object using `Tree.from_newick`, and appends it to the `trees` list of the `Forest` object.
### Symbol Name: `to_newick`
#### Description:
Writes the trees in the `Forest` object to a gzipped tar archive of Newick files.
#### Inputs:
| Name | Type | Description |
| :--------- | :--- | :------------------------------------------------------------------ |
| outfile | str | Path to the output tar.gz file. |
| \*\*kwargs | dict | Additional keyword arguments passed to the `Tree.to_newick` method. |
#### Outputs:
This method doesn't return any value. It writes the trees to the specified file.
#### Internal Logic:
The method iterates through the trees, writes each tree to a temporary Newick file using `Tree.to_newick`, adds all temporary files to a tar.gz archive, and finally deletes the temporary files.
### Symbol Name: `concat`
#### Description:
Concatenates two `Forest` objects into a new one.
#### Inputs:
| Name | Type | Description |
| :---- | :----- | :---------------------------------------- |
| other | Forest | The other `Forest` object to concatenate. |
#### Outputs:
| Name | Type | Description |
| :---------- | :----- | :---------------------------------------------------------------------- |
| new\_forest | Forest | A new `Forest` object containing the concatenated trees and attributes. |
#### Internal Logic:
The method combines the `trees` lists of both forests, concatenates other list-like attributes using `_concat_lists_safe`, and creates a new `Forest` object with the combined data.
### Symbol Name: `annotate_standard_node_features`
#### Description:
Annotates each node of each tree in the forest with standard features: depth, depth\_rank, num\_children, num\_descendants.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
This method doesn't return any value. It modifies the trees in place.
#### Internal Logic:
The method iterates through the trees and calls the `annotate_standard_node_features` method of each `Tree` object.
### Symbol Name: `annotate_imbalance`
#### Description:
Annotates each node of each tree in the forest with its imbalance (I), calculated as the maximum number of descendants among its children divided by the total number of descendants.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
This method doesn't return any value. It modifies the trees in place.
#### Internal Logic:
The method iterates through the trees and calls the `annotate_imbalance` method of each `Tree` object.
### Symbol Name: `annotate_colless`
#### Description:
Annotates each node of each tree in the forest with its Colless index.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
This method doesn't return any value. It modifies the trees in place.
#### Internal Logic:
The method iterates through the trees and calls the `annotate_colless` method of each `Tree` object.
### Symbol Name: `node_features`
#### Description:
Returns a Pandas DataFrame containing features for each node of each tree in the forest.
#### Inputs:
| Name | Type | Description |
| :----- | :--- | :---------------------------------------------------------- |
| subset | list | Optional list of feature names to include in the DataFrame. |
#### Outputs:
| Name | Type | Description |
| :----------- | :--------------- | :----------------------------------------------------------- |
| features\_df | pandas.DataFrame | A DataFrame containing the requested features for each node. |
#### Internal Logic:
The method retrieves the node features from each tree using `Tree.node_features`, concatenates them into a single DataFrame, and ensures unique tree names.
### Symbol Name: `rescale`
#### Description:
Rescales all trees in the forest to have a specific total branch length.
#### Inputs:
| Name | Type | Description |
| :-------------------- | :---- | :--------------------------------------------- |
| total\_branch\_length | float | The desired total branch length for each tree. |
#### Outputs:
This method doesn't return any value. It modifies the trees in place.
#### Internal Logic:
The method iterates through the trees and calls the `rescale` method of each `Tree` object with the specified `total_branch_length`.
### Symbol Name: `resolve_polytomy`
#### Description:
Resolves all polytomies (nodes with more than two children) in each tree by creating an arbitrary dichotomous structure.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
This method doesn't return any value. It modifies the trees in place.
#### Internal Logic:
The method iterates through the trees and calls the `resolve_polytomy` method of each `Tree` object.
### Symbol Name: `site_frequency_spectrum`
#### Description:
Calculates the site frequency spectrum (SFS) for each tree in the forest.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
| Name | Type | Description |
| :-------- | :--- | :--------------------------------------- |
| sfs\_list | list | A list of SFS arrays, one for each tree. |
#### Internal Logic:
The method iterates through the trees, calculates the SFS for each tree using `Tree.site_frequency_spectrum`, and stores them in a list.
### Symbol Name: `bin_site_frequency_spectrum`
#### Description:
Calculates the binned site frequency spectrum (SFS) for each tree in the forest.
#### Inputs:
| Name | Type | Description |
| :--------- | :--------- | :---------------------------------------------------------------------------- |
| bins | array-like | The bins to use for binning the SFS. |
| \*args | tuple | Additional positional arguments passed to `Tree.bin_site_frequency_spectrum`. |
| \*\*kwargs | dict | Additional keyword arguments passed to `Tree.bin_site_frequency_spectrum`. |
#### Outputs:
| Name | Type | Description |
| :---------------- | :--- | :---------------------------------------------- |
| binned\_sfs\_list | list | A list of binned SFS arrays, one for each tree. |
#### Internal Logic:
The method iterates through the trees, calculates the binned SFS for each tree using `Tree.bin_site_frequency_spectrum`, and stores them in a list.
### Symbol Name: `mean_site_frequency_spectrum`
#### Description:
Calculates the mean and standard error of the mean (SEM) of the site frequency spectrum (SFS) across all trees in the forest.
#### Inputs:
| Name | Type | Description |
| :---- | :--- | :------------------------------------------------------------------------------------------------------------------ |
| which | str | Specifies which SFS representation to use (e.g., "binned\_normalized\_cut"). Defaults to "binned\_normalized\_cut". |
#### Outputs:
| Name | Type | Description |
| :--- | :--------- | :------------------------------------ |
| mean | array-like | The mean of the SFS across all trees. |
| sem | array-like | The SEM of the SFS across all trees. |
#### Internal Logic:
The method extracts the specified SFS representation from each tree, calculates the mean and SEM across all trees using `np.nanmean` and `scipy.stats.sem`, and returns the results.
### Symbol Name: `fay_and_wus_H`
#### Description:
Calculates Fay and Wu's H statistic for each tree in the forest.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
| Name | Type | Description |
| :-------- | :--- | :-------------------------------------------------- |
| h\_values | list | A list of Fay and Wu's H values, one for each tree. |
#### Internal Logic:
The method iterates through the trees, calculates Fay and Wu's H for each tree using `Tree.fay_and_wus_H`, and stores them in a list.
### Symbol Name: `zengs_E`
#### Description:
Calculates Zeng's E statistic for each tree in the forest.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
| Name | Type | Description |
| :-------- | :--- | :-------------------------------------------- |
| e\_values | list | A list of Zeng's E values, one for each tree. |
#### Internal Logic:
The method iterates through the trees, calculates Zeng's E for each tree using `Tree.zengs_E`, and stores them in a list.
### Symbol Name: `tajimas_D`
#### Description:
Calculates Tajima's D statistic for each tree in the forest.
#### Inputs:
This method doesn't take any input arguments.
#### Outputs:
| Name | Type | Description |
| :--- | :--- | :---------- |
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/overview
## High-level description
This directory contains implementations of various algorithms and tools for analyzing evolutionary fitness and phylogenetic relationships in biological sequences. The main components are:
1. A `Forest` class representing a collection of phylogenetic trees.
2. A `Tree` class representing individual phylogenetic trees.
3. A `SFS` (Site Frequency Spectrum) class for analyzing genetic diversity.
4. A `SizeMatchedModel` class for statistical modeling based on data size.
These tools are designed to analyze genetic diversity, infer evolutionary fitness, and understand the shape of genealogical trees in populations.
## What does it do?
The code in this directory performs several key functions:
1. Manages collections of phylogenetic trees:
* Loads and saves trees from/to various formats (Newick, pickle, tar.gz).
* Generates trees through simulation.
* Concatenates forests of trees.
2. Analyzes individual phylogenetic trees:
* Annotates nodes with features like depth, number of descendants, imbalance, and Colless index.
* Calculates tree statistics such as total branch length and site frequency spectrum.
* Resolves polytomies and rescales trees.
3. Calculates and analyzes Site Frequency Spectra (SFS):
* Computes SFS from tree structures.
* Calculates various genetic diversity statistics (Tajima's D, Fay and Wu's H, Zeng's E, Ferretti's L).
* Bins SFS data for visualization and analysis.
4. Provides statistical modeling based on data size:
* Implements a model where parameters vary based on input data size.
* Calculates p-values and model means for given sizes.
5. Visualizes phylogenetic trees and results.
## Entry points
The main entry points for using this codebase are:
1. `Forest` class: For managing collections of phylogenetic trees.
2. `Tree` class: For analyzing individual phylogenetic trees.
3. `SFS` class: For calculating and analyzing Site Frequency Spectra.
4. `SizeMatchedModel` class: For statistical modeling based on data size.
## Key Files
1. `forest.py`: Defines the `Forest` class for managing collections of trees.
2. `tree.py`: Defines the `Tree` class for individual tree analysis.
3. `sfs.py`: Defines the `SFS` class for Site Frequency Spectrum analysis.
4. `size_matched_model.py`: Defines the `SizeMatchedModel` class for size-based statistical modeling.
## Dependencies
The codebase relies on several external libraries:
1. `ete3`: For phylogenetic tree manipulation and visualization.
2. `Bio.Phylo`: For interfacing with Biopython's phylogenetic tree representation.
3. `numpy`: For numerical computations and array manipulations.
4. `scipy`: For various scientific computing tasks and statistical functions.
5. `pandas`: For data manipulation and analysis.
6. `matplotlib`: For visualization of results and trees.
Additionally, the code uses custom modules for specific tasks like beta-tree simulation and fitness inference.
## Configuration
The main classes (`Forest`, `Tree`, `SFS`, `SizeMatchedModel`) use constructor parameters and method arguments for configuration. Key parameters include:
* For `Forest` and `Tree`:
* Paths to input files (Newick, pickle, tar.gz).
* Parameters for tree generation (e.g., number of leaves, alpha for beta-tree model).
* Options for tree manipulation (e.g., resolving polytomies, rescaling).
* For `SFS`:
* SFS data as numpy array.
* Binning parameters for SFS analysis.
* For `SizeMatchedModel`:
* Bins and parameters for size-based modeling.
* Statistical distribution for calculations.
Users can adjust these parameters to customize the analysis for their specific needs in evolutionary studies and population genetics research.
# README.md
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/README.md
### Inferring fitness from the shape of trees
This repository contains the code associated with the manuscript
Neher, Russell, Shraiman: "Predicting evolution from the shape of genealogical trees". accepted for publication in eLife
***
The directory *prediction\_src* contains the code base used for the fitness inference and prediction algorithms as well as classes to hold sequence data and trees adapted.
***
The directory *flu* contains the code specific to our analysis of historical influenza data, scripts that generate the figures, the influenza sequences and annotation, analysis results and figure files.
***
The directory *toy\_data* contains the code to simulate adapting populations building on the FFPopSim library. In addition, it contains scripts to analyze this simulated data, the data itself and the resulting figures.
***
#### Ranking sequences by the local branching index (LBI)
The script *rank\_sequences.py* is a simple wrapper for the prediction tool that takes a multiple sequence alignment and the name of the outgroup as input (this outgroup needs to be in the MSA). It produces a folder containing a ranking of sequences, the inferred ancestral sequences, the reconstructed tree, and optionally a pdf of the marked up tree. This script uses the local branching index (LBI), rather than the full fitness inference to rank sequences.
build-in help and optional arguments:
./rank\_sequences.py --help
usage: rank\_sequences.py \[-h] --aln ALN --outgroup OUTGROUP
\[--eps\_branch EPS\_BRANCH] \[--tau TAU]
\[--collapse \[COLLAPSE]] \[--plot \[PLOT]]
rank sequences in a multiple sequence aligment
optional arguments:
-h, --help show this help message and exit
\--aln ALN alignment of sequences to by ranked
\--outgroup OUTGROUP name of outgroup sequence
\--eps\_branch EPS\_BRANCH
minimal branch length for inference
\--tau TAU time scale for local tree length estimation (relative
to average pairwise distance)
\--collapse \[COLLAPSE]
collapse internal branches with identical sequences
\--plot \[PLOT] plot trees
#### Inferring fitness distribution of nodes in the tree
The script *infer\_fitness.py* also takes an alignment and outgroup as argument, but uses the full fitness inference to rank sequences and calculate the mean posterior and the variance of the posterior. Note that plausible posterior distributions require a that the parameter omega is well chosen. Also, the time conversion factor might need to be different from gamma=1 for optimal results.
./infer\_fitness.py --help
usage: infer\_fitness.py \[-h] --aln ALN --outgroup OUTGROUP
\[--eps\_branch EPS\_BRANCH] \[--diffusion DIFFUSION]
\[--gamma GAMMA] \[--omega OMEGA]
\[--collapse \[COLLAPSE]] \[--plot \[PLOT]]
rank sequences in a multiple sequence aligment
optional arguments:
-h, --help show this help message and exit
\--aln ALN alignment of sequences to by ranked
\--outgroup OUTGROUP name of outgroup sequence
\--eps\_branch EPS\_BRANCH
minimal branch length for inference
\--diffusion DIFFUSION
fitness diffusion coefficient
\--gamma GAMMA scale factor for time scale, choose high (>2) for
prediction, 1 for fitness inference
\--omega OMEGA approximate sampling fraction diveded by the fitness
standard deviation
\--collapse \[COLLAPSE]
collapse internal branches with identical sequences
\--plot \[PLOT] plot trees
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/__init__.py
empty here
# infer_fitness.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/infer_fitness.py
## High-level description
This script performs fitness inference on a given sequence alignment to predict evolutionary relationships and rank sequences based on their inferred fitness. It uses a phylogenetic approach, considering an outgroup sequence to root the tree and inferring ancestral states.
## Code Structure
The code first parses command-line arguments, reads and processes the input alignment, and identifies the outgroup sequence. Then, it instantiates a `sequence_ranking` object from the imported module and uses it to perform the fitness inference. Finally, it outputs the inferred tree, ancestral sequences, and sequence rankings.
## References
This script references the following code symbols:
* `alignment` (from `sequence_ranking` module)
* `sequence_ranking` (from `sequence_ranking` module)
* `tree_utils` (imported module)
## Symbols
### `ofunc`
#### Description
This function determines the appropriate file opening method based on the file extension. It uses `gzip.open` for '.gz' files and the built-in `open` function otherwise.
#### Inputs
| Name | Type | Description |
| :---- | :--- | :--------------------------------------------------------------------------- |
| fname | str | The name of the file to open. |
| mode | str | The mode in which to open the file (e.g., 'r' for reading, 'w' for writing). |
#### Outputs
| Name | Type | Description |
| :---------- | :--- | :------------------------------------------ |
| file object | file | A file object representing the opened file. |
### main execution block
#### Description
This block of code executes the main functionality of the script. It reads the alignment file, identifies the outgroup sequence, performs fitness inference, and writes the results to files.
#### Internal Logic
1. **Read alignment and outgroup:**
* Reads the alignment file in FASTA format.
* Identifies the outgroup sequence based on the provided `--outgroup` argument.
2. **Set up sequence data and perform prediction:**
* Creates an `alignment` object from the input alignment and outgroup.
* Instantiates a `sequence_ranking` object with specified parameters.
* Predicts the best node using the `predict()` method of the `sequence_ranking` object.
3. **Output results:**
* Creates a directory for output files based on the current date and time.
* Writes the reconstructed tree in Newick format to `reconstructed_tree.nwk`.
* Writes inferred ancestral sequences to `ancestral_sequences.fasta`.
* Writes sequence rankings for terminal and non-terminal nodes to separate files.
* Optionally plots the marked-up tree if the `--plot` flag is set.
## Dependencies
| Dependency | Purpose |
| :---------------- | :---------------------------------------------------------------------------------- |
| argparse | Parsing command-line arguments. |
| matplotlib | Plotting the marked-up tree (optional). |
| Bio | Handling sequence alignments and phylogenetic trees. |
| numpy | Numerical operations. |
| tree\_utils | Utility functions for tree manipulation and visualization. |
| sequence\_ranking | Module containing classes and functions for sequence ranking and fitness inference. |
### Configuration
This script uses command-line arguments for configuration. See the "parse the command line arguments" section for details on available options and their descriptions.
## Error Handling
The script includes a basic check for the presence of the outgroup sequence in the alignment. If not found, it prints an error message and exits.
## Logging
This script does not implement specific logging mechanisms.
## TODOs
This script does not contain any TODOs or notes.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/overview
## High-level description
This directory contains code for inferring fitness from the shape of genealogical trees, as described in the manuscript "Predicting evolution from the shape of genealogical trees" by Neher, Russell, and Shraiman. The code implements algorithms for fitness inference, sequence ranking, and phylogenetic tree analysis.
## What does it do?
The code in this directory performs several key functions:
1. Ranks sequences in a multiple sequence alignment based on their inferred fitness.
2. Infers fitness distributions for nodes in phylogenetic trees.
3. Reconstructs ancestral sequences.
4. Analyzes and visualizes phylogenetic trees.
5. Simulates adapting populations (in the toy\_data subdirectory).
The main workflows include:
1. Using the Local Branching Index (LBI) to rank sequences based on their evolutionary fitness.
2. Performing full fitness inference to calculate mean posterior and variance of fitness for sequences.
3. Reconstructing phylogenetic trees from sequence alignments.
4. Visualizing trees with fitness information.
## Entry points
The main entry points for using this codebase are:
1. `rank_sequences.py`: A script that ranks sequences in a multiple sequence alignment using the Local Branching Index (LBI).
2. `infer_fitness.py`: A script that performs full fitness inference on sequences in a multiple sequence alignment.
Both scripts take a multiple sequence alignment and an outgroup sequence as input, and produce various output files including ranked sequences, reconstructed trees, and inferred ancestral sequences.
## Key Files
1. `prediction_src/`: A directory containing the core implementation of fitness inference and sequence ranking algorithms. Key modules include:
* `sequence_ranking.py`: Combines sequence alignment processing with node ranking and fitness inference.
* `fitness_inference.py`: Implements the core algorithm for inferring fitness distributions on phylogenetic trees.
* `node_ranking.py`: Extends fitness inference functionality to provide methods for ranking and coloring tree nodes.
* `ancestral.py`: Implements maximum likelihood estimation for ancestral sequence reconstruction.
* `tree_utils.py`: Provides utility functions for manipulating and visualizing phylogenetic trees.
2. `rank_sequences.py`: Script for ranking sequences using the Local Branching Index.
3. `infer_fitness.py`: Script for performing full fitness inference on sequences.
## Dependencies
The codebase relies on several external libraries:
1. Biopython: For handling biological sequences, alignments, and phylogenetic trees.
2. NumPy: For numerical computations and array manipulations.
3. SciPy: For various scientific computing tasks.
4. Matplotlib: For visualization of phylogenetic trees and results.
Additionally, the code may use external tools like `fasttree` for phylogenetic tree construction.
## Configuration
The main scripts (`rank_sequences.py` and `infer_fitness.py`) use command-line arguments for configuration. Key parameters include:
* `--aln`: Path to the input alignment file.
* `--outgroup`: Name of the outgroup sequence.
* `--eps_branch`: Minimal branch length for inference.
* `--tau`: Time scale for local tree length estimation (for LBI).
* `--diffusion`: Fitness diffusion coefficient (for full inference).
* `--gamma`: Scale factor for time scale.
* `--omega`: Approximate sampling fraction divided by fitness standard deviation.
* `--collapse`: Option to collapse internal branches with identical sequences.
* `--plot`: Option to plot trees.
Users can adjust these parameters to customize the analysis for their specific needs.
In conclusion, this codebase provides a comprehensive set of tools for analyzing evolutionary fitness and phylogenetic relationships in biological sequences, with applications in predicting evolution and understanding the shape of genealogical trees.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/__init__.py
empty here
# ancestral.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/ancestral.py
## High-level description
The code defines a class `ancestral_sequences` that takes a phylogenetic tree and a multiple sequence alignment as input. It then performs maximum likelihood estimation to infer the most likely ancestral sequences for each internal node in the tree, along with the marginal probabilities of different character states at each position.
## Code Structure
The `ancestral_sequences` class is the main component of the code. It uses Biopython's `Phylo` and `Seq` objects to represent the tree and sequences, respectively. The class methods implement a message-passing algorithm to calculate the ancestral sequences and marginal probabilities.
## References
This code references Biopython library (`Phylo`, `Seq`, `SeqRecord`).
## Symbols
### `ancestral_sequences`
#### Description
This class performs ancestral sequence reconstruction using a maximum likelihood approach. It takes a phylogenetic tree and aligned sequences as input and infers the most likely ancestral sequences for internal nodes.
#### Inputs
| Name | Type | Description |
| :------------------ | :----------------------------- | :--------------------------------------------------------------------------------- |
| tree | Bio.Phylo.BaseTree.Tree | A Biopython phylogenetic tree object. |
| aln | Bio.Align.MultipleSeqAlignment | A Biopython multiple sequence alignment object. |
| alphabet | str | The set of allowed characters in the sequences (default: "ACGT"). |
| sub\_matrix | numpy.ndarray | A substitution matrix (optional, defaults to a flat matrix). |
| eps\_branch\_length | float | A small value added to branch lengths to prevent division by zero (default: 1e-7). |
| copy\_tree | bool | Whether to create a copy of the input tree (default: True). |
#### Outputs
The class itself doesn't return any output. However, it modifies the input tree object by adding ancestral sequences and marginal probabilities to the internal nodes.
#### Internal Logic
The class uses a message-passing algorithm, similar to dynamic programming, to calculate the ancestral sequences. The algorithm works by passing messages up and down the tree, representing the likelihood of different ancestral states at each node. The messages are calculated based on the observed sequences at the leaves, the branch lengths, and the substitution matrix.
1. **Initialization:** Each leaf node is assigned its observed sequence and a probability array representing the certainty of the observed state. Internal nodes are initialized with uniform probability arrays.
2. **Upward Pass (calc\_up\_messages):** Starting from the leaves, messages are passed upwards towards the root. Each node combines the messages from its children, weighting them by the branch lengths and the substitution probabilities.
3. **Downward Pass (calc\_down\_messages):** From the root, messages are passed downwards towards the leaves. Each node combines the message from its parent with the upward messages from its other children.
4. **Marginal Probabilities (calc\_marginal\_probabilities):** The upward and downward messages are combined at each node to calculate the marginal probability of each character state at each position.
5. **Most Likely Sequences (calc\_most\_likely\_sequences):** The most likely ancestral sequence at each node is determined by selecting the character state with the highest marginal probability at each position.
#### Side Effects
The main side effect is the modification of the input tree object. The ancestral sequences and marginal probabilities are stored within the internal nodes of the tree.
### `get_state_array`
#### Description
This method returns a NumPy array filled with ones, representing the initial state probabilities for a node.
#### Inputs
This method doesn't take any input arguments.
#### Outputs
| Name | Type | Description |
| :----------- | :------------ | :------------------------------------------------------------------------------------- |
| state\_array | numpy.ndarray | A 2D NumPy array with dimensions (sequence length, number of states) filled with ones. |
### `calc_eigendecomp`
#### Description
This method calculates the eigenvalues and eigenvectors of the substitution matrix. This decomposition is used to efficiently calculate the state probabilities over time.
#### Inputs
This method doesn't take any input arguments.
#### Outputs
The method stores the calculated eigenvalues and eigenvectors in the `self.evals` and `self.evecs` attributes, respectively.
### `calc_state_probabilites`
#### Description
This method calculates the probabilities of different character states at a given time point, given an initial state probability vector and a time interval.
#### Inputs
| Name | Type | Description |
| :--- | :------------ | :----------------------------------------------------------------- |
| P | numpy.ndarray | A probability vector representing the initial state probabilities. |
| t | float | The time interval. |
#### Outputs
| Name | Type | Description |
| :------------------- | :------------ | :---------------------------------------------------------------------- |
| state\_probabilities | numpy.ndarray | A probability vector representing the state probabilities after time t. |
### `normalize`
#### Description
This method normalizes the probability array of a given clade (node) such that the sum of probabilities at each position equals one.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :--------------------------------------------------------- |
| clade | Bio.Phylo.Clade | The clade (node) for which to normalize the probabilities. |
#### Outputs
This method doesn't return any output. It modifies the `clade.prob` array in place.
### `log_normalize`
#### Description
This method converts unnormalized logarithmic probabilities to linear probabilities and then normalizes them.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :--------------------------------------------------------- |
| clade | Bio.Phylo.Clade | The clade (node) for which to normalize the probabilities. |
#### Outputs
This method doesn't return any output. It modifies the `clade.prob` array in place.
### `calc_up_messages`
#### Description
This method recursively calculates the "up messages" for each node in the tree. These messages represent the likelihood of different ancestral states at a node, considering only the subtree below that node.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :------------------------------------------------------ |
| clade | Bio.Phylo.Clade | The clade (node) for which to calculate the up message. |
#### Outputs
This method doesn't return any output. It modifies the `clade.up_message` attribute of the input clade and recursively updates the up messages of its descendants.
### `calc_down_messages`
#### Description
This method recursively calculates the "down messages" for each node in the tree. These messages represent the likelihood of different ancestral states at a node, considering the entire tree except for the subtree below that node.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :-------------------------------------------------------- |
| clade | Bio.Phylo.Clade | The clade (node) for which to calculate the down message. |
#### Outputs
This method doesn't return any output. It modifies the `clade.down_message` attribute of the input clade and recursively updates the down messages of its descendants.
### `calc_marginal_probabilities`
#### Description
This method calculates the marginal probabilities of different character states at each position for each internal node in the tree.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :------------------------------------------------------------------ |
| clade | Bio.Phylo.Clade | The clade (node) for which to calculate the marginal probabilities. |
#### Outputs
This method doesn't return any output. It modifies the `clade.prob` attribute of the input clade and recursively updates the marginal probabilities of its descendants.
### `calc_most_likely_sequences`
#### Description
This method recursively determines the most likely ancestral sequence for each internal node in the tree based on the calculated marginal probabilities.
#### Inputs
| Name | Type | Description |
| :---- | :-------------- | :---------------------------------------------------------------- |
| clade | Bio.Phylo.Clade | The clade (node) for which to determine the most likely sequence. |
#### Outputs
This method doesn't return any output. It modifies the `clade.seq` attribute of the input clade and recursively updates the most likely sequences of its descendants.
### `calc_ancestral_sequences`
#### Description
This method orchestrates the entire ancestral sequence reconstruction process. It calls the necessary methods to calculate the up messages, down messages, marginal probabilities, and most likely sequences for all nodes in the tree.
#### Inputs
This method doesn't take any input arguments.
#### Outputs
This method doesn't return any output. It modifies the input tree object by adding ancestral sequences and marginal probabilities to the internal nodes.
## Dependencies
This code depends on the following external library:
| Dependency | Purpose |
| :--------- | :-------------------------------------------- |
| biopython | Handling phylogenetic trees and sequence data |
# fitness_inference.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/fitness_inference.py
## High-level description
The code defines a `fitness_inference` class in Python for inferring the ancestral fitness distribution of nodes in a phylogenetic tree. It leverages a message-passing algorithm to propagate fitness information up and down the tree, ultimately calculating the posterior fitness distribution for each node. This analysis helps understand the evolutionary trajectory of fitness within the tree.
## Code Structure
The `fitness_inference` class inherits from the `survival_gen_func` class, suggesting that the fitness inference process builds upon a survival analysis framework. The code defines methods for setting up the tree, calculating messages and polarizers that propagate fitness information, and ultimately computing marginal probabilities and fitness statistics for each node.
## References
* `solve_survival`: The code imports `survival_gen_func` from `.solve_survival`, indicating a dependency on this module likely for survival analysis functionalities.
## Symbols
### `fitness_inference`
#### Description
This class performs fitness inference on a phylogenetic tree, calculating the ancestral fitness distribution for each node. It uses a message-passing algorithm, propagating fitness information up and down the tree structure.
#### Inputs
The class constructor (`__init__`) takes the following inputs:
| Name | Type | Description |
| :------------------ | :---------------------- | :----------------------------------------------------------------------------------------------------------- |
| eps\_branch\_length | float | A small value added to branch lengths to avoid zero-length issues. Defaults to 1e-7. |
| D | float | Diffusion constant representing the rate of fitness change over time. Defaults to 0.2. |
| fit\_grid | numpy.ndarray, optional | A user-defined grid of fitness values. If not provided, a default grid is used. |
| samp\_frac | float | Sampling fraction, likely related to the proportion of individuals sampled. Defaults to 0.01. |
| mem | float | Memory parameter, potentially influencing the persistence of fitness information over time. Defaults to 1.0. |
#### Outputs
The class itself doesn't directly return an output. However, after calling the `infer_ancestral_fitness` method, the tree nodes are populated with inferred fitness information, including:
* `prob`: Marginal probability distribution of fitness for the node.
* `mean_fitness`: Mean of the posterior fitness distribution.
* `var_fitness`: Variance of the posterior fitness distribution.
#### Internal Logic
1. **Tree Initialization (`set_tree`):**
* The input tree is ladderized.
* Terminal nodes are identified and ranked.
* Time to present is calculated for each node.
* A time scale is determined, either from input or estimated from the tree.
* Nodes are initialized with probability vectors.
2. **Message Passing:**
* **Downward Messages (`calc_down_messages`):** Fitness information is propagated from parent nodes to children.
* **Upward Messages (`calc_up_messages`):** Fitness information is propagated from children to parent nodes.
* **Polarizers (`calc_down_polarizers`, `calc_up_polarizers`):** These likely contribute to adjusting the fitness propagation based on branch lengths and the memory parameter.
3. **Marginal Probability Calculation (`calc_marginal_probabilities`):** The product of incoming messages at each node is used to calculate the marginal probability of each fitness state.
4. **Fitness Statistics (`calc_mean_and_variance`):** The mean and variance of the fitness distribution are calculated for each node based on the marginal probabilities.
#### Performance Considerations
The code precomputes and stores the propagator function for different time intervals using `integrate_phi` and `integrate_prop` to optimize the message-passing steps.
## Dependencies
* `numpy`: Used for numerical operations and array manipulation.
* `biopython`: Assumed to be used for handling phylogenetic trees (not explicitly imported in this file).
## Error Handling
The code includes basic error handling by setting values below a certain threshold (`non_negativity_cutoff`) to the threshold itself, preventing potential issues with zero probabilities. However, more robust error handling mechanisms are not explicitly implemented in this code snippet.
# node_ranking.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/node_ranking.py
## High-level description
The `node_ranking` class extends the `fitness_inference` class and is responsible for ranking nodes in a phylogenetic tree based on various methods, including mean posterior fitness, polarizer scores, branch length, depth, and expansion score. It provides functionalities to compute these rankings, color-code trees based on rankings, and assess the quality of rankings.
## Code Structure
The `node_ranking` class inherits from the `fitness_inference` class. It utilizes the tree structure and inferred fitness values from the parent class to compute and assign rankings to each node. The ranking methods can be customized during initialization.
## References
This class references the `fitness_inference` class for inheriting functionality related to fitness inference and the `tree_utils` module for tree manipulation and calculations. Additionally, it uses the `sort_leafs_in_time_bins` method internally for calculating the expansion score.
## Symbols
### `node_ranking`
#### Description
This class extends the `fitness_inference` class to rank nodes in a phylogenetic tree using various methods. It provides methods for computing rankings, coloring trees based on rankings, and evaluating ranking quality.
#### Inputs
| Name | Type | Description |
| :------------ | :---- | :---------------------------------------------------------------------------------------------------------------------------------- |
| methods | list | List of methods to use for ranking nodes (default: \['mean\_fitness', 'polarizer', 'branch\_length', 'depth', 'expansion\_score']). |
| time\_bins | list | List of time points defining temporal bins for expansion score calculation (default: None). |
| pseudo\_count | int | Pseudo-count added to frequencies when calculating expansion score (default: 5). |
| \*args | tuple | Variable length argument list passed to the parent class constructor. |
| \*\*kwargs | dict | Arbitrary keyword arguments passed to the parent class constructor. |
#### Outputs
None
#### Internal Logic
The class initializes by calling the constructor of its parent class (`fitness_inference`) and storing the provided ranking methods, time bins, and pseudo-count. The `compute_rankings` method calculates the scores for each specified ranking method. It first infers ancestral fitness and calculates polarizer scores if required. Then, it computes depth, expansion score, and any other custom ranking methods. The `expansion_score` method calculates the fraction of leaves under each node within each time bin and performs linear regression on the log-transformed frequencies to determine the expansion score. The `rank_by_method` method sorts a given set of nodes based on the chosen ranking method. The `color_tree` method assigns colors to nodes based on their ranking using a colormap.
### `expansion_score`
#### Description
Calculates the expansion score for each internal node in the tree. The expansion score represents the rate of lineage expansion over time.
#### Inputs
None
#### Outputs
None
#### Internal Logic
The method first checks if time bins are provided. It then iterates through each internal node and calculates the number of leaves falling within each time bin. It then calculates the temporal frequency of the node within each bin, using a pseudo-count to avoid zero values. Finally, it performs linear regression on the log-transformed temporal frequencies against the time bins to obtain the expansion score (slope of the regression line).
### `sort_leafs_in_time_bins`
#### Description
Groups leaf nodes into their corresponding time bins based on their depth in the tree.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :-------------------------------------------------------------------------------------------------------- |
| node | Node | The node for which to sort its descendant leaves into time bins (default: None, uses all terminal nodes). |
#### Outputs
| Name | Type | Description |
| :------------------ | :--- | :------------------------------------------------------------------------------------------- |
| tmp\_leafs\_by\_bin | list | A list of lists, where each inner list contains leaf nodes belonging to a specific time bin. |
#### Internal Logic
The method first checks if time bins are defined. If a node is provided, it retrieves the descendant terminal nodes of that node; otherwise, it uses all terminal nodes in the tree. It then iterates through each terminal node and assigns it to the appropriate time bin based on its depth.
### `rank_by_method`
#### Description
Sorts a given set of nodes based on a specified ranking method.
#### Inputs
| Name | Type | Description |
| :------- | :--- | :-------------------------------------------------------------------------------------------- |
| nodes | list | List of nodes to be ranked (default: all terminal nodes). |
| method | str | Attribute name representing the ranking method to use (default: 'mean\_fitness'). |
| scramble | bool | Whether to shuffle the nodes before sorting to remove any pre-existing order (default: True). |
#### Outputs
| Name | Type | Description |
| :---- | :--- | :-------------------------------------------------------------- |
| nodes | list | The sorted list of nodes based on the specified ranking method. |
#### Internal Logic
The method first checks if a list of nodes is provided; otherwise, it uses all terminal nodes. It optionally shuffles the nodes to randomize the order. Then, it sorts the nodes in descending order based on the values of the specified ranking method attribute. Finally, it assigns a rank to each node based on its position in the sorted list.
### `correlation_between_scores`
#### Description
Calculates the Spearman's rank correlation between two specified node attributes.
#### Inputs
| Name | Type | Description |
| :-------- | :--- | :----------------------------------------------------------------------------------- |
| score1 | str | Attribute name of the first score to use for correlation calculation. |
| score2 | str | Attribute name of the second score to use for correlation calculation. |
| node\_set | list | List of nodes to consider for correlation calculation (default: all terminal nodes). |
#### Outputs
| Name | Type | Description |
| :---------- | :---- | :------------------------------------------------------------------------------------------------------------- |
| correlation | float | The Spearman's rank correlation coefficient between the two specified scores. Returns None if an error occurs. |
#### Internal Logic
The method retrieves the values of the two specified attributes for each node in the provided node set. It then calculates the Spearman's rank correlation coefficient between the two sets of values using the `spearmanr` function from the `scipy.stats` module.
### `ranking_quality`
#### Description
Evaluates the quality of the ranking based on the mean fitness score. It determines the rank of the lowest terminal node that falls within one standard deviation of the highest mean fitness score.
#### Inputs
None
#### Outputs
| Name | Type | Description |
| :--- | :--- | :----------------------------------------------------------------------------------------- |
| ci | int | The rank of the lowest terminal node within one standard deviation of the top-ranked node. |
#### Internal Logic
The method sorts the terminal nodes based on their mean fitness in descending order. It then calculates the cutoff value as the mean fitness of the top-ranked node minus one standard deviation. It iterates through the sorted nodes and increments a counter until it encounters a node with a mean fitness score below the cutoff value. The final counter value represents the rank of the last node within the one standard deviation range.
### `color_tree`
#### Description
Assigns colors to nodes in the tree based on their ranking using a specified colormap.
#### Inputs
| Name | Type | Description |
| :-------- | :------------------------- | :--------------------------------------------------------------------------------------------- |
| nodes | list | List of nodes to color (default: all terminal nodes). |
| method | str | Attribute name representing the ranking method to use for coloring (default: 'mean\_fitness'). |
| offset | float | Value to extend the branch length of terminal nodes for better visibility (default: 0.0). |
| n\_labels | int | Number of top-ranked nodes to display rank labels (default: 10). |
| scramble | bool | Whether to shuffle the nodes before coloring to remove any pre-existing order (default: True). |
| cmap | matplotlib.colors.Colormap | The colormap to use for assigning colors (default: matplotlib.cm.jet). |
#### Outputs
None
#### Internal Logic
The method first ranks the provided nodes based on the specified ranking method. It then iterates through the ranked nodes and assigns a color to each node based on its rank using the provided colormap. It also extends the branch length of terminal nodes for better visibility. Additionally, it adds rank labels to the top-ranked nodes.
### `interpolate_color`
#### Description
Assigns colors to internal nodes by averaging the colors of their descendant leaf nodes.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :----------------------------------------------------------------- |
| tree | Tree | The tree object to color (default: the tree stored in the object). |
#### Outputs
None
#### Internal Logic
The method iterates through each internal node in the tree. For each internal node, it retrieves its descendant leaf nodes and calculates the average color by summing the RGB values of each leaf node's color and dividing by the number of leaves. It then assigns the averaged color to the internal node.
### `color_other_tree`
#### Description
Transfers the ranking scores from the current tree to another tree based on matching node names and then colors the other tree.
#### Inputs
| Name | Type | Description |
| :-------- | :------------------------- | :--------------------------------------------------------------------------------------------- |
| nodes | list | List of nodes in the other tree to color. |
| method | str | Attribute name representing the ranking method to use for coloring (default: 'mean\_fitness'). |
| offset | float | Value to extend the branch length of terminal nodes for better visibility (default: 0.0). |
| n\_labels | int | Number of top-ranked nodes to display rank labels (default: 10). |
| scramble | bool | Whether to shuffle the nodes before coloring to remove any pre-existing order (default: True). |
| cmap | matplotlib.colors.Colormap | The colormap to use for assigning colors (default: matplotlib.cm.jet). |
#### Outputs
None
#### Internal Logic
The method first creates a dictionary mapping node names to nodes in the current tree. It then iterates through the nodes in the other tree and checks if a matching node exists in the current tree based on their names. If a match is found, it assigns the ranking score from the matching node in the current tree to the node in the other tree. Finally, it calls the `color_tree` method to color the nodes in the other tree based on the transferred ranking scores.
### `best_node`
#### Description
Returns the highest-ranked node from a given set of nodes based on a specified ranking method.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :-------------------------------------------------------------------------------- |
| method | str | Attribute name representing the ranking method to use (default: 'mean\_fitness'). |
| nodes | list | List of nodes to consider (default: all terminal nodes). |
#### Outputs
| Name | Type | Description |
| :--------- | :--- | :--------------------------------------------------------------------- |
| best\_node | Node | The node with the highest ranking score based on the specified method. |
#### Internal Logic
The method first ranks the provided nodes based on the specified ranking method. It then returns the first node in the sorted list, which represents the node with the highest ranking score.
### `rank_labels`
#### Description
Retrieves the rank label of a given node.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :--------------------------------------------- |
| node | Node | The node for which to retrieve the rank label. |
#### Outputs
| Name | Type | Description |
| :---------- | :--- | :-------------------------------------------------------------------------- |
| rank\_label | str | The rank label of the node, or None if the node does not have a rank label. |
#### Internal Logic
The method attempts to access the `rank_label` attribute of the provided node. If the attribute exists, it returns its value; otherwise, it returns None.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/overview
## High-level description
This directory contains a collection of Python modules that implement various algorithms and utilities for fitness estimation and phylogenetic tree analysis. The main components include:
1. Ancestral sequence reconstruction
2. Fitness inference on phylogenetic trees
3. Node ranking and coloring in trees
4. Sequence alignment processing and tree building
5. Survival analysis and generating function solutions
6. Tree manipulation and visualization utilities
## What does it do?
The code in this directory provides a comprehensive toolkit for analyzing evolutionary fitness and phylogenetic relationships in biological sequences. It allows researchers to:
1. Reconstruct ancestral sequences from a phylogenetic tree and multiple sequence alignment.
2. Infer fitness distributions for nodes in a phylogenetic tree using a message-passing algorithm.
3. Rank and color nodes in a tree based on various metrics, including inferred fitness.
4. Process sequence alignments, build phylogenetic trees, and annotate them with mutations.
5. Solve generating functions for branching processes with diffusive fitness changes.
6. Manipulate, visualize, and analyze phylogenetic trees with various utility functions.
These tools can be used to study the evolutionary dynamics of populations, predict fitness of sequences, and visualize evolutionary relationships with additional context provided by fitness estimates and other metrics.
## Entry points
The main entry points for using this toolkit are:
1. `sequence_ranking.py`: This module combines sequence alignment processing with node ranking and fitness inference. It provides a high-level interface for analyzing sequence data and predicting fitness.
2. `fitness_inference.py`: This module implements the core algorithm for inferring fitness distributions on a phylogenetic tree. It can be used directly for fitness analysis on pre-built trees.
3. `node_ranking.py`: This module extends the fitness inference functionality to provide various methods for ranking and coloring nodes in a tree. It's useful for visualizing and interpreting the results of fitness inference.
## Key Files
1. `ancestral.py`: Implements maximum likelihood estimation for ancestral sequence reconstruction.
2. `fitness_inference.py`: Contains the main algorithm for inferring fitness distributions on phylogenetic trees.
3. `node_ranking.py`: Extends fitness inference with methods for ranking and coloring tree nodes.
4. `sequence_ranking.py`: Combines sequence alignment processing with node ranking and fitness inference.
5. `solve_survival.py`: Implements numerical solvers for generating functions in branching processes.
6. `tree_utils.py`: Provides utility functions for building, manipulating, and visualizing phylogenetic trees.
## Dependencies
The code relies on several external libraries:
1. Biopython: Used extensively for handling biological sequences, alignments, and phylogenetic trees.
2. NumPy: Used for numerical computations and array manipulations.
3. SciPy: Used for various scientific computing tasks, including integration and statistical functions.
4. Matplotlib: Used for visualization of phylogenetic trees and results.
Additionally, the code assumes the availability of external tools like `fasttree` for phylogenetic tree construction.
## Configuration
The code does not use explicit configuration files. However, many of the classes and functions accept parameters that can be used to configure their behavior. For example:
* In `fitness_inference.py`, the `fitness_inference` class accepts parameters like `eps_branch_length`, `D`, `fit_grid`, `samp_frac`, and `mem` to configure the fitness inference process.
* In `sequence_ranking.py`, the `alignment` class accepts parameters like `outgroup`, `cds`, and `collapse` to configure the alignment processing and tree building.
Users can adjust these parameters when initializing the respective classes to customize the analysis for their specific needs.
# sequence_ranking.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/sequence_ranking.py
## High-level description
The code defines two classes, `alignment` and `sequence_ranking`, for analyzing and ranking sequences within a phylogenetic tree. The `alignment` class processes sequence data, calculates allele frequencies, and builds a phylogenetic tree. The `sequence_ranking` class extends the `node_ranking` class and uses the processed alignment data to predict and rank nodes based on fitness.
## Code Structure
The `alignment` class is responsible for handling sequence data, while the `sequence_ranking` class utilizes this data for ranking nodes in a phylogenetic tree. The `sequence_ranking` class inherits from the `node_ranking` class, which provides functionalities for ranking nodes based on different methods.
## References
* `tree_utils`: This module is used for tree building and manipulation functions.
* `node_ranking`: The `sequence_ranking` class inherits from this class, utilizing its ranking functionalities.
## Symbols
### `alignment`
#### Description
This class stores and processes sequence alignments, calculates allele frequencies, and builds phylogenetic trees.
#### Inputs
| Name | Type | Description |
| :---------- | :----------------------------- | :--------------------------------------------------------------------------- |
| aln | Bio.Align.MultipleSeqAlignment | A Biopython MultipleSeqAlignment object representing the sequence alignment. |
| outgroup | str | The outgroup sequence used for rooting the phylogenetic tree. |
| cds | dict | A dictionary specifying the coding region within the alignment. |
| collapse | bool | Flag indicating whether to collapse zero-length branches in the tree. |
| build\_tree | bool | Flag indicating whether to build a phylogenetic tree from the alignment. |
#### Outputs
The class does not have explicit return values but stores processed data and the constructed tree internally.
#### Internal Logic
1. **Initialization:** Stores input parameters, sets default values, and calls `process_alignment`.
2. **`process_alignment`:** Calculates alignment summary information, consensus sequence, allele frequencies, and optionally translates the alignment if it's a protein sequence. It then builds the phylogenetic tree using `tree_utils.calculate_tree`.
3. **`calculate_allele_frequencies`:** Computes the frequency of each nucleotide/amino acid at each position in the alignment.
4. **`calculate_aa_allele_frequencies`:** Specifically calculates amino acid frequencies for protein alignments.
5. **`translate_alignment`:** Translates the nucleotide alignment to an amino acid alignment based on the provided coding region.
6. **`mean_distance_to_sequence`:** Calculates the average Hamming distance between a query sequence and the alignment based on allele frequencies.
7. **`mean_distance_to_set`:** Calculates the average Hamming distance between two alignments based on their allele frequencies.
8. **`aa_distance_to_sequence`:** Calculates the average Hamming distance between a query amino acid sequence and the alignment.
9. **`aa_distance_to_set`:** Calculates the average Hamming distance between two amino acid alignments.
10. **`build_tree`:** Constructs a phylogenetic tree using the alignment and outgroup sequence, infers ancestral sequences, and optionally collapses zero-length branches.
### `sequence_ranking`
#### Description
This class extends the `node_ranking` class to handle sequence data. It builds a phylogenetic tree from the provided alignment and uses the ranking functionalities inherited from `node_ranking` to predict and rank nodes based on fitness.
#### Inputs
| Name | Type | Description |
| :-------------- | :-------- | :--------------------------------------------------------------------------- |
| sequence\_data | alignment | An instance of the `alignment` class containing the processed sequence data. |
| distance\_scale | float | A scaling factor for the branch lengths in the phylogenetic tree. |
| \*args | tuple | Additional arguments passed to the `node_ranking` constructor. |
| \*\*kwargs | dict | Additional keyword arguments passed to the `node_ranking` constructor. |
#### Outputs
The class does not have explicit return values but stores the ranking results internally.
#### Internal Logic
1. **Initialization:** Calls the `node_ranking` constructor, stores the input data and parameters, calculates the coalescence time scale, and sets the phylogenetic tree.
2. **`predict`:** Initializes the fitness inference process, computes node rankings, and returns the highest-ranked external node based on the specified ranking method.
## Dependencies
* Biopython (Bio)
* NumPy (numpy)
* SciPy (scipy)
## Error Handling
The code includes basic error handling using `try-except` blocks, primarily for handling potential translation errors and missing attributes. However, it does not implement specific error types or custom exception handling.
# solve_survival.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/solve_survival.py
## High-level description
This code implements a numerical solver for the generating function of a branching process, assuming diffusive changes in fitness over time. It provides solutions for the lineage propagator, considering the non-sampling constraint.
## Code Structure
The code defines a class `survival_gen_func` that encapsulates the logic for solving the generating function and lineage propagator. It uses numerical integration methods like `integrate_rk4` (a 4th order Runge-Kutta implementation) and `scipy.integrate.odeint`. The class stores precomputed solutions in the `phi_solutions` dictionary for efficiency.
## Symbols
### `survival_gen_func`
#### Description
This class provides methods to solve for the generating function of a branching process and the lineage propagator, considering non-branching.
#### Inputs
| Name | Type | Description |
| :------------ | :---------------------- | :----------------------------------------------------------------------- |
| fitness\_grid | numpy.ndarray, optional | Discretization grid for fitness values. If None, a default grid is used. |
#### Outputs
The class itself does not return any output. Its methods return the computed solutions.
#### Internal Logic
The class initializes with a fitness grid and precomputes values for numerical efficiency. It stores solutions in the `phi_solutions` dictionary. The main methods are:
* `integrate_phi`: Solves the generating function equation for given parameters.
* `dphi`: Computes the time derivative of the generating function.
* `integrate_prop_odeint`: Integrates the lineage propagator using `scipy.integrate.odeint`.
* `integrate_prop`: Integrates the lineage propagator using the custom `integrate_rk4` method.
* `dprop_backward`: Computes the time derivative of the propagator.
### `integrate_rk4`
#### Description
This function implements a 4th order Runge-Kutta integration scheme with a fixed time step.
#### Inputs
| Name | Type | Description |
| :--- | :------------ | :------------------------------------------------------------ |
| df | function | Function that returns the derivative of the system. |
| f0 | numpy.ndarray | Initial conditions for the integration. |
| T | numpy.ndarray | Time points at which to evaluate the solution. |
| dt | float | Time step for the integration. |
| args | tuple | Additional arguments to pass to the derivative function `df`. |
#### Outputs
| Name | Type | Description |
| :--- | :------------ | :-------------------------------------------------------- |
| sol | numpy.ndarray | Solution of the integration at the specified time points. |
### `non_negativity_cutoff`
#### Description
A global variable defining the minimum allowed value for the solutions, used to avoid numerical issues with very small or negative values.
## Error Handling
The code uses `np.maximum` to ensure non-negativity of the solutions, replacing values below `non_negativity_cutoff` with the cutoff value. It also includes checks for valid time ranges within the `dprop_backward` method and prints warning messages if necessary.
## Performance Considerations
The code uses a custom `integrate_rk4` implementation instead of `scipy.integrate.odeint` for the lineage propagator integration, claiming better performance for systems with different convergence criteria. The `phi_solutions` dictionary stores precomputed generating function solutions to avoid redundant computations.
# tree_utils.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/prediction_src/tree_utils.py
## High-level description
The `tree_utils.py` file provides a collection of functions for building, manipulating, annotating, and visualizing phylogenetic trees. These functions are primarily used for analyzing and understanding the evolutionary relationships between sequences, particularly in the context of fitness prediction.
## Code Structure
This file defines a set of independent utility functions that operate on phylogenetic trees represented using the Biopython library's `Phylo` objects. Some functions utilize external tools like `fasttree` for tree building.
## References
This code references the Biopython library (`Bio`) extensively for handling sequences, alignments, and phylogenetic trees. It also uses the `matplotlib` library for visualization purposes.
## Symbols
### `calculate_tree`
#### Description
Builds a phylogenetic tree from a given sequence alignment and outgroup sequence using the `fasttree` program. It infers ancestral sequences and performs basic tree manipulation like rooting and ladderizing.
#### Inputs
| Name | Type | Description |
| :-------- | :----------------------------- | :-------------------------------------------------------------------- |
| aln | Bio.Align.MultipleSeqAlignment | A Biopython alignment object containing the sequences. |
| outgroup | Bio.SeqRecord.SeqRecord | A Biopython SeqRecord object representing the outgroup sequence. |
| ancestral | bool | Flag indicating whether to infer ancestral sequences (default: True). |
#### Outputs
| Name | Type | Description |
| :-------------- | :---------------------- | :---------------------------------------------------------------------------------- |
| biopython\_tree | Bio.Phylo.BaseTree.Tree | A rooted and ladderized Biopython tree object constructed from the input alignment. |
#### Internal Logic
1. Writes the alignment and outgroup sequence to a temporary FASTA file.
2. Executes `fasttree` with the temporary file as input.
3. Parses the `fasttree` output into a Biopython `Phylo.Tree` object.
4. Roots the tree using the provided outgroup.
5. Ladderizes the tree for visual clarity.
6. If `ancestral` is True, infers ancestral sequences for internal nodes.
7. Removes the temporary file.
### `branch_label`
#### Description
Generates a label for a tree branch based on the mutations occurring along that branch.
#### Inputs
| Name | Type | Description |
| :----------------- | :-------------------- | :--------------------------------------------------------------------------------------------------------------------------------- |
| node | Bio.Phylo.Clade.Clade | The tree node representing the end of the branch to be labeled. |
| aa | bool | Flag indicating whether to use amino acid or nucleotide sequences for labeling (default: True). |
| display\_positions | list | Optional list of positions to consider for labeling. If provided, only mutations at these positions will be included in the label. |
#### Outputs
| Name | Type | Description |
| :---- | :--- | :------------------------------------------------------------------------------------------------------------------------------------------------ |
| label | str | A string representing the mutations along the branch, formatted as a series of underscore-separated mutation descriptions (e.g., "A123T\_G456C"). |
#### Internal Logic
1. Determines whether to use amino acid (`node.aa_mutations`) or nucleotide (`node.mutations`) mutations based on the `aa` flag.
2. If `display_positions` is provided, filters the mutations to include only those occurring at the specified positions.
3. Formats the selected mutations into a string representation.
### `collapse_zero_branches`
#### Description
Collapses branches in the tree where the sequences at the parent and child nodes are identical. This simplifies the tree by merging nodes with no evolutionary changes.
#### Inputs
| Name | Type | Description |
| :---- | :-------------------- | :--------------------------------------- |
| clade | Bio.Phylo.Clade.Clade | The root node of the subtree to process. |
#### Internal Logic
1. Recursively traverses the tree.
2. For each internal node, compares its sequence to its children's sequences.
3. If a child node has the same sequence as its parent, collapses the child into the parent.
### `annotate_leaf`
#### Description
Adds annotations to a leaf node in the tree.
#### Inputs
| Name | Type | Description |
| :--------- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------- |
| leaf | Bio.Phylo.Clade.Clade | The leaf node to annotate. |
| annotation | dict | A dictionary where keys represent annotation attributes and values are the corresponding values to be assigned to the leaf node. |
#### Internal Logic
Iterates through the `annotation` dictionary and sets the attributes of the `leaf` node using the provided key-value pairs.
### `translate_sequences_on_tree`
#### Description
Translates nucleotide sequences to amino acid sequences for all nodes in the tree.
#### Inputs
| Name | Type | Description |
| :--- | :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| T | Bio.Phylo.BaseTree.Tree | The phylogenetic tree object. |
| cds | dict | A dictionary specifying the coding sequence region with keys: 'begin' (start position), 'pad' (length of padding with 'X' at the beginning), and 'end' (end position). |
#### Internal Logic
1. Iterates through all nodes (internal and terminal) in the tree.
2. Extracts the nucleotide sequence corresponding to the coding region specified by `cds`.
3. Translates the nucleotide sequence to an amino acid sequence, handling potential translation errors by inserting 'X' characters.
4. Stores the translated amino acid sequence in the `aa_seq` attribute of each node.
### `mutations_on_branches`
#### Description
Identifies and annotates mutations occurring along each branch of the tree.
#### Inputs
| Name | Type | Description |
| :--- | :---------------------- | :------------------------------------------------------------------------------------- |
| T | Bio.Phylo.BaseTree.Tree | The phylogenetic tree object. |
| aa | bool | Flag indicating whether to analyze mutations at the amino acid level (default: False). |
#### Internal Logic
1. Initializes an empty list of mutations for the root node.
2. Calls the recursive function `mutations_on_branches_subtree` to traverse the tree.
3. For each branch, compares the sequences of the parent and child nodes to identify mutations.
4. Stores the identified mutations in the `mutations` (and `aa_mutations` if `aa` is True) attribute of the child node.
### `mutations_on_branches_subtree`
#### Description
Recursive helper function for `mutations_on_branches` that traverses the tree and identifies mutations along each branch.
#### Inputs
| Name | Type | Description |
| :------ | :-------------------- | :-------------------------------------------------------------------- |
| subtree | Bio.Phylo.Clade.Clade | The current subtree being processed. |
| aa | bool | Flag indicating whether to analyze mutations at the amino acid level. |
#### Internal Logic
1. Iterates through the child nodes of the current subtree.
2. Compares the nucleotide sequences of the parent and child nodes to identify mutations.
3. If `aa` is True, also compares the amino acid sequences.
4. Stores the identified mutations in the `mutations` and `aa_mutations` attributes of the child node.
5. Recursively calls itself for each child node that is not a terminal node.
### `find_internal_nodes`
#### Description
Finds corresponding internal nodes between two trees (`sourceT` and `destT`) based on the most recent common ancestor (MRCA) of their leaf nodes.
#### Inputs
| Name | Type | Description |
| :------ | :---------------------- | :-------------------- |
| sourceT | Bio.Phylo.BaseTree.Tree | The source tree. |
| destT | Bio.Phylo.BaseTree.Tree | The destination tree. |
#### Internal Logic
1. Creates a lookup table for leaf nodes in `destT`.
2. Calculates the path from the root to each leaf node in `sourceT` within the `destT` tree.
3. Iterates through internal nodes in `sourceT`.
4. For each internal node, finds the MRCA of its leaf nodes in `destT` using the calculated paths.
5. Links the internal nodes in `sourceT` and `destT` by setting their `mirror_node` attributes.
### `label_nodes`
#### Description
Sets labels for specific nodes in the tree based on a provided dictionary.
#### Inputs
| Name | Type | Description |
| :-------------- | :---------------------- | :------------------------------------------------------- |
| T | Bio.Phylo.BaseTree.Tree | The phylogenetic tree object. |
| seqs\_to\_label | dict | A dictionary mapping node names to their desired labels. |
#### Internal Logic
1. Clears any existing labels for all nodes.
2. Iterates through nodes and sets labels based on the provided `seqs_to_label` dictionary.
### `node_label_func`
#### Description
A simple function that attempts to return the `name` attribute of a tree node. Used as a default node label function for tree plotting.
#### Inputs
| Name | Type | Description |
| :--- | :-------------------- | :------------- |
| node | Bio.Phylo.Clade.Clade | The tree node. |
#### Outputs
| Name | Type | Description |
| :---- | :--- | :----------------------------------------------------------------------- |
| label | str | The `name` attribute of the node, or None if the attribute is not found. |
### `erase_color`
#### Description
Resets the color of all nodes in the tree to None.
#### Inputs
| Name | Type | Description |
| :--- | :---------------------- | :---------------------------- |
| tree | Bio.Phylo.BaseTree.Tree | The phylogenetic tree object. |
#### Internal Logic
Iterates through all nodes and sets the `color` attribute to None.
### `plot_prediction_tree`
#### Description
Visualizes a phylogenetic tree with nodes colored according to a prediction.
#### Inputs
| Name | Type | Description |
| :---------------- | :----------------------- | :------------------------------------------------------------------------ |
| prediction | object | An object containing prediction data and methods for coloring the tree. |
| method | str | The method used for coloring the tree (default: 'mean\_fitness'). |
| internal | bool | Flag indicating whether to color internal nodes (default: False). |
| fig | matplotlib.figure.Figure | Optional matplotlib figure object. |
| axes | matplotlib.axes.Axes | Optional matplotlib axes object. |
| cb | bool | Flag indicating whether to display a colorbar (default: True). |
| scalebar | bool | Flag indicating whether to display a scale bar (default: True). |
| offset | float | Offset value for color scaling (default: 0.0001). |
| node\_label\_func | function | Function to generate node labels (default: a function that returns None). |
#### Internal Logic
1. Colors the tree using the provided `prediction` object and specified `method`.
2. Draws the tree using the `draw_tree` function.
### `draw_tree`
#### Description
Draws a phylogenetic tree using matplotlib.
#### Inputs
| Name | Type | Description |
| :------------ | :------------------------- | :--------------------------------------------------------------- |
| tree | Bio.Phylo.BaseTree.Tree | The phylogenetic tree object. |
| node\_label | function | Function to generate node labels (default: `node_label_func`). |
| branch\_label | function | Function to generate branch labels (default: `node_label_func`). |
| cmap | matplotlib.colors.Colormap | The colormap to use for coloring branches (default: `cm.jet`). |
| fig | matplotlib.figure.Figure | Optional matplotlib figure object. |
| axes | matplotlib.axes.Axes | Optional matplotlib axes object. |
| cb | bool | Flag indicating whether to display a colorbar (default: True). |
| scalebar | bool | Flag indicating whether to display a scale bar (default: True). |
#### Internal Logic
1. Creates a matplotlib figure and axes if not provided.
2. Uses Biopython's `Phylo.draw` function to render the tree.
3. Adds a scale bar and colorbar if enabled.
### `plot_combined_tree`
#### Description
Plots a combined tree with nodes colored based on predictions from a separate dataset.
#### Inputs
| Name | Type | Description |
| :------------- | :------------------- | :---------------------------------------------------------------------- |
| prediction | object | An object containing prediction data and methods for coloring the tree. |
| combined\_data | object | An object containing the combined tree and data. |
| method | str | The method used for coloring the tree (default: 'mean\_fitness'). |
| internal | bool | Flag indicating whether to color internal nodes (default: False). |
| axes | matplotlib.axes.Axes | Optional matplotlib axes object. |
| cb | bool | Flag indicating whether to display a colorbar (default: True). |
| offset | float | Offset value for color scaling (default: 0.0001). |
#### Internal Logic
1. Colors the combined tree using predictions from the `prediction` object.
2. Draws the tree using the `draw_tree` function.
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------------ |
| biopython | Handling sequences, alignments, and phylogenetic trees. |
| matplotlib | Visualization of phylogenetic trees. |
## TODOs
There are no TODOs or other notes left in the code.
# rank_sequences.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/FitnessInference/rank_sequences.py
## High-level description
This script performs fitness inference on sequences in a multiple sequence alignment (MSA) to rank them based on their inferred fitness. It uses a phylogenetic approach, reconstructing the ancestral relationships between sequences and estimating the fitness of each sequence based on its position in the tree.
The script takes an alignment file in FASTA format and the name of an outgroup sequence as input. It then infers the phylogenetic tree, estimates fitness values for each sequence (including ancestral sequences), and outputs the tree, ancestral sequences, and sequence rankings to files.
## Code Structure
The code first parses command-line arguments, then reads the input alignment and identifies the outgroup sequence. It then sets up a `sequence_ranking` object from the `sequence_ranking.py` module (see References) and uses it to perform the fitness inference. Finally, it outputs the results to files and optionally plots the tree.
## References
This script references the following code symbols:
* `sequence_ranking` (from `sequence_ranking.py`): This class handles the core logic of sequence ranking based on phylogenetic analysis.
## Symbols
### `ofunc`
#### Description
This function determines the appropriate file opening method based on the file extension. If the filename ends with '.gz', it assumes a gzipped file and uses `gzip.open`. Otherwise, it uses the standard `open` function.
#### Inputs
| Name | Type | Description |
| :---- | :--- | :--------------------------------------------------------------------------- |
| fname | str | The name of the file to open. |
| mode | str | The mode in which to open the file (e.g., 'r' for reading, 'w' for writing). |
#### Outputs
| Name | Type | Description |
| :---------- | :--- | :------------------------------------------ |
| file object | file | A file object representing the opened file. |
### `main execution block`
#### Description
This block of code executes the main logic of the script. It reads the alignment file, identifies the outgroup sequence, sets up the sequence data, performs the fitness inference, and outputs the results.
#### Internal Logic
1. **Read alignment and outgroup:** Reads the input alignment file in FASTA format and identifies the outgroup sequence.
2. **Set up sequence data:** Creates a `sequence_ranking` object from the `alignment` module, providing it with the alignment and outgroup information.
3. **Perform fitness inference:** Uses the `sequence_ranking` object to perform the fitness inference, including tree reconstruction and fitness estimation.
4. **Output results:** Writes the reconstructed tree, inferred ancestral sequences, and sequence rankings to files.
5. **Optional plotting:** If the `--plot` flag is set, plots the tree with fitness information.
## Dependencies
| Dependency | Purpose |
| :---------------- | :--------------------------------------------------------------------------- |
| argparse | Parsing command-line arguments. |
| matplotlib | Plotting the phylogenetic tree (optional). |
| Bio | Handling biological sequence data and phylogenetic trees. |
| tree\_utils | Utility functions for working with phylogenetic trees. |
| sequence\_ranking | Custom module containing the `sequence_ranking` class for fitness inference. |
### Configuration
This script uses command-line arguments for configuration. See the "parse the command line arguments" section for details on each argument.
## Error Handling
The script checks if the specified outgroup sequence is present in the alignment. If not, it prints an error message and exits.
## Logging
The script prints the time scale (`tau`) used for local tree length estimation.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/__init__.py
empty here
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/__init__.py
empty here
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/overview
## High-level description
This directory contains the implementation of a beta coalescent tree simulator and a Site Frequency Spectrum (SFS) calculator. The code is designed to model and analyze the evolutionary relationships and genetic diversity in populations using coalescent theory and beta coalescent processes. The main functionality is implemented in the `src` subdirectory, while the `resources` directory appears to be empty or contains placeholder files.
## What does it do?
The code in this directory simulates the evolutionary history of genetic sequences under different coalescent models, particularly the beta coalescent model. It generates genealogical trees that represent the ancestral relationships between individuals in a population sample. Using these trees, it calculates the Site Frequency Spectrum (SFS), which is a summary statistic of genetic variation in the population. The SFS represents the distribution of allele frequencies in the sample, providing insights into the population's genetic diversity and evolutionary history.
## Key Files
1. `betatree.py`: This file contains the `betatree` class, which is responsible for simulating beta coalescent trees. It implements methods for tree initialization, coalescence events, and tree structure manipulation.
2. `sfs.py` and `sfs_py3.py`: These files define the `SFS` class, which extends the functionality of `betatree` to calculate the Site Frequency Spectrum. The `SFS` class generates multiple trees, accumulates allele frequency information, and computes the SFS. It also provides methods for binning the SFS and saving/loading SFS data.
## Dependencies
The code relies on several external libraries:
* NumPy: Used for numerical computations and array operations.
* SciPy: Specifically, `scipy.special` is used for special mathematical functions like the gamma function.
* BioPython: The `Bio.Phylo` module is used for working with phylogenetic trees.
* Matplotlib: Used for plotting the SFS (in example usage).
## Configuration
The code does not use explicit configuration files. Instead, key parameters are passed as arguments to the class constructors:
* `sample_size`: The number of individuals in the sample.
* `alpha`: The alpha parameter of the beta coalescent model (default is 2, which corresponds to the Kingman coalescent).
These parameters can be adjusted when initializing the `betatree` or `SFS` objects to simulate different evolutionary scenarios.
The `SFS` class also allows for configuration of the SFS calculation and binning process through method parameters:
* `ntrees`: The number of trees to generate for SFS calculation.
* `mode`: The binning mode for the SFS (linear, log, or logit).
* `bins`: The number of bins or custom bin edges for SFS binning.
These configurations allow users to fine-tune the SFS calculation and analysis based on their specific research needs.
# __init__.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/src/__init__.py
empty here
# betatree.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/src/betatree.py
## High-level description
This code defines a `betatree` class that simulates a beta coalescent tree, a model used in population genetics to represent the ancestral relationships of a sample of individuals. The class allows for the generation of trees under different coalescent models, including the Kingman coalescent (α=2) and the Bolthausen-Sznitman coalescent (α=1).
## Code Structure
The `betatree` class contains methods for initializing the tree structure, simulating coalescence events, calculating waiting times between mergers, determining the number of lineages involved in each merger, and cleaning up the tree structure for visualization. The `__main__` section provides examples of how to use the class to generate and visualize trees under different coalescent models.
## Symbols
### `betatree`
#### Description
This class simulates a beta coalescent tree. It provides methods for initializing the tree, simulating coalescence events, and visualizing the resulting tree.
#### Inputs
| Name | Type | Description |
| :----------- | :---- | :------------------------------------------------------------------------------ |
| sample\_size | int | The number of leaves in the tree, representing the sample size. |
| alpha | float | The parameter of the beta coalescent model. Defaults to 2 (Kingman coalescent). |
#### Outputs
The class does not directly return any values. It modifies its internal state to represent the simulated tree.
#### Internal Logic
The `coalesce` method drives the simulation process:
1. **Initialization:** Creates a list of `Clade` objects, each representing a single lineage.
2. **Coalescence loop:** Iteratively performs coalescence events until a single root lineage remains.
* `waiting_time`: Calculates the time until the next coalescence event based on the current number of lineages and the `alpha` parameter.
* `whichp`: Determines the number of lineages involved in the next merger event.
* `coalescence_event`: Selects lineages to merge, updates branch lengths, and creates a new parent `Clade`.
3. **Cleanup:** Adjusts branch lengths to reflect coalescence times and calculates the number of descendants for each node.
#### Side Effects
The `coalesce` method modifies the internal state of the `betatree` object, constructing a tree structure represented by interconnected `Clade` objects.
### `init_tree`
#### Description
Initializes the tree structure by creating a list of `Clade` objects, each representing a single leaf node.
#### Inputs
None
#### Outputs
None
#### Internal Logic
Creates a list of `Phylo.BaseTree.Clade` objects, each with a unique name and an initial branch length of 0.
### `coalescence_event`
#### Description
Simulates a single coalescence event, merging a random subset of lineages.
#### Inputs
None
#### Outputs
None
#### Internal Logic
1. `whichp`: Determines the number of lineages to merge.
2. `waiting_time`: Calculates the time elapsed until this event.
3. Updates the branch lengths of all existing lineages by the waiting time.
4. Randomly selects lineages to merge.
5. `merge_clades`: Creates a new parent `Clade` representing the merged lineages.
### `merge_clades`
#### Description
Merges a given set of lineages into a new parent `Clade`.
#### Inputs
| Name | Type | Description |
| :-------------- | :--- | :---------------------------------------------------- |
| merging\_blocks | list | A list of indices representing the lineages to merge. |
#### Outputs
None
#### Internal Logic
1. Creates a new `Phylo.BaseTree.Clade` object.
2. Assigns the selected lineages as children of the new `Clade`.
3. Sets the branch length of the new `Clade` to the current branch length of its children.
4. Removes the merged lineages from the list of active lineages.
5. Adds the new parent `Clade` to the list of active lineages.
### `clean_up_subtree`
#### Description
Recursively traverses the tree structure to calculate branch lengths and the number of descendants for each node.
#### Inputs
| Name | Type | Description |
| :---- | :------------------- | :----------------------------------- |
| clade | Phylo.BaseTree.Clade | The current `Clade` being processed. |
#### Outputs
None
#### Internal Logic
1. If the `Clade` is a leaf node, sets its weight to 1 and returns.
2. If the `Clade` is an internal node:
* Subtracts the branch length of its first child from its own branch length to avoid double-counting.
* Recursively calls `clean_up_subtree` for each child node.
* Calculates its own weight as the sum of its children's weights.
### `waiting_time`
#### Description
Calculates the waiting time until the next coalescence event based on the current number of lineages and the `alpha` parameter.
#### Inputs
None
#### Outputs
| Name | Type | Description |
| :--- | :---- | :------------------------------------------------- |
| dt | float | The waiting time until the next coalescence event. |
#### Internal Logic
The waiting time is drawn from an exponential distribution with a rate parameter determined by the coalescent model:
* **Kingman coalescent (α=2):** Rate = 2 / (b \* (b - 1)), where b is the number of lineages.
* **Bolthausen-Sznitman coalescent (α=1):** Rate = 1 / (b - 1).
* **General beta coalescent (1 \< α \< 2):** Rate is calculated using gamma functions and the `normalizer` precomputed during initialization.
### `whichp`
#### Description
Determines the number of lineages involved in the next merger event based on the coalescent model.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------------ |
| b | int | The current number of lineages. |
#### Outputs
| Name | Type | Description |
| :--- | :------------------------------------------------- | :---------- |
| int | The number of lineages to merge in the next event. | |
#### Internal Logic
* **Kingman coalescent (α=2):** Always merges 2 lineages.
* **Bolthausen-Sznitman coalescent (α=1):** Samples the merger size from a distribution proportional to 1 / (k \* (k - 1)).
* **General beta coalescent (1 \< α \< 2):** Samples the merger size from a distribution calculated using gamma functions and precomputed values.
## Dependencies
| Dependency | Purpose |
| :------------ | :----------------------------------------------------- |
| random | Used for random number generation. |
| numpy | Used for numerical operations and array manipulation. |
| scipy.special | Used for the gamma function. |
| Bio.Phylo | Used for creating and manipulating phylogenetic trees. |
## Error Handling
The code does not implement specific error handling beyond basic exception raising.
## Logging
The code does not implement logging.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/src/overview
## High-level description
This directory contains the implementation of a beta coalescent tree simulator and a Site Frequency Spectrum (SFS) calculator. The code is designed to model and analyze the evolutionary relationships and genetic diversity in populations using coalescent theory and beta coalescent processes.
## What does it do?
The code in this directory simulates the evolutionary history of genetic sequences under different coalescent models, particularly the beta coalescent model. It generates genealogical trees that represent the ancestral relationships between individuals in a population sample. Using these trees, it calculates the Site Frequency Spectrum (SFS), which is a summary statistic of genetic variation in the population. The SFS represents the distribution of allele frequencies in the sample, providing insights into the population's genetic diversity and evolutionary history.
## Key Files
1. `betatree.py`: This file contains the `betatree` class, which is responsible for simulating beta coalescent trees. It implements methods for tree initialization, coalescence events, and tree structure manipulation.
2. `sfs.py` and `sfs_py3.py`: These files define the `SFS` class, which extends the functionality of `betatree` to calculate the Site Frequency Spectrum. The `SFS` class generates multiple trees, accumulates allele frequency information, and computes the SFS. It also provides methods for binning the SFS and saving/loading SFS data.
## Dependencies
The code relies on several external libraries:
* NumPy: Used for numerical computations and array operations.
* SciPy: Specifically, `scipy.special` is used for special mathematical functions like the gamma function.
* BioPython: The `Bio.Phylo` module is used for working with phylogenetic trees.
* Matplotlib: Used for plotting the SFS (in example usage).
## Configuration
The code does not use explicit configuration files. Instead, key parameters are passed as arguments to the class constructors:
* `sample_size`: The number of individuals in the sample.
* `alpha`: The alpha parameter of the beta coalescent model (default is 2, which corresponds to the Kingman coalescent).
These parameters can be adjusted when initializing the `betatree` or `SFS` objects to simulate different evolutionary scenarios.
The `SFS` class also allows for configuration of the SFS calculation and binning process through method parameters:
* `ntrees`: The number of trees to generate for SFS calculation.
* `mode`: The binning mode for the SFS (linear, log, or logit).
* `bins`: The number of bins or custom bin edges for SFS binning.
These configurations allow users to fine-tune the SFS calculation and analysis based on their specific research needs.
# sfs.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/src/sfs.py
## High-level description
This code defines a class `SFS` that simulates the evolution of genetic sequences under a beta coalescent model and calculates the Site Frequency Spectrum (SFS) from the generated genealogical trees. The SFS represents the distribution of allele frequencies in a population sample.
## Code Structure
The `SFS` class inherits from the `betatree` class, which is responsible for generating the beta coalescent trees. The `SFS` class extends this functionality by accumulating allele frequency information from multiple trees and calculating the SFS. It also provides methods for binning and saving the calculated SFS.
## References
This code references the `betatree` class from the same package, specifically the `coalesce` method for generating the trees and accessing tree nodes for allele frequency calculations. It also uses the `Bio.Phylo` module for working with phylogenetic trees.
## Symbols
### Symbol Name: `SFS`
#### Description:
This class simulates the evolution of genetic sequences under a beta coalescent model and calculates the Site Frequency Spectrum (SFS).
#### Inputs:
| Name | Type | Description |
| :----------- | :---- | :------------------------------------------------------------- |
| sample\_size | int | The number of individuals in the sample. |
| alpha | float | The alpha parameter of the beta coalescent model (default: 2). |
#### Outputs:
The class does not have a direct return value. It stores the calculated SFS and related data in its internal attributes.
#### Internal Logic:
1. **Initialization:** Initializes the `SFS` object by calling the constructor of the parent class `betatree` and initializing internal variables to store allele frequencies and the SFS.
2. **Tree Generation and SFS Calculation:** The `glob_trees` method generates multiple beta coalescent trees using the inherited `coalesce` method. For each tree, it extracts the branch lengths and corresponding allele counts (weights) and stores them. The `getSFS` method then uses this accumulated data to calculate the average SFS over all generated trees.
3. **SFS Binning:** The `binSFS` method allows for binning the calculated SFS using different modes (linear, log, logit) and binning schemes. It calculates the bin edges, bin widths, and the binned SFS values.
4. **SFS Input/Output:** The `saveSFS` method saves a calculated SFS to a file, while the `loadSFS` method loads a previously saved SFS from a file.
#### Side Effects:
The `glob_trees` and `getSFS` methods modify the internal state of the `SFS` object by generating trees, accumulating allele frequencies, and calculating the SFS.
### Symbol Name: `logit`
#### Description:
This function calculates the logit of a given value.
#### Inputs:
| Name | Type | Description |
| :--- | :---- | :--------------- |
| x | float | The input value. |
#### Outputs:
| Name | Type | Description |
| :------- | :---- | :---------------------------- |
| logit(x) | float | The logit of the input value. |
#### Internal Logic:
The function calculates the logit using the formula: `log(x / (1 - x))`.
## Dependencies
| Dependency | Purpose |
| :---------------- | :------------------------------------------------------------------------------------ |
| numpy | Numerical operations, array handling. |
| scipy.special | Special functions, including the gamma function used in beta coalescent calculations. |
| Bio.Phylo | Working with phylogenetic trees. |
| matplotlib.pyplot | Plotting the SFS (only in the example usage). |
## Error Handling
The code includes basic error handling when loading an SFS from a file. It checks if the file exists and if the loaded data has the expected one-dimensional shape. If an error occurs, it prints an error message but does not raise an exception.
## Logging
The code does not implement any specific logging mechanisms. It uses `print` statements for informational output.
## API/Interface Reference
This code does not define an explicit API. It provides a class `SFS` with methods for simulating the evolutionary process, calculating the SFS, and interacting with SFS data.
## TODOs
There are no explicit TODOs or notes left in the code.
# sfs_py3.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/betatree/src/sfs_py3.py
## High-level description
The code defines a class `SFS` that simulates the evolution of genetic sequences under a beta coalescent model and calculates the Site Frequency Spectrum (SFS) from the generated genealogical trees. The SFS represents the distribution of allele frequencies in a sample of DNA sequences.
## Code Structure
The `SFS` class inherits from the `betatree` class, which is responsible for generating the beta coalescent trees. The `SFS` class extends this functionality by accumulating allele frequency information from multiple trees to calculate and analyze the SFS.
## References
* `betatree`: The `SFS` class inherits from the `betatree` class, which is defined in the `.betatree` module.
## Symbols
### Symbol Name: `SFS`
#### Description:
This class simulates the evolution of genetic sequences under a beta coalescent model and calculates the Site Frequency Spectrum (SFS).
#### Inputs:
| Name | Type | Description |
| :----------- | :---- | :------------------------------------------------------------- |
| sample\_size | int | The number of individuals in the sample. |
| alpha | float | The alpha parameter of the beta coalescent model (default: 2). |
#### Outputs:
The class does not have a dedicated output method. Instead, it stores the calculated SFS in the `self.sfs` attribute.
#### Internal Logic:
1. **Tree Generation:** The class utilizes the inherited `coalesce()` method from the `betatree` class to generate multiple beta coalescent trees.
2. **Allele Frequency Calculation:** For each tree, the code traverses the branches and records the number of descendants (weight) and branch length for each node. This information represents the allele frequency and time to coalescence, respectively.
3. **SFS Calculation:** The `getSFS()` method accumulates the branch lengths for each allele frequency across all generated trees, resulting in the SFS.
4. **SFS Binning:** The `binSFS()` method allows for binning the SFS using different modes (linear, log, logit) and bin numbers or custom bin edges.
#### Side Effects:
* Modifies the `self.alleles` and `self.sfs` attributes during the SFS calculation.
### Symbol Name: `glob_trees`
#### Description:
Generates multiple beta coalescent trees and stores the allele frequency information for each tree.
#### Inputs:
| Name | Type | Description |
| :----- | :--- | :--------------------------------------------- |
| ntrees | int | The number of trees to generate (default: 10). |
#### Outputs:
The method does not return any value. It updates the `self.alleles` attribute with the allele frequency information from the generated trees.
#### Internal Logic:
1. **Iterative Tree Generation:** The method iterates `ntrees` times, generating a new tree in each iteration using the `coalesce()` method.
2. **Allele Information Extraction:** For each tree, it extracts the weight (number of descendants) and branch length for all nodes (terminal and non-terminal) and appends this information to the `self.alleles` list.
### Symbol Name: `getSFS`
#### Description:
Calculates the Site Frequency Spectrum (SFS) based on the generated trees.
#### Inputs:
| Name | Type | Description |
| :----- | :--- | :------------------------------------------------------------ |
| ntrees | int | The number of trees to use for SFS calculation (default: 10). |
#### Outputs:
The method does not return any value. It updates the `self.sfs` attribute with the calculated SFS.
#### Internal Logic:
1. **Initialization:** Creates a NumPy array `self.sfs` filled with zeros and a length equal to the sample size plus one.
2. **SFS Accumulation:** Iterates through the specified number of trees (`ntrees`) and for each tree:
* Generates a new tree using the `coalesce()` method.
* Traverses the tree nodes and increments the corresponding element in the `self.sfs` array based on the node's weight (allele frequency) and branch length (time to coalescence).
3. **Normalization:** Divides the `self.sfs` array by the number of trees to obtain the average SFS.
### Symbol Name: `binSFS`
#### Description:
Bins the pre-calculated SFS using different modes and bin numbers or custom bin edges.
#### Inputs:
| Name | Type | Description |
| :--- | :---------------- | :------------------------------------------------------------------- |
| mode | str | The binning mode. Options: "logit" (default), "linear", "log". |
| bins | int or array-like | The number of bins or an array-like object specifying the bin edges. |
#### Outputs:
The method does not return any value. It updates the `self.binned_sfs`, `self.bin_center`, and `self.bin_width` attributes based on the chosen binning parameters.
#### Internal Logic:
1. **Bin Edge Determination:** Determines the bin edges based on the chosen `mode` and `bins` parameters.
* If `bins` is an iterable, it is used directly as bin edges.
* Otherwise, bin edges are calculated based on the chosen `mode`:
* "logit": Logit-spaced bins.
* "linear": Linearly spaced bins.
* "log": Logarithmically spaced bins.
2. **SFS Binning:** Uses the `np.histogram()` function to bin the pre-calculated SFS (`self.sfs`) based on the determined bin edges.
3. **Normalization:** Divides the binned SFS by the bin widths to obtain the density.
### Symbol Name: `saveSFS`
#### Description:
Saves the calculated SFS to a file.
#### Inputs:
| Name | Type | Description |
| :---- | :--- | :------------------------------------------------------------------------------------------ |
| fname | str | The filename to save the SFS to. If the filename ends with ".gz", the file will be gzipped. |
#### Outputs:
The method does not return any value. It saves the SFS to the specified file.
#### Internal Logic:
1. **SFS Calculation (if necessary):** If the SFS has not been calculated yet (`self.sfs` is None), it calls the `getSFS()` method to calculate it.
2. **File Saving:** Uses the `np.savetxt()` function to save the SFS to the specified file.
### Symbol Name: `loadSFS`
#### Description:
Loads a previously saved SFS from a file.
#### Inputs:
| Name | Type | Description |
| :---- | :---- | :----------------------------------------------------------- |
| fname | str | The filename to load the SFS from. |
| alpha | float | The alpha parameter of the beta coalescent model (optional). |
#### Outputs:
The method does not return any value. It updates the `self.n`, `self.sfs`, and `self.alpha` attributes based on the loaded SFS.
#### Internal Logic:
1. **File Loading:** Attempts to load the SFS from the specified file using the `np.loadtxt()` function.
2. **Data Validation:** Checks if the loaded data is a one-dimensional vector. If not, it prints an error message and sets `self.sfs` to None.
3. **Attribute Update:** If the loaded data is valid, it updates the `self.n` (sample size), `self.sfs`, and `self.alpha` attributes accordingly.
## Dependencies
| Dependency | Purpose |
| :------------ | :----------------------------------------------------------------------------- |
| numpy | Numerical computations, array operations, and file I/O. |
| scipy.special | Special functions, including the gamma function used in the beta distribution. |
| Bio.Phylo | Phylogenetic tree manipulation and analysis. |
## Error Handling
The code includes basic error handling for file loading in the `loadSFS` method. If the specified file does not exist or the loaded data is not a one-dimensional vector, it prints an error message and sets the `self.sfs` attribute to None.
## Logging
The code does not implement any specific logging mechanisms.
## API/Interface Reference
The code does not explicitly define an API or public interface.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/resources/overview
## High-level description
This directory contains implementations of various algorithms and tools for analyzing evolutionary fitness and phylogenetic relationships in biological sequences. The main components are:
1. A fitness inference module based on the shape of genealogical trees.
2. A beta coalescent tree simulator and Site Frequency Spectrum (SFS) calculator.
These tools are designed to predict evolution, understand the shape of genealogical trees, and analyze genetic diversity in populations.
## What does it do?
The code in this directory performs several key functions:
1. Infers fitness from genealogical tree shapes:
* Ranks sequences in multiple sequence alignments based on inferred fitness.
* Calculates fitness distributions for nodes in phylogenetic trees.
* Reconstructs ancestral sequences.
* Analyzes and visualizes phylogenetic trees.
2. Simulates and analyzes beta coalescent trees:
* Generates genealogical trees representing ancestral relationships in a population sample.
* Calculates the Site Frequency Spectrum (SFS) to summarize genetic variation.
3. Provides tools for evolutionary analysis:
* Implements the Local Branching Index (LBI) for sequence ranking.
* Performs full fitness inference on sequences.
* Simulates adapting populations (in a toy data subdirectory).
## Entry points
The main entry points for using this codebase are:
1. `rank_sequences.py`: Ranks sequences in a multiple sequence alignment using the Local Branching Index (LBI).
2. `infer_fitness.py`: Performs full fitness inference on sequences in a multiple sequence alignment.
Both scripts take a multiple sequence alignment and an outgroup sequence as input, producing various output files including ranked sequences, reconstructed trees, and inferred ancestral sequences.
## Key Files
1. FitnessInference directory:
* `prediction_src/`: Core implementation of fitness inference and sequence ranking algorithms.
* `rank_sequences.py`: Script for ranking sequences using the Local Branching Index.
* `infer_fitness.py`: Script for performing full fitness inference on sequences.
2. betatree directory:
* `betatree.py`: Implements the `betatree` class for simulating beta coalescent trees.
* `sfs.py` and `sfs_py3.py`: Define the `SFS` class for calculating the Site Frequency Spectrum.
## Dependencies
The codebase relies on several external libraries:
1. Biopython: For handling biological sequences, alignments, and phylogenetic trees.
2. NumPy: For numerical computations and array manipulations.
3. SciPy: For various scientific computing tasks and special mathematical functions.
4. Matplotlib: For visualization of phylogenetic trees, results, and SFS plots.
Additionally, the code may use external tools like `fasttree` for phylogenetic tree construction.
## Configuration
The main scripts use command-line arguments for configuration. Key parameters include:
* `--aln`: Path to the input alignment file.
* `--outgroup`: Name of the outgroup sequence.
* `--eps_branch`: Minimal branch length for inference.
* `--tau`: Time scale for local tree length estimation (for LBI).
* `--diffusion`: Fitness diffusion coefficient (for full inference).
* `--gamma`: Scale factor for time scale.
* `--omega`: Approximate sampling fraction divided by fitness standard deviation.
* `--collapse`: Option to collapse internal branches with identical sequences.
* `--plot`: Option to plot trees.
For the beta coalescent simulator and SFS calculator, key parameters are passed as arguments to class constructors:
* `sample_size`: The number of individuals in the sample.
* `alpha`: The alpha parameter of the beta coalescent model.
The `SFS` class allows for configuration of the SFS calculation and binning process through method parameters such as `ntrees`, `mode`, and `bins`.
Users can adjust these parameters to customize the analysis for their specific needs in evolutionary studies and population genetics research.
# sfs.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/sfs.py
## High-level description
The code defines a class `SFS` that represents a Site Frequency Spectrum (SFS), a fundamental concept in population genetics used to summarize genetic variation within a population. It provides methods to calculate various statistics from the SFS, such as Tajima's D, Fay and Wu's H, Zeng's E, and Ferretti's L, which are used to infer demographic history and selection pressures.
## Code Structure
The `SFS` class is the main component of the code. It holds the SFS data (`counts`) and provides methods to calculate different statistics. The statistics calculation methods (`fay_and_wus_H`, `zengs_E`, `tajimas_D`, `ferrettis_L`) rely on the SFS data and use helper methods (`_calculate_fay_and_wus_H`, `_calculate_zengs_E`, `_calculate_tajimas_D`, `_calculate_ferrettis_L`) to perform the actual calculations. The `bin` method is used to group the SFS data into bins for easier visualization and analysis.
## Symbols
### Symbol Name
```python theme={null}
SFS
```
#### Description
This class represents a Site Frequency Spectrum (SFS) and provides methods for calculating various statistics from it.
#### Inputs
| Name | Type | Description |
| :----- | :----------------------------- | :---------------------------------------------------------------------- |
| counts | numpy.ndarray | An array representing the counts of mutations at different frequencies. |
| T | cassiopeia.data.CassiopeiaTree | Optional. A CassiopeiaTree object representing the phylogenetic tree. |
#### Outputs
An `SFS` object.
#### Internal Logic
The constructor initializes the `counts` and `T` attributes. It also initializes several attributes to `None`, which will be calculated lazily when their corresponding methods are called.
***
### Symbol Name
```python theme={null}
SFS.from_tree
```
#### Description
This class method constructs an SFS object from a given phylogenetic tree.
#### Inputs
| Name | Type | Description |
| :--- | :----------------------------- | :---------------------------------------------------------- |
| T | cassiopeia.data.CassiopeiaTree | A CassiopeiaTree object representing the phylogenetic tree. |
#### Outputs
An `SFS` object.
#### Internal Logic
The method iterates through the clades of the tree and counts the number of mutations at each frequency (number of leaves in each clade). It then creates an `SFS` object with the calculated counts and the input tree.
***
### Symbol Name
```python theme={null}
SFS.bin
```
#### Description
This method bins the SFS data into specified bins for easier visualization and analysis.
#### Inputs
| Name | Type | Description |
| :-------- | :---------------- | :--------------------------------------------------------------------------------------------------------------- |
| bins | str or array-like | Specifies the binning strategy. Can be "linear", "log", "logit", or an array-like object defining the bin edges. |
| n\_bins | int | Optional. Number of bins to use if `bins` is "linear" or "log". |
| min\_edge | float | Optional. Minimum edge for binning if `bins` is "linear" or "log". |
| max\_edge | float | Optional. Maximum edge for binning if `bins` is "linear" or "log". |
#### Outputs
None. The method modifies the `SFS` object in place, adding attributes related to the binning.
#### Internal Logic
The method first determines the bin edges and centers based on the `bins` argument and optional parameters. It then bins the SFS data by counting the number of mutations falling into each bin. The binned data is normalized by the bin size. If a tree is provided, the method also calculates a cut version of the normalized binned data, setting bins beyond the tree size to `np.nan`.
***
### Symbol Name
```python theme={null}
SFS.fay_and_wus_H
```
#### Description
This method calculates and returns Fay and Wu's H statistic from the SFS.
#### Inputs
None.
#### Outputs
| Name | Type | Description |
| :--------------- | :---- | :------------------------ |
| fay\_and\_wus\_H | float | Fay and Wu's H statistic. |
#### Internal Logic
The method first checks if the statistic has been previously calculated and cached. If not, it calls the `_calculate_fay_and_wus_H` method to calculate it.
***
### Symbol Name
```python theme={null}
SFS._calculate_fay_and_wus_H
```
#### Description
This private method calculates Fay and Wu's H statistic from the SFS data.
#### Inputs
None.
#### Outputs
None. The method modifies the `SFS` object in place, setting the `_fay_and_wus_H` attribute.
#### Internal Logic
The method calculates the statistic based on the formula: `theta_pi - theta_H`, where `theta_pi` and `theta_H` are calculated from the SFS data.
***
### Symbol Name
```python theme={null}
SFS.zengs_E
```
#### Description
This method calculates and returns Zeng's E statistic from the SFS.
#### Inputs
None.
#### Outputs
| Name | Type | Description |
| :------- | :---- | :------------------ |
| zengs\_E | float | Zeng's E statistic. |
#### Internal Logic
The method first checks if the statistic has been previously calculated and cached. If not, it calls the `_calculate_zengs_E` method to calculate it.
***
### Symbol Name
```python theme={null}
SFS._calculate_zengs_E
```
#### Description
This private method calculates Zeng's E statistic from the SFS data.
#### Inputs
None.
#### Outputs
None. The method modifies the `SFS` object in place, setting the `_zengs_E` attribute.
#### Internal Logic
The method calculates the statistic based on the formula: `theta_L - theta_W`, where `theta_L` and `theta_W` are calculated from the SFS data.
***
### Symbol Name
```python theme={null}
SFS.tajimas_D
```
#### Description
This method calculates and returns Tajima's D statistic from the SFS.
#### Inputs
None.
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :-------------------- |
| tajimas\_D | float | Tajima's D statistic. |
#### Internal Logic
The method first checks if the statistic has been previously calculated and cached. If not, it calls the `_calculate_tajimas_D` method to calculate it.
***
### Symbol Name
```python theme={null}
SFS._calculate_tajimas_D
```
#### Description
This private method calculates Tajima's D statistic from the SFS data.
#### Inputs
None.
#### Outputs
None. The method modifies the `SFS` object in place, setting the `_tajimas_D` attribute.
#### Internal Logic
The method calculates the statistic based on the formula: `theta_pi - theta_W`, where `theta_pi` and `theta_W` are calculated from the SFS data.
***
### Symbol Name
```python theme={null}
SFS.ferrettis_L
```
#### Description
This method calculates and returns Ferretti's L statistic from the SFS.
#### Inputs
None.
#### Outputs
| Name | Type | Description |
| :----------- | :---- | :---------------------- |
| ferrettis\_L | float | Ferretti's L statistic. |
#### Internal Logic
The method first checks if the statistic has been previously calculated and cached. If not, it calls the `_calculate_ferrettis_L` method to calculate it.
***
### Symbol Name
```python theme={null}
SFS._calculate_ferrettis_L
```
#### Description
This private method calculates Ferretti's L statistic from the SFS data.
#### Inputs
None.
#### Outputs
None. The method modifies the `SFS` object in place, setting the `_ferrettis_L` attribute.
#### Internal Logic
The method calculates the statistic based on the formula: `theta_W - theta_H`, where `theta_W` and `theta_H` are calculated from the SFS data.
***
# size_matched_model.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/size_matched_model.py
## High-level description
The code defines a `SizeMatchedModel` class that represents a statistical model where parameters vary based on the size of the input data. It provides methods for loading and saving the model, calculating p-values, and determining model means based on input size.
## Code Structure
The `SizeMatchedModel` class is the central component. It uses a list of bins to partition the input size range and associates a set of parameters with each bin. The `distribution` attribute holds a statistical distribution object (not defined in this file) used for calculations.
## Symbols
### `SizeMatchedModel`
#### Description
This class represents a statistical model where parameters are determined by the size of the input data. It uses bins to divide the size range and associates a set of parameters with each bin.
#### Inputs
| Name | Type | Description |
| :----------- | :----- | :------------------------------------------------------------------------ |
| bins | list | List of bin edges defining the size ranges. |
| params | list | List of parameter sets, one for each bin. |
| distribution | object | An instance of a statistical distribution class (e.g., from scipy.stats). |
| name | str | Optional name for the model. |
#### Outputs
N/A - This is a class definition, not a function.
#### Internal Logic
The class stores the bins, parameters, distribution, and name. It provides methods for:
* Loading and saving the model from/to JSON files.
* Finding the appropriate parameters for a given size using `_params_for_size`.
* Calculating p-values using the provided distribution and parameters.
* Determining the model mean for a given size.
### `SizeMatchedModel.from_json`
#### Description
This class method loads a `SizeMatchedModel` instance from a JSON file.
#### Inputs
| Name | Type | Description |
| :------- | :--- | :----------------------------------------------- |
| filename | str | Path to the JSON file containing the model data. |
#### Outputs
| Name | Type | Description |
| :---- | :--------------- | :------------------------------------------------------ |
| model | SizeMatchedModel | A new `SizeMatchedModel` instance loaded from the file. |
#### Internal Logic
* Opens the JSON file and loads the data.
* Converts string representations of lists and the distribution name to their respective Python objects.
* Instantiates a new `SizeMatchedModel` with the loaded data.
### `SizeMatchedModel.to_json`
#### Description
This method saves the `SizeMatchedModel` instance to a JSON file.
#### Inputs
| Name | Type | Description |
| :------ | :--- | :---------------------------- |
| outfile | str | Path to the output JSON file. |
#### Outputs
N/A - The method writes data to a file as a side effect.
#### Internal Logic
* Creates a dictionary containing the model's attributes.
* Converts lists to JSON-serializable strings.
* Extracts the distribution's class name for serialization.
* Writes the dictionary to the specified JSON file.
### `SizeMatchedModel._params_for_size`
#### Description
This private method retrieves the appropriate parameters for a given size based on the defined bins.
#### Inputs
| Name | Type | Description |
| :------------- | :---- | :------------------------------------------------------------------------------------------------------------------ |
| size | float | The size value for which to find the parameters. |
| strict\_bounds | bool | If True, raises an error if the size is outside the defined bins. If False, uses the parameters of the nearest bin. |
#### Outputs
| Name | Type | Description |
| :------------ | :---- | :---------------------------------------------------------------- |
| params\_match | tuple | The parameters associated with the bin containing the given size. |
#### Internal Logic
* Uses `np.digitize` to find the index of the bin corresponding to the input size.
* Handles cases where the size is outside the defined bins based on `strict_bounds`.
* Adjusts the bin index to match the zero-based indexing of the `params` list.
* Returns the parameters for the selected bin.
### `SizeMatchedModel.pvalue`
#### Description
This method calculates the p-value of a given value `x` under the model for a specific size.
#### Inputs
| Name | Type | Description |
| :------------- | :---- | :----------------------------------------------------------------------------------- |
| x | float | The value for which to calculate the p-value. |
| size | float | The size value used to determine the model parameters. |
| invert\_cdf | bool | If True, calculates 1 - CDF(x) instead of CDF(x). |
| strict\_bounds | bool | Passed to `_params_for_size` to control behavior for sizes outside the defined bins. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :---------------------- |
| p | float | The calculated p-value. |
#### Internal Logic
* Retrieves the model parameters for the given size using `_params_for_size`.
* Calculates the cumulative distribution function (CDF) of `x` using the model's distribution and parameters.
* Inverts the CDF if `invert_cdf` is True.
* Returns the calculated p-value.
### `SizeMatchedModel.model_mean`
#### Description
This method calculates the mean of the model for a given size.
#### Inputs
| Name | Type | Description |
| :------------- | :---- | :----------------------------------------------------------------------------------- |
| size | float | The size value used to determine the model parameters. |
| strict\_bounds | bool | Passed to `_params_for_size` to control behavior for sizes outside the defined bins. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :-------------------------------- |
| mean | float | The calculated mean of the model. |
#### Internal Logic
* Retrieves the model parameters for the given size using `_params_for_size`.
* Calculates the mean of the distribution using the retrieved parameters.
* Returns the calculated mean.
## TODOs
* Evaluate distribution in global namespace, so that import is not necessary here
# tree.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/jungle/tree.py
## High-level description
The `Tree` class represents a phylogenetic tree and provides methods for manipulating, analyzing, and visualizing it. It supports constructing trees from Newick files, generating trees by simulation, and performing various operations like annotating nodes with features, calculating tree statistics, and inferring fitness.
## Code Structure
The `Tree` class encapsulates an `ete3.Tree` object and extends its functionality with custom methods. It utilizes external libraries like `ete3` for tree manipulation, `Bio.Phylo` for interfacing with Biopython, and other modules for specific tasks like site frequency spectrum analysis and fitness inference.
## References
This code references the following external libraries and modules:
* `ete3`: Used for phylogenetic tree manipulation and visualization.
* `Bio.Phylo`: Used for interfacing with Biopython's phylogenetic tree representation.
* `.resources.betatree.src.betatree`: Used for simulating trees using the beta-tree model.
* `.resources.FitnessInference.prediction_src.node_ranking`: Used for inferring fitness metrics of nodes.
* `jungle.sfs.SFS`: Used for calculating and analyzing site frequency spectra.
## Symbols
### `Tree`
#### Description
This class represents a phylogenetic tree and provides methods for its manipulation, analysis, and visualization.
#### Inputs
| Name | Type | Description |
| :----- | :-------- | :--------------------------------------------------------------- |
| T | ete3.Tree | An ete3.Tree object representing the phylogenetic tree. |
| name | str | The name of the tree. |
| params | dict | A dictionary of parameters used to generate or analyze the tree. |
#### Outputs
This class does not have a return value. It is used to create and manipulate Tree objects.
#### Internal Logic
The `Tree` class stores the tree structure in an `ete3.Tree` object and provides methods for:
* Loading and saving trees from/to files.
* Generating trees by simulation.
* Annotating nodes with various features like depth, number of descendants, imbalance, etc.
* Calculating tree statistics like total branch length, site frequency spectrum, and various diversity indices.
* Inferring fitness of nodes using external packages.
* Visualizing the tree with node coloring and rendering.
## Side Effects
Some methods of the `Tree` class modify the internal state of the `ete3.Tree` object, such as adding new attributes to nodes or changing branch lengths.
#### Performance Considerations
The performance of some methods, especially those involving tree traversal or complex calculations, may vary depending on the size and complexity of the tree.
### `from_newick`
#### Description
Constructs a `Tree` object from a Newick file.
#### Inputs
| Name | Type | Description |
| :------- | :--- | :------------------------------------- |
| filename | str | The path to the Newick file. |
| name | str | The name of the tree (optional). |
| params | dict | A dictionary of parameters (optional). |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :---------------------------------------------------------- |
| tree | Tree | A `Tree` object representing the tree from the Newick file. |
#### Internal Logic
* Reads the Newick file using `ete3.Tree`.
* Ladderizes the tree for consistent visualization.
* Assigns names to internal nodes if not already present.
### `from_pickle`
#### Description
Loads a `Tree` object from a pickle file.
#### Inputs
| Name | Type | Description |
| :------- | :--- | :------------------------------------------------------------------------------ |
| filename | str | The path to the pickle file. |
| gzip | bool | Whether the file is gzipped (optional, inferred from filename if not provided). |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :------------------------------------------- |
| tree | Tree | A `Tree` object loaded from the pickle file. |
#### Internal Logic
* Opens the pickle file, handling gzip compression if necessary.
* Loads the `Tree` object using `pickle.load`.
### `generate`
#### Description
Generates a random `Tree` object using the beta-tree model.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| params | dict | A dictionary of parameters for tree generation, including 'n\_leaves' (number of leaves) and 'alpha' (shape parameter of the beta distribution). |
| name | str | The name of the tree (optional). |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :---------------------------------- |
| tree | Tree | A randomly generated `Tree` object. |
#### Internal Logic
* Uses the `betatree` module to simulate a tree with the given parameters.
* Converts the simulated tree to an `ete3.Tree` object.
* Ladderizes the tree and assigns names to internal nodes.
### `to_newick`
#### Description
Writes the `Tree` object to a Newick file.
#### Inputs
| Name | Type | Description |
| :--------- | :---------------- | :------------------------------------------------ |
| outfile | str | The path to the output Newick file. |
| \*\*kwargs | keyword arguments | Additional arguments passed to `ete3.Tree.write`. |
#### Outputs
This method does not return any value. It writes the tree to a file.
#### Internal Logic
* Uses the `ete3.Tree.write` method to write the tree to the specified file in Newick format.
### `annotate_standard_node_features`
#### Description
Annotates each node in the tree with standard features like depth, number of children, and number of descendants.
#### Inputs
This method does not take any inputs.
#### Outputs
This method does not return any value. It modifies the tree in place.
#### Internal Logic
* Traverses the tree in postorder.
* For each node, calculates and sets the following attributes:
* `depth`: Distance from the root.
* `num_children`: Number of direct children.
* `num_descendants`: Total number of descendants, including the node itself.
* `num_leaf_descendants`: Total number of leaf nodes that descend from this node.
* Sets the `is_depth_annotated` flag to True.
* Calculates and sets the `depth_normalized` attribute for each node, normalized by the maximum depth.
### `node_features`
#### Description
Returns a Pandas DataFrame containing selected features of all nodes in the tree.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :-------------------------------------------------------------------------------------------------- |
| subset | list | A list of feature names to include in the DataFrame (optional, defaults to all available features). |
#### Outputs
| Name | Type | Description |
| :------- | :--------------- | :----------------------------------------------------------------------------------- |
| features | pandas.DataFrame | A DataFrame with rows representing nodes and columns representing selected features. |
#### Internal Logic
* Determines the set of features to include in the DataFrame.
* Iterates through each feature and each node, retrieving the feature value (or None if not available) and storing it in the DataFrame.
* Adds an 'is\_leaf' column indicating whether each node is a leaf.
### `annotate_imbalance`
#### Description
Calculates and annotates each node with its imbalance, defined as the ratio of the maximum number of descendants among its children to the total number of descendants.
#### Inputs
This method does not take any inputs.
#### Outputs
This method does not return any value. It modifies the tree in place.
#### Internal Logic
* Traverses the tree.
* For each non-leaf node:
* Retrieves the number of descendants for each child node.
* Calculates the imbalance as the maximum number of descendants divided by the total number of descendants.
* Sets the 'imbalance' attribute of the node.
### `annotate_colless`
#### Description
Calculates and annotates each node with its Colless index, a measure of tree balance.
#### Inputs
This method does not take any inputs.
#### Outputs
This method does not return any value. It modifies the tree in place.
#### Internal Logic
* Traverses the tree in postorder.
* For each node:
* If it's a leaf node, sets the 'colless' attribute to 0.
* If it's an internal node:
* Calculates the absolute difference in the number of descendants between its two children.
* Recursively calculates the Colless index for each child.
* Calculates the Colless index for the current node by summing the difference in descendants and the Colless indices of its children.
* Sets the 'difference\_num\_descendants', 'colless', and 'log10p\_colless' attributes of the node.
* Sets the 'colless' and 'log10p\_colless' attributes of the `Tree` object itself.
### `max_depth`
#### Description
Returns the maximum depth of the tree.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :--------- | :--- | :----------------------------- |
| max\_depth | int | The maximum depth of the tree. |
#### Internal Logic
* Checks if the `is_depth_annotated` flag is True. If not, raises an error.
* Retrieves the 'depth' attribute of all nodes and returns the maximum value.
### `total_branch_length`
#### Description
Calculates and returns the total branch length of the tree.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :-------------------- | :---- | :----------------------------------- |
| total\_branch\_length | float | The total branch length of the tree. |
#### Internal Logic
* Sums the 'dist' attribute (branch length) of all nodes in the tree.
### `rescale`
#### Description
Rescales the tree so that its total branch length equals a given value.
#### Inputs
| Name | Type | Description |
| :-------------------- | :---- | :----------------------------------------------- |
| total\_branch\_length | float | The desired total branch length after rescaling. |
#### Outputs
This method does not return any value. It modifies the tree in place.
#### Internal Logic
* Calculates the current total branch length.
* Calculates the scaling factor as the desired length divided by the current length.
* Multiplies the 'dist' attribute (branch length) of each node by the scaling factor.
### `resolve_polytomy`
#### Description
Resolves any polytomies (nodes with more than two children) in the tree by creating an arbitrary dichotomous structure.
#### Inputs
This method does not take any inputs.
#### Outputs
This method does not return any value. It modifies the tree in place.
#### Internal Logic
* Uses the `ete3.Tree.resolve_polytomy` method to resolve polytomies.
* Ladderizes the tree to maintain a consistent visualization.
### `site_frequency_spectrum`
#### Description
Calculates and returns the site frequency spectrum (SFS) of the tree.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :--- | :--- | :-------------------------------------------------------------------- |
| sfs | SFS | An `SFS` object representing the site frequency spectrum of the tree. |
#### Internal Logic
* If the `_site_frequency_spectrum` attribute is not already calculated:
* Calculates the SFS using the `SFS.from_tree` method.
* Stores the result in the `_site_frequency_spectrum` attribute.
* Returns the `_site_frequency_spectrum` attribute.
### `bin_site_frequency_spectrum`
#### Description
Bins the site frequency spectrum (SFS) of the tree using specified bins.
#### Inputs
| Name | Type | Description |
| :--------- | :--------- | :-------------------------------------------------------------- |
| bins | array-like | An array-like object defining the bin edges. |
| \*args | tuple | Additional positional arguments passed to the `SFS.bin` method. |
| \*\*kwargs | dict | Additional keyword arguments passed to the `SFS.bin` method. |
#### Outputs
| Name | Type | Description |
| :---------- | :--------- | :-------------------------------------------------------------------- |
| binned\_sfs | array-like | An array-like object containing the binned and normalized SFS values. |
#### Internal Logic
* Calls the `SFS.bin` method on the tree's SFS with the provided arguments.
* Returns the `binned_normalized_cut` attribute of the SFS object, which represents the binned and normalized SFS values.
### `fay_and_wus_H`
#### Description
Calculates and returns Fay and Wu's H statistic, a measure of genetic diversity, from the tree's SFS.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :--- | :---- | :------------------------ |
| H | float | Fay and Wu's H statistic. |
#### Internal Logic
* If the `_fay_and_wus_H` attribute is not already calculated:
* Calculates H using the `SFS.fay_and_wus_H` method.
* Stores the result in the `_fay_and_wus_H` attribute.
* Returns the `_fay_and_wus_H` attribute.
### `zengs_E`
#### Description
Calculates and returns Zeng's E statistic, a measure of genetic diversity, from the tree's SFS.
#### Inputs
This method does not take any inputs.
#### Outputs
| Name | Type | Description |
| :--- | :--- | :---------- |
| | | |
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/overview
## High-level description
This directory contains implementations of various algorithms and tools for analyzing evolutionary fitness and phylogenetic relationships in biological sequences. The main components are a `Forest` class representing a collection of phylogenetic trees, a `Tree` class for individual phylogenetic trees, a `SFS` (Site Frequency Spectrum) class for analyzing genetic diversity, and a `SizeMatchedModel` class for statistical modeling based on data size. Additionally, there is a script for generating and annotating reference data in the form of phylogenetic trees.
## What does it do?
The code in this directory performs several key functions:
1. Manages collections of phylogenetic trees:
* Loads and saves trees from/to various formats (Newick, pickle, tar.gz).
* Generates trees through simulation.
* Concatenates forests of trees.
2. Analyzes individual phylogenetic trees:
* Annotates nodes with features like depth, number of descendants, imbalance, and Colless index.
* Calculates tree statistics such as total branch length and site frequency spectrum.
* Resolves polytomies and rescales trees.
3. Calculates and analyzes Site Frequency Spectra (SFS):
* Computes SFS from tree structures.
* Calculates various genetic diversity statistics (Tajima's D, Fay and Wu's H, Zeng's E, Ferretti's L).
* Bins SFS data for visualization and analysis.
4. Provides statistical modeling based on data size:
* Implements a model where parameters vary based on input data size.
* Calculates p-values and model means for given sizes.
5. Generates reference data:
* Creates a collection of phylogenetic trees with specified parameters.
* Annotates the trees with various features.
* Saves the annotated forest to a compressed pickle file for later use.
## Entry points
The main entry points for using this codebase are:
1. `Forest` class: For managing collections of phylogenetic trees.
2. `Tree` class: For analyzing individual phylogenetic trees.
3. `SFS` class: For calculating and analyzing Site Frequency Spectra.
4. `SizeMatchedModel` class: For statistical modeling based on data size.
5. `generate_annotate_forest.py`: Script for generating reference data.
## Key Files
1. `forest.py`: Defines the `Forest` class for managing collections of trees.
2. `tree.py`: Defines the `Tree` class for individual tree analysis.
3. `sfs.py`: Defines the `SFS` class for Site Frequency Spectrum analysis.
4. `size_matched_model.py`: Defines the `SizeMatchedModel` class for size-based statistical modeling.
5. `generate_annotate_forest.py`: Script for generating and annotating reference data.
## Dependencies
The codebase relies on several external libraries:
1. `ete3`: For phylogenetic tree manipulation and visualization.
2. `Bio.Phylo`: For interfacing with Biopython's phylogenetic tree representation.
3. `numpy`: For numerical computations and array manipulations.
4. `scipy`: For various scientific computing tasks and statistical functions.
5. `pandas`: For data manipulation and analysis.
6. `matplotlib`: For visualization of results and trees.
7. `sys`: For accessing command line arguments.
8. `time`: For tracking script execution time.
9. `uuid`: For generating unique identifiers.
10. `pickle`: For serializing and saving objects.
11. `gzip`: For compressing output files.
Additionally, the code uses custom modules for specific tasks like beta-tree simulation and fitness inference.
## Configuration
The main classes (`Forest`, `Tree`, `SFS`, `SizeMatchedModel`) use constructor parameters and method arguments for configuration. Key parameters include:
* For `Forest` and `Tree`:
* Paths to input files (Newick, pickle, tar.gz).
* Parameters for tree generation (e.g., number of leaves, alpha for beta-tree model).
* Options for tree manipulation (e.g., resolving polytomies, rescaling).
* For `SFS`:
* SFS data as numpy array.
* Binning parameters for SFS analysis.
* For `SizeMatchedModel`:
* Bins and parameters for size-based modeling.
* Statistical distribution for calculations.
The `generate_annotate_forest.py` script takes command-line arguments for configuration:
* `n_leaves`: Number of leaves (tips) in each generated tree.
* `n_trees`: Number of trees to generate for the forest.
* `alpha`: Shape parameter influencing the tree structure.
* `output_dir`: Path to the directory where the output file will be saved.
Users can adjust these parameters to customize the analysis for their specific needs in evolutionary studies and population genetics research.
# generate_annotate_forest.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/reference_data/generate_annotate_forest.py
## High-level description
This script generates and annotates a collection of phylogenetic trees (a "forest") using the `jungle` library. It allows specifying the number of trees, leaves per tree, and a shape parameter controlling the tree structure. The resulting annotated forest is then saved to a compressed pickle file.
## References
This script references the `jungle` library (`jg`) for phylogenetic tree generation and manipulation.
## Symbols
### `generate_annotate_forest.py`
#### Description
This script generates a collection of phylogenetic trees, annotates them with various features, and saves the resulting forest data structure to a file.
#### Inputs
This script takes its input from command line arguments:
| Name | Type | Description |
| :----------- | :---- | :-------------------------------------------------------------------------------------------- |
| `n_leaves` | int | Number of leaves (tips) in each generated tree. |
| `n_trees` | int | Number of trees to generate for the forest. |
| `alpha` | float | Shape parameter influencing the tree structure (2.0 for neutral, 1.0 for positive selection). |
| `output_dir` | str | Path to the directory where the output file will be saved. |
#### Outputs
The script generates a gzipped pickle file containing the annotated forest data structure. The file name is based on the input parameters and a unique identifier.
#### Internal Logic
1. **Parameter Parsing:** Reads input parameters from command line arguments.
2. **Output File Naming:** Constructs the output file name based on input parameters and a UUID.
3. **Forest Generation:** Uses the `jungle` library to generate a forest of `n_trees` trees, each with `n_leaves` leaves, using the specified `alpha` shape parameter.
4. **Forest Annotation:**
* Resolves potential multifurcations (polytomies) in the generated trees.
* Annotates the trees with standard node features (e.g., distance to root).
* Calculates and annotates the Colless index for each tree, a measure of tree balance.
5. **Forest Serialization:** Saves the annotated forest object to a gzipped pickle file.
#### Side Effects
* Creates a file in the specified output directory.
* Prints status messages and timing information to the console if `verbose` is True.
## Dependencies
| Dependency | Purpose |
| :--------- | :----------------------------------------------------------------------------------- |
| `jungle` | Provides functionality for phylogenetic tree generation, manipulation, and analysis. |
| `sys` | Used for accessing command line arguments. |
| `time` | Used for tracking the script's execution time. |
| `uuid` | Used for generating a unique identifier for the output file. |
| `pickle` | Used for serializing and saving the forest object. |
| `gzip` | Used for compressing the output pickle file. |
## Configuration
The script's behavior is controlled by command line arguments, as described in the "Inputs" section.
## Logging
The script prints informative messages to the console if the `verbose` variable is set to `True`. This includes parameter values, progress updates, and timing information.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_jungle/reference_data/overview
## High-level description
This script generates and annotates a collection of phylogenetic trees (a "forest") using the `jungle` library. It allows specifying the number of trees, leaves per tree, and a shape parameter controlling the tree structure. The resulting annotated forest is then saved to a compressed pickle file.
## References
This script references the `jungle` library (`jg`) for phylogenetic tree generation and manipulation.
## Symbols
### `generate_annotate_forest.py`
#### Description
This script generates a collection of phylogenetic trees, annotates them with various features, and saves the resulting forest data structure to a file.
#### Inputs
This script takes its input from command line arguments:
| Name | Type | Description |
| :----------- | :---- | :-------------------------------------------------------------------------------------------- |
| `n_leaves` | int | Number of leaves (tips) in each generated tree. |
| `n_trees` | int | Number of trees to generate for the forest. |
| `alpha` | float | Shape parameter influencing the tree structure (2.0 for neutral, 1.0 for positive selection). |
| `output_dir` | str | Path to the directory where the output file will be saved. |
#### Outputs
The script generates a gzipped pickle file containing the annotated forest data structure. The file name is based on the input parameters and a unique identifier.
#### Internal Logic
1. **Parameter Parsing:** Reads input parameters from command line arguments.
2. **Output File Naming:** Constructs the output file name based on input parameters and a UUID.
3. **Forest Generation:** Uses the `jungle` library to generate a forest of `n_trees` trees, each with `n_leaves` leaves, using the specified `alpha` shape parameter.
4. **Forest Annotation:**
* Resolves potential multifurcations (polytomies) in the generated trees.
* Annotates the trees with standard node features (e.g., distance to root).
* Calculates and annotates the Colless index for each tree, a measure of tree balance.
5. **Forest Serialization:** Saves the annotated forest object to a gzipped pickle file.
#### Side Effects
* Creates a file in the specified output directory.
* Prints status messages and timing information to the console if `verbose` is True.
## Dependencies
| Dependency | Purpose |
| :--------- | :----------------------------------------------------------------------------------- |
| `jungle` | Provides functionality for phylogenetic tree generation, manipulation, and analysis. |
| `sys` | Used for accessing command line arguments. |
| `time` | Used for tracking the script's execution time. |
| `uuid` | Used for generating a unique identifier for the output file. |
| `pickle` | Used for serializing and saving the forest object. |
| `gzip` | Used for compressing the output pickle file. |
## Configuration
The script's behavior is controlled by command line arguments, as described in the "Inputs" section.
## Logging
The script prints informative messages to the console if the `verbose` variable is set to `True`. This includes parameter values, progress updates, and timing information.
# _lbi_jungle.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/_lbi_jungle.py
## High-level description
The `LBIJungle` class in `cassiopeia/tools/fitness_estimator/_lbi_jungle.py` implements the Lineage Branching Index (LBI) fitness estimator as described by Neher et al. (2014) using the `jungle` package, which is a wrapper around Neher et al.'s original code. This estimator calculates the fitness of each node in a tree based on its branching pattern, with more branching indicating higher fitness.
## Code Structure
The `LBIJungle` class is a subclass of `FitnessEstimator` and overrides the `estimate_fitness` method. It uses the `_to_newick` function to convert the tree to a newick string, which is then used by the `jungle` package to estimate fitness.
## References
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.tools.fitness_estimator._FitnessEstimator`
* `jungle` package
## Symbols
### `LBIJungle`
#### Description
This class implements the LBI fitness estimator using the `jungle` package.
#### Inputs
| Name | Type | Description |
| :----------- | :------------- | :----------------------------------------------------------- |
| random\_seed | Optional\[int] | Random seed to use for fitness estimation. Defaults to None. |
#### Outputs
This class does not directly return any outputs. It modifies the input `CassiopeiaTree` object in place.
#### Internal Logic
1. **Check for invalid leaf names:** Raises a `FitnessEstimatorError` if any leaf name starts with an underscore, as this is not supported by the `jungle` package.
2. **Convert tree to newick format:** Uses the `_to_newick` function to convert the tree to a newick string.
3. **Estimate fitness using `jungle` package:**
* Creates a temporary file to store the newick string.
* Reads the newick string into a `jungle.Tree` object.
* Annotates the tree with standard node features.
* Infers fitness using the `infer_fitness` method of the `jungle.Tree` object.
4. **Set fitness attribute for each node:**
* Iterates through the nodes and their corresponding fitness values.
* Sets the `fitness` attribute of each node in the `CassiopeiaTree` object.
## Side Effects
* Modifies the input `CassiopeiaTree` object in place by setting the `fitness` attribute for each node.
## Dependencies
| Dependency | Purpose |
| :--------- | :-------------------------------------------------------- |
| jungle | Used to estimate fitness using the LBI method. |
| networkx | Used to represent and manipulate the tree topology. |
| numpy | Used for random number generation and array manipulation. |
### \_to\_newick
#### Description
This function converts a networkx graph to a newick string.
#### Inputs
| Name | Type | Description |
| :---------------------- | :--------- | :------------------------------------------------------------------------------------ |
| tree | nx.DiGraph | A networkx tree |
| record\_branch\_lengths | bool | Whether to record branch lengths on the tree in the newick string. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :------------- | :--- | :---------------------------------------------------- |
| newick\_string | str | A newick string representing the topology of the tree |
#### Internal Logic
The function uses a recursive approach to traverse the tree and build the newick string. It starts from the root node and recursively visits each child node. For each node, it checks if it is a leaf node. If it is, it appends the node name to the string. Otherwise, it recursively calls the function for each child node and encloses the results in parentheses. Branch lengths are added to the string if `record_branch_lengths` is set to True.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/fitness_estimator/overview
## High-level description
The `cassiopeia/tools/fitness_estimator` directory contains implementations of fitness estimation algorithms for phylogenetic trees, specifically designed for use with the Cassiopeia library. The main components include an abstract base class `FitnessEstimator` and a concrete implementation `LBIJungle` that uses the Lineage Branching Index (LBI) method.
## What does it do?
This module provides tools for estimating the fitness of nodes in phylogenetic trees. The main functionality includes:
1. Defining an interface for fitness estimation algorithms through the abstract `FitnessEstimator` class.
2. Implementing the LBI fitness estimator using the `jungle` package, which calculates fitness based on the branching patterns of the tree.
3. Converting Cassiopeia trees to Newick format for compatibility with external libraries.
4. Annotating nodes in the tree with fitness values.
The fitness estimation process helps in understanding the evolutionary dynamics of the sequences represented in the phylogenetic tree, with higher fitness values indicating potentially more successful or rapidly evolving lineages.
## Entry points
The main entry points for using this module are:
1. `FitnessEstimator`: An abstract base class that defines the interface for all fitness estimation algorithms in Cassiopeia.
2. `LBIJungle`: A concrete implementation of the `FitnessEstimator` class that uses the Lineage Branching Index method for fitness estimation.
Developers can use these classes to estimate fitness for `CassiopeiaTree` objects, which represent phylogenetic trees in the Cassiopeia library.
## Key Files
1. `_FitnessEstimator.py`: Defines the abstract `FitnessEstimator` class and the `FitnessEstimatorError` exception.
2. `_lbi_jungle.py`: Implements the `LBIJungle` class, which uses the `jungle` package to estimate fitness using the LBI method.
3. `__init__.py`: Serves as the top-level entry point for the module, exposing the main components.
## Dependencies
The module relies on several external libraries:
1. `jungle`: A wrapper around Neher et al.'s original code for LBI calculations.
2. `networkx`: Used for representing and manipulating tree topologies.
3. `numpy`: Used for random number generation and array manipulation.
4. `ete3`: For phylogenetic tree manipulation and visualization (used in the `_jungle` subdirectory).
5. `Bio.Phylo`: For interfacing with Biopython's phylogenetic tree representation (used in the `_jungle` subdirectory).
6. `scipy`: For various scientific computing tasks and statistical functions (used in the `_jungle` subdirectory).
7. `pandas`: For data manipulation and analysis (used in the `_jungle` subdirectory).
8. `matplotlib`: For visualization of results and trees (used in the `_jungle` subdirectory).
## Configuration
The main classes use constructor parameters and method arguments for configuration:
* `LBIJungle`:
* `random_seed`: Optional integer to set the random seed for reproducibility.
* `estimate_fitness` method:
* Takes a `CassiopeiaTree` object as input and modifies it in place by adding a 'fitness' attribute to each node.
Users can adjust these parameters to customize the fitness estimation process for their specific needs in evolutionary studies and population genetics research.
The `_jungle` subdirectory contains additional classes and functions for more advanced phylogenetic analysis, including:
* `Forest`: For managing collections of phylogenetic trees.
* `Tree`: For analyzing individual phylogenetic trees.
* `SFS`: For calculating and analyzing Site Frequency Spectra.
* `SizeMatchedModel`: For statistical modeling based on data size.
These components provide a comprehensive toolkit for in-depth analysis of evolutionary fitness and phylogenetic relationships in biological sequences.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/overview
## High-level description
The `cassiopeia/tools` directory contains a collection of utility functions and classes for analyzing phylogenetic trees and performing various calculations related to lineage tracing experiments. This toolset provides functionality for estimating parameters, computing evolutionary metrics, and performing statistical analyses on tree structures.
## What does it do?
The tools in this directory perform several key functions:
1. Autocorrelation analysis: Computes Moran's I statistic to measure spatial autocorrelation of numerical data associated with tree leaves.
2. Branch length estimation: Implements Maximum Likelihood Estimation (MLE) and Bayesian approaches for estimating branch lengths in phylogenetic trees.
3. Evolutionary coupling: Calculates how closely related different categories are based on their distribution across the tree structure.
4. Fitness estimation: Estimates the fitness of nodes in a phylogenetic tree using methods like Lineage-Based Inference (LBI).
5. Parameter estimation: Estimates mutation rates and missing data rates from tree structures and character matrices.
6. Small parsimony analysis: Implements algorithms for ancestral state reconstruction and parsimony scoring.
7. Topology analysis: Assesses topological properties of trees, such as balance and expansion, and computes metrics like cophenetic correlation.
8. Tree metrics: Calculates various metrics on phylogenetic trees, including parsimony scores and likelihood under different evolutionary models.
These tools enable researchers to extract meaningful information from lineage tracing data, analyze evolutionary relationships, and make inferences about the underlying biological processes.
## Entry points
The main entry points for developers are:
1. `autocorrelation.py`: `compute_morans_i` function for spatial autocorrelation analysis.
2. `branch_length_estimator/`: `IIDExponentialMLE` and `IIDExponentialBayesian` classes for branch length estimation.
3. `coupling.py`: `compute_evolutionary_coupling` function for category relationship analysis.
4. `fitness_estimator/`: `FitnessEstimator` abstract base class and `LBIJungle` implementation.
5. `parameter_estimators.py`: Functions for estimating mutation rates and missing data rates.
6. `small_parsimony.py`: Functions for ancestral state reconstruction and parsimony scoring.
7. `topology.py`: Functions for computing expansion p-values and cophenetic correlation.
8. `tree_metrics.py`: Functions for calculating parsimony and likelihood scores on trees.
## Dependencies
The tools in this directory rely on several external libraries:
1. numpy: For numerical computations and array operations.
2. pandas: For data manipulation and analysis.
3. scipy: For scientific computing and statistical functions.
4. networkx: For graph operations and tree representation.
5. ete3: For tree manipulation and visualization.
6. cvxpy: Used in the MLE branch length estimator.
7. Cython: Used for interfacing with C++ code in the Bayesian branch length estimator.
8. tqdm: For progress bars in long-running computations.
Additionally, the tools depend on other parts of the Cassiopeia package, particularly the `cassiopeia.data` module for tree data structures and the `cassiopeia.mixins` module for custom error classes.
## Configuration
Most of the tools in this directory do not require explicit configuration files. Instead, they use parameter-based configuration through function arguments and class constructors. Key configuration options include:
1. Random seeds for reproducibility in stochastic processes.
2. Thresholds and minimum values for various calculations (e.g., minimum clade size for expansion p-values).
3. Options for handling missing data and ancestral state reconstruction.
4. Choices between discrete and continuous models for certain calculations.
Users can customize the behavior of these tools by adjusting the input parameters when calling the functions or instantiating the classes.
This toolset provides a comprehensive suite of functions for analyzing phylogenetic trees and lineage tracing data, offering flexibility and power for researchers working with evolutionary data in the context of the Cassiopeia framework.
# parameter_estimators.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/parameter_estimators.py
## High-level description
This file provides functions for estimating parameters related to lineage tracing, specifically the mutation rate and missing data rates. These functions are used to infer the underlying biological processes from observed data in a CassiopeiaTree.
## References
This code references the `CassiopeiaTree` class from `cassiopeia.data.CassiopeiaTree` and the `ParameterEstimateError` and `ParameterEstimateWarning` exceptions from `cassiopeia.mixins`.
## Symbols
### `get_proportion_of_missing_data`
#### Description
Calculates the proportion of missing entries in the character matrix of a CassiopeiaTree. The missing state is indicated by the `tree.missing_state_indicator` attribute.
#### Inputs
| Name | Type | Description |
| :---- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the character matrix. |
| layer | str, optional | The name of the layer in the `tree.layers` dictionary to use for the character matrix. If None, the default `character_matrix` is used. |
#### Outputs
| Name | Type | Description |
| :------------------ | :---- | :------------------------------------------------------------------------ |
| missing\_proportion | float | The proportion of missing cell/character entries in the character matrix. |
#### Internal Logic
The function counts the number of entries in the character matrix that are equal to the `missing_state_indicator`. This count is then divided by the total number of entries in the matrix to calculate the proportion of missing data.
#### Error Handling
Raises a `ParameterEstimateError` if no character matrix is found in the tree.
### `get_proportion_of_mutation`
#### Description
Calculates the proportion of mutated entries in the character matrix of a CassiopeiaTree, excluding missing entries.
#### Inputs
| Name | Type | Description |
| :---- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the character matrix. |
| layer | str, optional | The name of the layer in the `tree.layers` dictionary to use for the character matrix. If None, the default `character_matrix` is used. |
#### Outputs
| Name | Type | Description |
| :------------------- | :---- | :--------------------------------------------------------------------- |
| mutation\_proportion | float | The proportion of non-missing cell/character entries that are mutated. |
#### Internal Logic
The function first calculates the number of missing entries. Then, it counts the number of non-missing entries that are not equal to 0 (uncut state). This count is then divided by the total number of non-missing entries to calculate the proportion of mutated entries.
#### Error Handling
Raises a `ParameterEstimateError` if no character matrix is found in the tree.
### `estimate_mutation_rate`
#### Description
Estimates the mutation rate from a CassiopeiaTree, considering either a continuous or per-generation model.
#### Inputs
| Name | Type | Description |
| :----------------------------- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the character matrix and tree topology. |
| continuous | bool, optional | Whether to calculate a continuous mutation rate (accounting for branch lengths) or a discrete per-generation rate. Defaults to True. |
| assume\_root\_implicit\_branch | bool, optional | Whether to assume an implicit branch leading from the root if it doesn't exist. Defaults to True. |
| layer | str, optional | The name of the layer in the `tree.layers` dictionary to use for the character matrix. If None, the default `character_matrix` is used. |
#### Outputs
| Name | Type | Description |
| :------------- | :---- | :--------------------------- |
| mutation\_rate | float | The estimated mutation rate. |
#### Internal Logic
The function first retrieves the proportion of mutated entries from the tree's parameters or calculates it using `get_proportion_of_mutation`. Then, it estimates the mutation rate based on the chosen model:
* **Discrete (per-generation):** The rate is estimated using the formula: `mutation_rate = 1 - (1 - mutation_proportion) ^ (1 / mean_depth)`, where `mean_depth` is the average depth of the tree.
* **Continuous:** The rate is estimated using the formula: `mutation_rate = -np.log(1 - mutation_proportion) / mean_time`, where `mean_time` is the average time of the tree.
If `assume_root_implicit_branch` is True and the tree doesn't have a single leading edge from the root, an implicit branch is added to the calculation of `mean_depth` or `mean_time`.
#### Error Handling
Raises a `ParameterEstimateError` if the `mutation_proportion` parameter is not between 0 and 1.
### `estimate_missing_data_rates`
#### Description
Estimates both the stochastic missing probability and the heritable missing rate from a CassiopeiaTree, given one of the two parameters.
#### Inputs
| Name | Type | Description |
| :------------------------------- | :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the character matrix and tree topology. |
| continuous | bool, optional | Whether to calculate a continuous missing rate (accounting for branch lengths) or a discrete per-generation rate. Defaults to True. |
| assume\_root\_implicit\_branch | bool, optional | Whether to assume an implicit branch leading from the root if it doesn't exist. Defaults to True. |
| stochastic\_missing\_probability | float, optional | The stochastic missing probability. If provided, overrides the value on the tree. |
| heritable\_missing\_rate | float, optional | The heritable missing rate. If provided, overrides the value on the tree. |
| layer | str, optional | The name of the layer in the `tree.layers` dictionary to use for the character matrix. If None, the default `character_matrix` is used. |
#### Outputs
| Name | Type | Description |
| :------------------------------- | :---- | :---------------------------------- |
| stochastic\_missing\_probability | float | The stochastic missing probability. |
| heritable\_missing\_rate | float | The heritable missing rate. |
#### Internal Logic
The function first retrieves the total missing proportion from the tree's parameters or calculates it using `get_proportion_of_missing_data`. It then checks if either the stochastic missing probability or the heritable missing rate is provided as an argument or stored in the tree's parameters. If both or neither are provided, it raises a `ParameterEstimateError`.
If only one of the parameters is provided, the function estimates the other parameter based on the chosen model (continuous or discrete) and the formula: `total_missing_proportion = heritable_proportion + stochastic_proportion - heritable_proportion * stochastic_proportion`.
The heritable proportion is calculated using either the average depth or average time of the tree, depending on the chosen model and whether to assume an implicit root branch.
#### Error Handling
* Raises a `ParameterEstimateError` if:
* The `total_missing_proportion`, `stochastic_missing_probability`, or `heritable_missing_rate` have invalid values (e.g., outside the range of 0 to 1).
* Both or neither of `stochastic_missing_probability` and `heritable_missing_rate` are provided.
* Raises a `ParameterEstimateWarning` if the estimated parameter is negative, indicating a potential issue with the provided parameter.
# small_parsimony.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/small_parsimony.py
## High-level description
This file provides tools for analyzing phylogenies using small-parsimony. It includes functions for Fitch-Hartigan reconstruction, parsimony scoring, and the FitchCount algorithm. These tools are used to infer ancestral states and quantify transitions between states in a phylogenetic tree.
## Code Structure
The code consists of several functions that work together to perform small-parsimony analysis on a `CassiopeiaTree` object. The main functions are `fitch_hartigan`, `score_small_parsimony`, and `fitch_count`. `fitch_hartigan` is further divided into `fitch_hartigan_bottom_up` and `fitch_hartigan_top_down` for the two phases of the algorithm. `fitch_count` uses two helper functions, `_N_fitch_count` and `_C_fitch_count`, to implement the dynamic programming algorithm.
## References
This code references the `CassiopeiaTree` class from `cassiopeia.data` and several exception classes from `cassiopeia.mixins.errors`.
## Symbols
### `fitch_hartigan`
#### Description
Runs the full Fitch-Hartigan small parsimony algorithm to infer the most-parsimonious set of ancestral states for a given categorical variable in a `CassiopeiaTree`. It first performs the bottom-up phase to determine the optimal set of states for each node, and then the top-down phase to select a random solution that satisfies the maximum-parsimony criterion.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :-------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object containing cell meta data. |
| meta\_item | str | The name of the column in the cell meta data representing the categorical variable. |
| root | Optional\[str] | The root node from which to start the refinement. If None, the tree's root is used. |
| state\_key | str | The attribute key to store the ancestral states from the bottom-up phase. Defaults to "S1". |
| label\_key | str | The attribute key to store the maximum-parsimony assignment from the top-down phase. Defaults to "label". |
| copy | bool | Whether to modify the tree in place or return a copy. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :--------------- | :------------------------ | :--------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | Optional\[CassiopeiaTree] | The modified tree object with inferred ancestral states, or a copy if `copy=True`. Returns None if `copy=False`. |
#### Internal Logic
1. Copies the tree if `copy=True`.
2. Runs `fitch_hartigan_bottom_up` to perform the bottom-up phase of the algorithm.
3. Runs `fitch_hartigan_top_down` to perform the top-down phase of the algorithm.
4. Returns the modified tree or a copy.
### `fitch_hartigan_bottom_up`
#### Description
Performs the bottom-up phase of the Fitch-Hartigan algorithm. It traverses the tree in post-order and infers the optimal set of ancestral states for each node based on the states of its children.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :---------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object containing cell meta data. |
| meta\_item | str | The name of the column in the cell meta data representing the categorical variable. |
| add\_key | str | The attribute key to store the inferred ancestral states. Defaults to "S1". |
| copy | bool | Whether to modify the tree in place or return a copy. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :--------------- | :------------------------ | :--------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | Optional\[CassiopeiaTree] | The modified tree object with inferred ancestral states, or a copy if `copy=True`. Returns None if `copy=False`. |
#### Internal Logic
1. Copies the tree if `copy=True`.
2. For each node in the tree, traversed in post-order:
* If the node is a leaf, its state is taken from the cell meta data.
* If the node is internal, the intersection of the states of its children is taken as the optimal set of states. If the intersection is empty, the union of the states is taken.
3. Returns the modified tree or a copy.
### `fitch_hartigan_top_down`
#### Description
Performs the top-down phase of the Fitch-Hartigan algorithm. It traverses the tree from the root and selects a random solution from the optimal set of states for each node, ensuring consistency with the state assigned to its parent.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :----------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object with ancestral states inferred from the bottom-up phase. |
| root | Optional\[str] | The root node from which to start the refinement. If None, the tree's root is used. |
| state\_key | str | The attribute key storing the ancestral states from the bottom-up phase. Defaults to "S1". |
| label\_key | str | The attribute key to store the maximum-parsimony assignment. Defaults to "label". |
| copy | bool | Whether to modify the tree in place or return a copy. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :--------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | Optional\[CassiopeiaTree] | The modified tree object with a maximum-parsimony assignment, or a copy if `copy=True`. Returns None if `copy=False`. |
#### Internal Logic
1. Copies the tree if `copy=True`.
2. For each node in the tree, traversed from the root:
* If the node is the root, a random state is chosen from its optimal set of states.
* If the node is not the root, the state assigned to its parent is checked. If the parent's state is in the node's optimal set of states, it is assigned to the node. Otherwise, a random state is chosen from the node's optimal set.
3. Returns the modified tree or a copy.
### `score_small_parsimony`
#### Description
Computes the small-parsimony score of a tree, which is the number of state changes along the edges of the tree.
#### Inputs
| Name | Type | Description |
| :----------------------- | :------------- | :-------------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object containing cell meta data. |
| meta\_item | str | The name of the column in the cell meta data representing the categorical variable. |
| root | Optional\[str] | The root node from which to start the score calculation. If None, the tree's root is used. |
| infer\_ancestral\_states | bool | Whether to infer ancestral states using `fitch_hartigan` before scoring. Defaults to True. |
| label\_key | Optional\[str] | The attribute key storing the inferred ancestral states if `infer_ancestral_states=False`. Defaults to "label". |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------- |
| parsimony | int | The parsimony score of the tree. |
#### Internal Logic
1. Copies the tree.
2. If `infer_ancestral_states=True`, runs `fitch_hartigan` to infer ancestral states.
3. Traverses the edges of the tree and increments the parsimony score for each edge where the parent and child nodes have different states.
4. Returns the parsimony score.
### `fitch_count`
#### Description
Implements the FitchCount algorithm to infer the number of transitions between states across all equally-parsimonious solutions returned by the Fitch-Hartigan algorithm.
#### Inputs
| Name | Type | Description |
| :----------------------- | :-------------------- | :----------------------------------------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object containing cell meta data. |
| meta\_item | str | The name of the column in the cell meta data representing the categorical variable. |
| root | Optional\[str] | The root node from which to start the count. If None, the tree's root is used. |
| infer\_ancestral\_states | bool | Whether to infer ancestral states using `fitch_hartigan_bottom_up` before counting. Defaults to True. |
| state\_key | str | The attribute key storing the inferred ancestral states if `infer_ancestral_states=False`. Defaults to "S1". |
| unique\_states | Optional\[List\[str]] | The state space for the categorical variable. If None, the unique values in the cell meta data are used. |
#### Outputs
| Name | Type | Description |
| :--- | :----------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| M | pd.DataFrame | An MxM count matrix, where M is the number of unique states. The value at (i, j) indicates the number of times state i transitioned to state j across all equally-parsimonious solutions. |
#### Internal Logic
1. Copies the tree.
2. Determines the state space based on `unique_states` or the unique values in the cell meta data.
3. If `infer_ancestral_states=True`, runs `fitch_hartigan_bottom_up` to infer ancestral states.
4. Creates mappings from nodes and states to unique integers.
5. Calls `_N_fitch_count` to compute the dynamic programming table N, which stores the number of solutions below a node given its state.
6. Calls `_C_fitch_count` to compute the dynamic programming table C, which stores the number of transitions between states below a node given its state.
7. Constructs the count matrix M from the values in C.
8. Returns the count matrix M.
### `_N_fitch_count`
#### Description
Helper function for `fitch_count` that computes the dynamic programming table N.
#### Inputs
| Name | Type | Description |
| :--------------- | :-------------- | :------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object. |
| unique\_states | List\[str] | The state space for the categorical variable. |
| node\_to\_i | Dict\[str, int] | A mapping from nodes to unique integers. |
| label\_to\_j | Dict\[str, int] | A mapping from states to unique integers. |
| state\_key | str | The attribute key storing the inferred ancestral states. Defaults to "S1". |
#### Outputs
| Name | Type | Description |
| :--- | :------------ | :-------------------------------------------------------------------------------------------------------------------------- |
| N | np.array(int) | A 2-dimensional array storing N\[v, s], the number of equally-parsimonious solutions below node v given v takes on state s. |
#### Internal Logic
1. Defines a helper function `_fill` to compute a single entry in N.
2. Initializes N with zeros.
3. For each node in the tree, and for each state in its optimal set of states, calls `_fill` to compute the corresponding entry in N.
4. Returns N.
### `_C_fitch_count`
#### Description
Helper function for `fitch_count` that computes the dynamic programming table C.
#### Inputs
| Name | Type | Description |
| :--------------- | :-------------- | :------------------------------------------------------------------------- |
| cassiopeia\_tree | CassiopeiaTree | The tree object. |
| N | np.array | The N array computed by `_N_fitch_count`. |
| unique\_states | List\[str] | The state space for the categorical variable. |
| node\_to\_i | Dict\[str, int] | A mapping from nodes to unique integers. |
| label\_to\_j | Dict\[str, int] | A mapping from states to unique integers. |
| state\_key | str | The attribute key storing the inferred ancestral states. Defaults to "S1". |
#### Outputs
| Name | Type | Description |
| :--- | :------------ | :----------------------------------------------------------------------------------------------------------------------------------------- |
| C | np.array(int) | A 4-dimensional array storing C\[v, s, s1, s2], the number of transitions from state s1 to state s2 below node v given v takes on state s. |
#### Internal Logic
1. Defines a helper function `_fill` to compute a single entry in C.
2. Initializes C with zeros.
3. For each node in the tree, for each state in its optimal set of states, and for each pair of states (s1, s2), calls `_fill` to compute the corresponding entry in C.
4. Returns C.
# topology.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/topology.py
Here's a documentation for the target file `cassiopeia/tools/topology.py`:
## High-level description
This file contains utility functions to assess topological properties of phylogenetic trees, specifically focusing on balance and expansion. It provides methods to compute expansion p-values and cophenetic correlation for CassiopeiaTree objects.
## Code Structure
The main functions in this file are `compute_expansion_pvalues` and `compute_cophenetic_correlation`. These functions operate on CassiopeiaTree objects and utilize various helper functions and statistical methods to analyze tree topology.
## Symbols
### `compute_expansion_pvalues`
#### Description
This function calculates expansion p-values for nodes in a phylogenetic tree, assessing the probability of observing a given subclade size under a neutral coalescent model.
#### Inputs
| Name | Type | Description |
| :--------------- | :------------- | :------------------------------------------------------ |
| tree | CassiopeiaTree | The phylogenetic tree to analyze |
| min\_clade\_size | int | Minimum number of leaves in a subtree to be considered |
| min\_depth | int | Minimum depth of clade to be considered |
| copy | bool | Whether to return a copy of the tree or modify in-place |
#### Outputs
| Name | Type | Description |
| :--- | :--------------------- | :------------------------------------------------------------ |
| tree | CassiopeiaTree or None | The tree with added expansion p-values (if copy=True) or None |
#### Internal Logic
1. Initializes attributes for each node in the tree
2. Traverses the tree in depth-first order
3. For each node, calculates the expansion p-value based on the number of leaves in its subtree
4. Adds the calculated p-values as attributes to the tree nodes
#### Performance Considerations
The function runs in O(n log n) time for balanced trees, but can be up to O(n^3) for highly unbalanced trees.
### `compute_cophenetic_correlation`
#### Description
Computes the cophenetic correlation of a lineage, which is the Pearson correlation between the phylogenetic distance and character dissimilarity.
#### Inputs
| Name | Type | Description |
| :---------------------- | :---------------------- | :-------------------------------------------------- |
| tree | CassiopeiaTree | The phylogenetic tree to analyze |
| weights | pd.DataFrame (optional) | Phylogenetic weights matrix |
| dissimilarity\_map | pd.DataFrame (optional) | Dissimilarity matrix between samples |
| dissimilarity\_function | Callable | Function to compute dissimilarities between samples |
#### Outputs
| Name | Type | Description |
| :----------- | :---- | :---------------------------------- |
| correlation | float | The cophenetic correlation value |
| significance | float | The significance of the correlation |
#### Internal Logic
1. Computes or uses provided phylogenetic weight matrix and dissimilarity map
2. Aligns matrices to ensure they correspond to the same set of leaves
3. Converts matrices to condensed distance matrices
4. Calculates Pearson correlation between phylogenetic distances and dissimilarities
#### Performance Considerations
If weights and dissimilarity map are not precomputed, the function runs in O(mn^2 + n^2logn + n^2) time, where n is the number of leaves and m is the number of characters.
### `simple_coalescent_probability`
#### Description
Computes the probability of observing a given number of samples in a lineage under a simple coalescent model.
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------------ |
| n | int | Number of leaves in subtree |
| b | int | Number of leaves in one lineage |
| k | int | Number of lineages |
#### Outputs
| Name | Type | Description |
| :---------- | :---- | :--------------------------------------------------------------------------- |
| probability | float | Probability of observing b leaves on one lineage in a tree of n total leaves |
### `nCk`
#### Description
Computes the binomial coefficient (n choose k).
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------ |
| n | int | Total number of items |
| k | int | Number of items to choose |
#### Outputs
| Name | Type | Description |
| :---------- | :---- | :------------------------------------------ |
| coefficient | float | The number of ways to choose k items from n |
## Dependencies
* numpy
* pandas
* scipy
* cassiopeia.data
* cassiopeia.mixins
* cassiopeia.solver.dissimilarity\_functions
## Error Handling
The code uses custom exceptions from `cassiopeia.mixins` to handle errors specific to CassiopeiaTree operations.
# tree_metrics.py
Source: https://demo.agenticlabs.com/detailed-code-docs/cassiopeia/tools/tree_metrics.py
## High-level description
This file provides functions for calculating tree metrics, specifically parsimony and likelihood, for a given `CassiopeiaTree`. It includes functions for calculating the number of mutations on a tree, the log likelihood of a character on a tree, and wrapper functions for calculating the overall likelihood of a tree under both discrete and continuous models of lineage tracing.
## References
This code references the `CassiopeiaTree` class from `cassiopeia.data` and the `parameter_estimators` module from `cassiopeia.tools`.
## Symbols
### `calculate_parsimony`
#### Description
Calculates the parsimony score of a tree, defined as the number of character/state mutations that occur on the edges of the tree.
#### Inputs
| Name | Type | Description |
| :--------------------------- | :------------- | :---------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree to calculate parsimony over |
| infer\_ancestral\_characters | bool | Whether to infer the ancestral character states of the tree using Camin-Sokal Parsimony. Defaults to False. |
| treat\_missing\_as\_mutation | bool | Whether to treat transitions from a non-missing state to a missing state as mutations. Defaults to False. |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :----------------------------------------------------- |
| parsimony | int | The number of mutations that have occurred on the tree |
#### Internal Logic
1. If `infer_ancestral_characters` is True, the ancestral character states of the tree are inferred using Camin-Sokal Parsimony.
2. The function iterates over all edges in the tree.
3. For each edge, it retrieves the character states of the parent and child nodes.
4. It then calculates the number of mutations along the edge using the `get_mutations_along_edge` method of the `CassiopeiaTree` object.
5. The total number of mutations is summed over all edges and returned.
### `log_transition_probability`
#### Description
Calculates the log transition probability between two given states for a specific character.
#### Inputs
| Name | Type | Description |
| :---------------------------------------- | :------------------------- | :------------------------------------------------------------------------------------------------------ |
| tree | CassiopeiaTree | The tree on which the priors are stored |
| character | int | The index of the character |
| s | Union\[int, str] | The original state |
| s\_ | Union\[int, str] | The state being transitioned to |
| t | float | The length of time that the transition can occur along |
| mutation\_probability\_function\_of\_time | Callable\[\[float], float] | The function defining the probability of a lineage acquiring a mutation within a given time |
| missing\_probability\_function\_of\_time | Callable\[\[float], float] | The function defining the probability of a lineage acquiring heritable missing data within a given time |
#### Outputs
| Name | Type | Description |
| :--------------- | :---- | :------------------------------------------------ |
| log\_probability | float | The log transition probability between the states |
#### Internal Logic
The function calculates the log transition probability based on the following assumptions:
1. State `0` represents the uncut state.
2. Only state `0` can transition to non-zero states.
3. Any non-missing state can mutate to the missing state, specified by `tree.missing_state_indicator`.
4. The probability of acquiring a mutation is given by the time `t` and the `mutation_probability_function_of_time`.
5. The probability of acquiring heritable missing data is given by the time `t` and the `missing_probability_function_of_time`.
6. The priors on the tree (`tree.priors`) are used to determine the probability of acquiring a non-missing, non-zero state.
The function uses a series of conditional statements to handle different combinations of `s` and `s_` and returns the corresponding log transition probability.
### `log_likelihood_of_character`
#### Description
Calculates the log likelihood of a given character on the tree using Felsenstein's Pruning Algorithm.
#### Inputs
| Name | Type | Description |
| :---------------------------------------- | :------------------------- | :-------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree on which to calculate the likelihood |
| character | int | The index of the character to calculate the likelihood of |
| use\_internal\_character\_states | bool | Indicates if internal node character states should be assumed to be specified exactly. Defaults to False. |
| mutation\_probability\_function\_of\_time | Callable\[\[float], float] | The function defining the probability of a lineage acquiring a mutation within a given time |
| missing\_probability\_function\_of\_time | Callable\[\[float], float] | The function defining the probability of a lineage acquiring heritable missing data within a given time |
| stochastic\_missing\_probability | float | The probability that a cell/character pair acquires stochastic missing data at the end of the lineage |
| implicit\_root\_branch\_length | float | The length of the implicit root branch. Used if the implicit root needs to be added. Defaults to 1. |
#### Outputs
| Name | Type | Description |
| :-------------- | :---- | :---------------------------------------------- |
| log\_likelihood | float | The log likelihood of the tree on one character |
#### Internal Logic
1. The function performs a depth-first traversal of the tree in postorder.
2. For each node, it calculates the likelihood of each possible state at that node based on the likelihoods of its children and the transition probabilities between states.
3. If `use_internal_character_states` is False, the function assumes an implicit root with the uncut state (0) at each character and marginalizes over the transition from the implicit root to its child.
4. If `use_internal_character_states` is True, the function returns the likelihood of the state annotated at the root for the given character.
### `get_lineage_tracing_parameters`
#### Description
Retrieves or estimates the lineage tracing parameters (mutation rate, heritable missing rate, and stochastic missing probability) from a `CassiopeiaTree`.
#### Inputs
| Name | Type | Description |
| :----------------------------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree from which to retrieve or estimate the parameters |
| continuous | bool | If the parameters are to be estimated, whether to estimate them as continuous or discrete parameters |
| assume\_root\_implicit\_branch | bool | In the tree depth/time in estimating the rate parameters, whether or not to include an implicit root |
| layer | Optional\[str] | Layer to use for the character matrix in estimating parameters. If None, the current `character_matrix` variable will be used. Defaults to None. |
#### Outputs
| Name | Type | Description |
| :------------------------------- | :---- | :--------------------------------- |
| mutation\_rate | float | The mutation rate |
| heritable\_missing\_rate | float | The heritable missing rate |
| stochastic\_missing\_probability | float | The stochastic missing probability |
#### Internal Logic
1. The function attempts to retrieve the parameters from `tree.parameters`.
2. If any of the parameters are not found, they are estimated using the `estimate_mutation_rate` and `estimate_missing_data_rates` functions.
3. The function checks if the estimated parameters have valid values and raises a `TreeMetricError` if they are invalid.
### `calculate_likelihood_discrete`
#### Description
Calculates the log likelihood of a tree under a discrete model of lineage tracing.
#### Inputs
| Name | Type | Description |
| :------------------------------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree on which to calculate likelihood over |
| use\_internal\_character\_states | bool | Indicates if internal node character states should be assumed to be specified exactly. Defaults to False. |
| layer | Optional\[str] | Layer to use for the character matrix in estimating parameters. If None, the current `character_matrix` variable will be used. Defaults to None. |
#### Outputs
| Name | Type | Description |
| :-------------- | :---- | :----------------------------------------------------------------- |
| log\_likelihood | float | The log likelihood of the tree given the observed character states |
#### Internal Logic
1. The function retrieves the lineage tracing parameters using `get_lineage_tracing_parameters`.
2. It calculates the log likelihood for each character using `log_likelihood_of_character` with discrete rate functions.
3. The log likelihoods of all characters are summed to obtain the overall log likelihood of the tree.
### `calculate_likelihood_continuous`
#### Description
Calculates the log likelihood of a tree under a continuous model of lineage tracing.
#### Inputs
| Name | Type | Description |
| :------------------------------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The tree on which to calculate likelihood over |
| use\_internal\_character\_states | bool | Indicates if internal node character states should be assumed to be specified exactly. Defaults to False. |
| layer | Optional\[str] | Layer to use for the character matrix in estimating parameters. If None, the current `character_matrix` variable will be used. Defaults to None. |
#### Outputs
| Name | Type | Description |
| :-------------- | :---- | :----------------------------------------------------------------- |
| log\_likelihood | float | The log likelihood of the tree given the observed character states |
#### Internal Logic
1. The function retrieves the lineage tracing parameters using `get_lineage_tracing_parameters`.
2. It calculates the log likelihood for each character using `log_likelihood_of_character` with continuous rate functions.
3. The log likelihoods of all characters are summed to obtain the overall log likelihood of the tree.
## Error Handling
The functions in this file raise `TreeMetricError` exceptions if the input tree is not initialized or if the character states are not properly initialized. They also raise `TreeMetricError` exceptions if the lineage tracing parameters have invalid values.
# codecov.yml
Source: https://demo.agenticlabs.com/detailed-code-docs/codecov.yml
## High-level description
This YAML file configures Codecov, a code coverage reporting tool. It sets specific coverage requirements for the project and disables patch coverage checks.
## Symbols
### coverage
#### Description
The `coverage` key is the main configuration block for Codecov settings. It contains nested configurations for different aspects of coverage reporting.
### status
#### Description
The `status` key, nested under `coverage`, defines how Codecov should report the status of code coverage.
### project
#### Description
The `project` key, nested under `status`, configures settings for the overall project coverage.
### default
#### Description
The `default` key, nested under `project`, specifies the default coverage requirements for the project.
#### Internal Logic
* Sets a target coverage percentage of 85%
* Allows a threshold of 1% deviation from the target
### patch
#### Description
The `patch` key, nested under `status`, configures settings for patch coverage (coverage of changed lines in pull requests).
#### Internal Logic
* Set to `off`, which disables patch coverage checks
## Configuration
| Option | Type | Default | Description |
| :-------- | :--------- | :------ | :--------------------------------------------------- |
| target | percentage | 85% | The desired code coverage percentage for the project |
| threshold | percentage | 1% | The allowed deviation from the target coverage |
| patch | string | off | Configuration for patch coverage checks |
## TODOs
The file includes a comment suggesting a method to validate the YAML configuration:
```
# Run to check if valid
# curl --data-binary @codecov.yml https://codecov.io/validate
```
This command can be used to ensure the YAML file is correctly formatted and valid according to Codecov's requirements.
# conftest.py
Source: https://demo.agenticlabs.com/detailed-code-docs/conftest.py
## High-level description
This file, `conftest.py`, is a pytest configuration file that sets up custom options and behaviors for running tests. It specifically introduces a mechanism to mark and optionally skip slow tests, allowing for more efficient test execution in different scenarios.
## Symbols
### `pytest_addoption`
#### Description
This function adds a custom command-line option `--runslow` to pytest.
#### Inputs
| Name | Type | Description |
| :----- | :------------ | :------------------------------------- |
| parser | pytest.Parser | The pytest command-line options parser |
#### Internal Logic
The function adds a boolean option `--runslow` to the pytest command-line interface. When used, this option allows slow tests to run.
### `pytest_configure`
#### Description
This function adds a custom marker `slow` to pytest's configuration.
#### Inputs
| Name | Type | Description |
| :----- | :------------ | :------------------------------ |
| config | pytest.Config | The pytest configuration object |
#### Internal Logic
The function adds a new line to pytest's ini-file configuration, defining a `slow` marker that can be used to identify slow tests.
### `pytest_collection_modifyitems`
#### Description
This function modifies the collected test items based on whether the `--runslow` option is provided.
#### Inputs
| Name | Type | Description |
| :----- | :----------------- | :------------------------------- |
| config | pytest.Config | The pytest configuration object |
| items | List\[pytest.Item] | The list of collected test items |
#### Internal Logic
1. Checks if the `--runslow` option is provided.
2. If not provided, it creates a `skip_slow` marker.
3. Iterates through all test items and adds the `skip_slow` marker to any test marked as "slow".
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------- |
| pytest | The testing framework being configured |
## Configuration
| Option | Type | Default | Description |
| :-------- | :------ | :------ | :--------------------------------- |
| --runslow | boolean | False | When set, allows slow tests to run |
## API/Interface Reference
| Marker | Description |
| :----- | :-------------------------- |
| slow | Marks a test as slow to run |
This configuration file extends pytest's functionality by introducing a way to mark tests as slow and optionally skip them during test execution. It provides a command-line option `--runslow` that, when not specified, will cause all tests marked as "slow" to be skipped. This allows for faster test runs in scenarios where running slow tests is not necessary or desired, while still maintaining the ability to run all tests when needed.
# override.css
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_static/css/override.css
## High-level description
This CSS file overrides default styles in the Furo Sphinx theme to customize the appearance of Streamlink's documentation. It specifically targets elements like code blocks, citations, headings, and buttons to enhance readability and visual appeal.
The file also includes styles for elements specific to the "Getting Started" page, such as intro cards and custom buttons, and it adjusts the display of Jupyter Notebook elements for better integration within the documentation.
## Code Structure
This CSS file is structured as a series of style rules. Each rule targets specific HTML elements or classes and applies styles to modify their appearance. There's no complex interrelation between the rules, they primarily function independently to style different aspects of the documentation.
## Symbols
### `dt:target, span.highlighted`
#### Description
This rule styles the definition term (`<dt>`) of a definition list when it's the target of a hyperlink and any element with the class "highlighted". It gives them a light gray background color.
#### Internal Logic
The rule uses the `:target` pseudo-class to select the `dt` element that is currently being linked to. It also selects any element with the class `highlighted`. Both are styled with a light gray background.
### `code.docutils.literal.notranslate.xref, a code`
#### Description
This rule styles inline code snippets within links (`<a>`) and cross-references (`xref`). It removes any background color, sets the font weight to bold, and inherits the text color from the parent element.
#### Internal Logic
The rule targets `<code>` elements with specific classes (`docutils`, `literal`, `notranslate`, `xref`) that are either direct children of `<a>` elements or standalone within an `<a>` element. It then applies the defined styles.
### `a > code`
#### Description
This rule ensures that inline code snippets within links inherit the link's text color.
#### Internal Logic
It targets `<code>` elements that are direct children of `<a>` elements and sets the `color` property to `inherit`.
### `code.docutils.literal.notranslate`
#### Description
This rule styles standard inline code snippets. It applies a light gray background, reduces font size slightly, rounds the corners, sets text color to black, allows line breaks within the code, and adds padding.
#### Internal Logic
The rule targets `<code>` elements with specific classes (`docutils`, `literal`, `notranslate`) and applies the defined styles to create a visually distinct code block.
### `dl.citation > dt`
#### Description
This rule styles the definition term in a citation list. It floats the term to the left, adds a right margin, and sets the font weight to bold.
#### Internal Logic
The rule targets `dt` elements that are direct children of `dl` elements with the class `citation`. It then applies the float, margin, and font weight styles.
### `dl.c .field-list dt, dl.cpp .field-list dt, dl.js .field-list dt, dl.py .field-list dt`
#### Description
This rule normalizes the font size and removes text transformation for parameter lists in code blocks for various programming languages (C, C++, JavaScript, Python).
#### Internal Logic
The rule targets `dt` elements within nested `dl` and `.field-list` elements that are descendants of `dl` elements with specific language classes (`c`, `cpp`, `js`, `py`). It then sets the font size and removes any text transformation.
### `p.rubric`
#### Description
This rule styles paragraphs with the class "rubric" (often used for examples and headings within classes). It sets the font size, removes text transformation, and sets a medium font weight.
#### Internal Logic
The rule targets `<p>` elements with the class `rubric` and applies the defined font size, text transformation, and font weight styles.
### `.intro-card`
#### Description
This rule styles elements with the class "intro-card", likely used on the "Getting Started" page. It sets a white background, removes border radius, adds padding, and sets top and bottom margins.
#### Internal Logic
The rule targets elements with the class `intro-card` and applies the background, border radius, padding, and margin styles.
### `.intro-card .card-text`
#### Description
This rule styles elements with the class "card-text" that are descendants of elements with the class "intro-card". It sets top and bottom margins.
#### Internal Logic
The rule targets elements with the class `card-text` that are nested within elements with the class `intro-card` and applies the margin styles.
### `.custom-button`
#### Description
This rule styles elements with the class "custom-button". It sets a light gray background, removes default border, sets text color to dark gray, centers text, removes underlines from links, styles it as an inline-block element, sets font size, rounds corners, sets maximum width, and adds padding.
#### Internal Logic
The rule targets elements with the class `custom-button` and applies the defined styles to create a custom button appearance.
### `.custom-button a`
#### Description
This rule styles links within elements with the class "custom-button". It sets the link color to dark gray and makes it a block-level element.
#### Internal Logic
The rule targets `<a>` elements that are descendants of elements with the class `custom-button` and applies the color and display styles.
### `.custom-button p`
#### Description
This rule styles paragraphs within elements with the class "custom-button". It removes top and bottom margins and sets the text color to dark gray.
#### Internal Logic
The rule targets `<p>` elements that are descendants of elements with the class `custom-button` and applies the margin and color styles.
### `.nbinput .prompt, .nboutput .prompt`
#### Description
This rule hides the prompt elements within Jupyter Notebook input and output cells.
#### Internal Logic
The rule targets elements with the class `prompt` that are descendants of elements with the classes `nbinput` and `nboutput` and sets their display to `none`.
### `.nboutput .stderr`
#### Description
This rule hides the standard error output within Jupyter Notebook output cells.
#### Internal Logic
The rule targets elements with the class `stderr` that are descendants of elements with the class `nboutput` and sets their display to `none`.
### `div.nblast.container`
#### Description
This rule styles the container for the last cell in a Jupyter Notebook. It adds padding to the bottom, removes padding from the sides, and adds a bottom margin.
#### Internal Logic
The rule targets `div` elements with the classes `nblast` and `container` and applies the padding and margin styles.
### `div.nbinput.container, div.nboutput.container`
#### Description
This rule styles the containers for input and output cells in a Jupyter Notebook. It uses flexbox for layout, aligns items to the start, sets margins, removes padding, and sets the width to 100%.
#### Internal Logic
The rule targets `div` elements with the class combinations `nbinput` and `container`, and `nboutput` and `container`. It then applies the flexbox properties, alignment, margins, padding, and width.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_static/css/overview
## High-level description
This directory contains CSS files that customize the appearance of Streamlink's documentation. The main file, `override.css`, modifies the default styles of the Furo Sphinx theme, while `sphinx_gallery.css` focuses on enhancing the visual presentation of Sphinx Gallery thumbnails.
## What does it do?
These CSS files work together to improve the readability and visual appeal of Streamlink's documentation. They accomplish this by:
1. Customizing the appearance of code blocks, citations, headings, and buttons.
2. Styling elements specific to the "Getting Started" page, such as intro cards and custom buttons.
3. Adjusting the display of Jupyter Notebook elements for better integration within the documentation.
4. Enhancing the presentation of Sphinx Gallery thumbnails, including their containers, links, timings, and figure alignment.
## Key Files
1. `override.css`: This is the main CSS file that overrides the default styles of the Furo Sphinx theme. It contains a wide range of style rules targeting various elements of the documentation.
2. `sphinx_gallery.css`: This file focuses specifically on styling Sphinx Gallery thumbnails, improving their visual presentation within the documentation.
## Dependencies
The CSS files in this directory rely on the following dependencies:
1. Furo Sphinx theme: The base theme that `override.css` customizes.
2. Sphinx: The documentation generator that uses these CSS files.
3. Sphinx Gallery: Provides the HTML structure and classes targeted by `sphinx_gallery.css`.
## Configuration
While these CSS files don't have explicit configuration options, they do use specific class names and HTML structures. Here are some key styling targets:
1. Code blocks: Inline code snippets and code blocks are styled for better readability.
```css theme={null}
code.docutils.literal.notranslate {
background-color: #f0f0f0;
font-size: 0.9em;
border-radius: 3px;
color: black;
word-break: break-word;
padding: 0.1em 0.2em;
}
```
2. Citations: Citation lists are styled for better visual separation.
```css theme={null}
dl.citation > dt {
float: left;
margin-right: 15px;
font-weight: bold;
}
```
3. Custom buttons: Used on the "Getting Started" page.
```css theme={null}
.custom-button {
background-color: #f0f0f0;
border: none;
color: #333;
text-align: center;
text-decoration: none;
display: inline-block;
font-size: 16px;
border-radius: 5px;
max-width: 300px;
padding: 10px 20px;
}
```
4. Jupyter Notebook elements: Styled for better integration with the documentation.
```css theme={null}
.nbinput .prompt, .nboutput .prompt {
display: none;
}
div.nbinput.container, div.nboutput.container {
display: flex;
align-items: flex-start;
margin: 0;
padding: 0;
width: 100%;
}
```
5. Sphinx Gallery thumbnails: Styled in `sphinx_gallery.css` for improved visual presentation.
These CSS files play a crucial role in shaping the look and feel of Streamlink's documentation, ensuring that it's not only informative but also visually appealing and easy to navigate.
# sphinx_gallery.css
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_static/css/sphinx_gallery.css
## High-level description
This CSS file provides custom styling for Sphinx Gallery thumbnails, aiming to enhance their visual presentation on the Sphinx documentation website. It specifically targets elements like thumbnail containers, links, timings, and figure alignment.
## Symbols
There are no named symbols (functions, classes, etc.) in this code. It consists solely of CSS rules targeting specific HTML elements and classes used by Sphinx Gallery.
## Dependencies
This CSS code depends on the following:
| Dependency | Purpose |
| :------------- | :----------------------------------------------------------------------------------- |
| Sphinx Gallery | Provides the HTML structure and classes targeted by this CSS. |
| Sphinx | The documentation generator that uses Sphinx Gallery and likely integrates this CSS. |
## Configuration
This CSS code does not define any configuration options. It directly applies styles to the targeted elements.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_static/overview
## High-level description
This directory contains CSS files that customize the appearance of Streamlink's documentation. The main file, `override.css`, modifies the default styles of the Furo Sphinx theme, while `sphinx_gallery.css` focuses on enhancing the visual presentation of Sphinx Gallery thumbnails.
## What does it do?
These CSS files work together to improve the readability and visual appeal of Streamlink's documentation. They accomplish this by:
1. Customizing the appearance of code blocks, citations, headings, and buttons.
2. Styling elements specific to the "Getting Started" page, such as intro cards and custom buttons.
3. Adjusting the display of Jupyter Notebook elements for better integration within the documentation.
4. Enhancing the presentation of Sphinx Gallery thumbnails, including their containers, links, timings, and figure alignment.
## Key Files
1. `override.css`: This is the main CSS file that overrides the default styles of the Furo Sphinx theme. It contains a wide range of style rules targeting various elements of the documentation.
2. `sphinx_gallery.css`: This file focuses specifically on styling Sphinx Gallery thumbnails, improving their visual presentation within the documentation.
## Dependencies
The CSS files in this directory rely on the following dependencies:
1. Furo Sphinx theme: The base theme that `override.css` customizes.
2. Sphinx: The documentation generator that uses these CSS files.
3. Sphinx Gallery: Provides the HTML structure and classes targeted by `sphinx_gallery.css`.
## Configuration
While these CSS files don't have explicit configuration options, they do use specific class names and HTML structures. Here are some key styling targets:
1. Code blocks: Inline code snippets and code blocks are styled for better readability.
```css theme={null}
code.docutils.literal.notranslate {
background-color: #f0f0f0;
font-size: 0.9em;
border-radius: 3px;
color: black;
word-break: break-word;
padding: 0.1em 0.2em;
}
```
2. Citations: Citation lists are styled for better visual separation.
```css theme={null}
dl.citation > dt {
float: left;
margin-right: 15px;
font-weight: bold;
}
```
3. Custom buttons: Used on the "Getting Started" page.
```css theme={null}
.custom-button {
background-color: #f0f0f0;
border: none;
color: #333;
text-align: center;
text-decoration: none;
display: inline-block;
font-size: 16px;
border-radius: 5px;
max-width: 300px;
padding: 10px 20px;
}
```
4. Jupyter Notebook elements: Styled for better integration with the documentation.
```css theme={null}
.nbinput .prompt, .nboutput .prompt {
display: none;
}
div.nbinput.container, div.nboutput.container {
display: flex;
align-items: flex-start;
margin: 0;
padding: 0;
width: 100%;
}
```
5. Sphinx Gallery thumbnails: Styled in `sphinx_gallery.css` for improved visual presentation.
These CSS files play a crucial role in shaping the look and feel of Streamlink's documentation, ensuring that it's not only informative but also visually appealing and easy to navigate.
# layout.html
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_templates/layout.html
## High-level description
This code defines a custom layout template for a Sphinx documentation site, inheriting from the `pydata_sphinx_theme/layout.html` template and overriding the `fonts` block to specify font families for the body and header elements.
## Symbols
### `{% extends "pydata_sphinx_theme/layout.html" %}`
#### Description
This line instructs the template engine to inherit the base layout from the `pydata_sphinx_theme/layout.html` file. This means that the current template will use the structure and elements defined in the base template, and only the overridden blocks will be different.
### `{% block fonts %}` ... `{% endblock %}`
#### Description
This block overrides the `fonts` block defined in the base template. It allows customizing the fonts used in the documentation site.
#### Internal Logic
The code within this block defines a `<style>` tag containing CSS rules for the `body` and `.header-style` elements.
* `body { font-family: ...; }`: This rule sets the font family for the entire body of the documentation pages. It uses a font stack starting with system fonts for better cross-platform consistency and falls back to generic sans-serif fonts.
* `.header-style { font-family: ...; }`: This rule sets the font family for elements with the class `header-style`. It uses the same font stack as the `body` rule.
## Dependencies
This template depends on the **pydata-sphinx-theme** package, which provides the base layout template (`pydata_sphinx_theme/layout.html`).
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/_templates/overview
## High-level description
This code defines a custom layout template for a Sphinx documentation site, inheriting from the `pydata_sphinx_theme/layout.html` template and overriding the `fonts` block to specify font families for the body and header elements.
## Symbols
### `{% extends "pydata_sphinx_theme/layout.html" %}`
#### Description
This line instructs the template engine to inherit the base layout from the `pydata_sphinx_theme/layout.html` file. This means that the current template will use the structure and elements defined in the base template, and only the overridden blocks will be different.
### `{% block fonts %}` ... `{% endblock %}`
#### Description
This block overrides the `fonts` block defined in the base template. It allows customizing the fonts used in the documentation site.
#### Internal Logic
The code within this block defines a `<style>` tag containing CSS rules for the `body` and `.header-style` elements.
* `body { font-family: ...; }`: This rule sets the font family for the entire body of the documentation pages. It uses a font stack starting with system fonts for better cross-platform consistency and falls back to generic sans-serif fonts.
* `.header-style { font-family: ...; }`: This rule sets the font family for elements with the class `header-style`. It uses the same font stack as the `body` rule.
## Dependencies
This template depends on the **pydata-sphinx-theme** package, which provides the base layout template (`pydata_sphinx_theme/layout.html`).
# conf.py
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/conf.py
## High-level description
This file, `docs/conf.py`, is a configuration file for Sphinx, a documentation generation tool. It sets up various options and extensions for building the documentation for the Cassiopeia project, including autodoc, intersphinx, and custom extensions for generating API documentation.
## Code Structure
The code is structured as a series of configuration settings for Sphinx. It imports necessary modules, sets up paths, and configures various Sphinx extensions and options. The file also defines a custom AutoAutoSummary class for generating summaries of class methods and attributes.
## Symbols
### Configuration Settings
#### Description
Various configuration settings for Sphinx are defined throughout the file. These include:
* Path setup
* Extension configuration
* Project information
* HTML output options
* Intersphinx mapping
* Custom CSS and static files
#### Inputs
N/A
#### Outputs
N/A
### AutoAutoSummary
#### Description
A custom Sphinx directive that automatically generates summaries for class methods and attributes.
#### Inputs
| Name | Type | Description |
| :---- | :--- | :------------------------------------------------- |
| clazz | str | The fully qualified name of the class to summarize |
#### Outputs
| Name | Type | Description |
| :------ | :--- | :---------------------------------------------------------------- |
| content | list | A list of method or attribute names to be included in the summary |
#### Internal Logic
1. Parses the class name from the input
2. Imports the class
3. Retrieves methods or attributes based on the options provided
4. Generates a list of content items for the summary
### setup
#### Description
A function to set up the Sphinx application by adding the custom AutoAutoSummary directive.
#### Inputs
| Name | Type | Description |
| :--- | :----------------- | :------------------------------ |
| app | Sphinx application | The Sphinx application instance |
#### Outputs
N/A
## Dependencies
| Dependency | Purpose |
| :------------------------- | :--------------------------------- |
| sphinx | Main documentation generation tool |
| nbsphinx | Jupyter notebook integration |
| sphinx\_autodoc\_typehints | Type hints in documentation |
| scanpydoc | Additional documentation utilities |
| pydata\_sphinx\_theme | HTML theme for documentation |
## Configuration
| Option | Type | Default | Description |
| :--------------------- | :--- | :-------------------------------------- | :------------------------------------- |
| needs\_sphinx | str | "3.4" | Minimum required Sphinx version |
| extensions | list | \[...] | List of Sphinx extensions to use |
| exclude\_patterns | list | \["\_build", "\*\*.ipynb\_checkpoints"] | Patterns to exclude from documentation |
| autodoc\_mock\_imports | list | \["gurobipy"] | Modules to mock for autodoc |
| html\_theme | str | "pydata\_sphinx\_theme" | HTML theme to use |
## TODOs
* The commented out line `# "scanpydoc.autosummary_generate_imported",` suggests a potential future addition to the extensions list.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/extensions/overview
## High-level description
This code provides a Sphinx extension that customizes the parsing of the "Returns" section in NumPy-style docstrings. It specifically formats the return values to include hyperlinks to their types using the `:class:` role.
## References
This code references the `NumpyDocstring` class from the `sphinx.ext.napoleon` module.
## Symbols
### `process_return`
#### Description
This function iterates through a list of lines (presumably from a docstring's "Returns" section) and reformats lines that match a specific pattern.
#### Inputs
| Name | Type | Description |
| :---- | :--------- | :----------------------------------------------------------------------- |
| lines | list\[str] | A list of strings representing lines in a docstring's "Returns" section. |
#### Outputs
| Name | Type | Description |
| :--- | :-------------- | :------------------------------------------------------------------ |
| | generator\[str] | Yields reformatted lines or the original line if no match is found. |
#### Internal Logic
1. Iterates through each line in the input `lines`.
2. Attempts to match the line with the regular expression `r"(?P<param>\w+)\s+:\s+(?P<type>[\w.]+)"`. This pattern captures a "parameter" and its "type" separated by a colon.
3. If a match is found:
* It reformats the line to `f'-{m["param"]} (:class:`\~`)'`, which creates a hyperlink for the type using the `:class:` role.
4. If no match is found, it yields the original line.
### `scanpy_parse_returns_section`
#### Description
This function overrides the `_parse_returns_section` method of the `NumpyDocstring` class. It processes the "Returns" section of a docstring to format return values with hyperlinks to their types.
#### Inputs
| Name | Type | Description |
| :------ | :------------- | :------------------------------------------------------- |
| self | NumpyDocstring | The `NumpyDocstring` instance. |
| section | str | The content of the "Returns" section from the docstring. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :---------------------------------------------------------------------- |
| | list\[str] | Returns a list of strings representing the formatted "Returns" section. |
#### Internal Logic
1. Calls `process_return` to reformat the lines of the "Returns" section.
2. Formats the processed lines using `_format_block` with ":returns: " as the prefix.
3. Appends an empty line if the formatted lines are not empty.
4. Returns the formatted lines.
### `setup`
#### Description
This function is the entry point for the Sphinx extension. It modifies the behavior of the `NumpyDocstring` class to use the custom `scanpy_parse_returns_section` method.
#### Inputs
| Name | Type | Description |
| :--- | :----- | :----------------------------- |
| app | Sphinx | The Sphinx application object. |
#### Internal Logic
Sets the `_parse_returns_section` attribute of the `NumpyDocstring` class to the custom `scanpy_parse_returns_section` function.
## Dependencies
| Dependency | Purpose |
| :------------------ | :--------------------------------------------------- |
| sphinx | Used for interacting with Sphinx and its components. |
| sphinx.ext.napoleon | Used for parsing NumPy-style docstrings. |
| re | Used for regular expression matching. |
## TODOs
```python theme={null}
# code from https://github.com/theislab/scanpy/blob/master/docs/extensions/typed_returns.py
# with some minor adjustment
```
# typed_returns.py
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/extensions/typed_returns.py
## High-level description
This code provides a Sphinx extension that customizes the parsing of the "Returns" section in NumPy-style docstrings. It specifically formats the return values to include hyperlinks to their types using the `:class:` role.
## References
This code references the `NumpyDocstring` class from the `sphinx.ext.napoleon` module.
## Symbols
### `process_return`
#### Description
This function iterates through a list of lines (presumably from a docstring's "Returns" section) and reformats lines that match a specific pattern.
#### Inputs
| Name | Type | Description |
| :---- | :--------- | :----------------------------------------------------------------------- |
| lines | list\[str] | A list of strings representing lines in a docstring's "Returns" section. |
#### Outputs
| Name | Type | Description |
| :--- | :-------------- | :------------------------------------------------------------------ |
| | generator\[str] | Yields reformatted lines or the original line if no match is found. |
#### Internal Logic
1. Iterates through each line in the input `lines`.
2. Attempts to match the line with the regular expression `r"(?P<param>\w+)\s+:\s+(?P<type>[\w.]+)"`. This pattern captures a "parameter" and its "type" separated by a colon.
3. If a match is found:
* It reformats the line to `f'-{m["param"]} (:class:`\~`)'`, which creates a hyperlink for the type using the `:class:` role.
4. If no match is found, it yields the original line.
### `scanpy_parse_returns_section`
#### Description
This function overrides the `_parse_returns_section` method of the `NumpyDocstring` class. It processes the "Returns" section of a docstring to format return values with hyperlinks to their types.
#### Inputs
| Name | Type | Description |
| :------ | :------------- | :------------------------------------------------------- |
| self | NumpyDocstring | The `NumpyDocstring` instance. |
| section | str | The content of the "Returns" section from the docstring. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :---------------------------------------------------------------------- |
| | list\[str] | Returns a list of strings representing the formatted "Returns" section. |
#### Internal Logic
1. Calls `process_return` to reformat the lines of the "Returns" section.
2. Formats the processed lines using `_format_block` with ":returns: " as the prefix.
3. Appends an empty line if the formatted lines are not empty.
4. Returns the formatted lines.
### `setup`
#### Description
This function is the entry point for the Sphinx extension. It modifies the behavior of the `NumpyDocstring` class to use the custom `scanpy_parse_returns_section` method.
#### Inputs
| Name | Type | Description |
| :--- | :----- | :----------------------------- |
| app | Sphinx | The Sphinx application object. |
#### Internal Logic
Sets the `_parse_returns_section` attribute of the `NumpyDocstring` class to the custom `scanpy_parse_returns_section` function.
## Dependencies
| Dependency | Purpose |
| :------------------ | :--------------------------------------------------- |
| sphinx | Used for interacting with Sphinx and its components. |
| sphinx.ext.napoleon | Used for parsing NumPy-style docstrings. |
| re | Used for regular expression matching. |
## TODOs
```python theme={null}
# code from https://github.com/theislab/scanpy/blob/master/docs/extensions/typed_returns.py
# with some minor adjustment
```
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/docs/overview
## High-level description
This directory contains custom documentation-related files for the Streamlink project. It includes CSS files for styling the documentation, custom Sphinx templates, configuration files, and extensions to enhance the documentation generation process.
## What does it do?
The contents of this directory work together to customize and improve the appearance and functionality of Streamlink's documentation. Here's a breakdown of the main components:
1. CSS files: These customize the appearance of the documentation, overriding default styles and enhancing readability.
2. Custom templates: These modify the layout of the documentation pages, including font specifications.
3. Configuration file: This sets up various options for the Sphinx documentation generator.
4. Custom extension: This enhances the parsing of docstrings, particularly for the "Returns" section.
Together, these components create a cohesive, visually appealing, and informative documentation experience for Streamlink users and contributors.
## Key Files
1. `_static/override.css`: This is the main CSS file that overrides the default styles of the Furo Sphinx theme. It customizes the appearance of code blocks, citations, headings, buttons, and other elements specific to Streamlink's documentation.
2. `_static/sphinx_gallery.css`: This CSS file focuses on styling Sphinx Gallery thumbnails, improving their visual presentation within the documentation.
3. `_templates/layout.html`: This custom template inherits from the PyData Sphinx theme and overrides the fonts block to specify custom font families for the body and header elements.
4. `conf.py`: This is the main configuration file for Sphinx. It sets various options for the documentation generation process, including theme settings, extensions, and autodoc options.
5. `extensions/typed_returns.py`: This custom Sphinx extension enhances the parsing of the "Returns" section in NumPy-style docstrings, formatting return values with hyperlinks to their types.
## Dependencies
The documentation system relies on several key dependencies:
1. Sphinx: The main documentation generator.
2. Furo Sphinx theme: The base theme that `override.css` customizes.
3. PyData Sphinx theme: Used as a base for the custom layout template.
4. Sphinx Gallery: Provides the HTML structure and classes targeted by `sphinx_gallery.css`.
5. Napoleon extension: Used for parsing NumPy-style docstrings.
## Configuration
The `conf.py` file contains various configuration options for Sphinx, including:
1. Project information (name, version, etc.)
2. General configuration (extensions, templates path, etc.)
3. HTML output options (theme, static files, etc.)
4. Theme-specific options
5. Autodoc options
The CSS files (`override.css` and `sphinx_gallery.css`) contain numerous style rules that target specific elements of the documentation. These rules customize the appearance of various components, such as code blocks, citations, buttons, and Sphinx Gallery thumbnails.
The custom extension in `typed_returns.py` modifies the behavior of the `NumpyDocstring` class to enhance the formatting of the "Returns" section in docstrings. It adds hyperlinks to the types of return values using the `:class:` role.
Overall, this directory provides a comprehensive set of tools and configurations to create a polished, user-friendly documentation experience for the Streamlink project.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/github_workflows/overview
## High-level description
This GitHub Actions workflow, named "cassiopeia", is designed to automate the testing and linting process for a Python project. It runs on push events to the master branch and pull requests targeting the master branch, using multiple Python versions to ensure compatibility.
## Code Structure
The workflow is defined as a single job named "build" with multiple steps that set up the environment, install dependencies, run tests, and report code coverage.
## Symbols
### `name`
#### Description
Defines the name of the GitHub Actions workflow as "cassiopeia".
### `on`
#### Description
Specifies the events that trigger the workflow: pushes to the master branch and pull requests targeting the master branch.
### `jobs.build`
#### Description
Defines the main job of the workflow, named "build".
#### Internal Logic
1. Runs on the latest Ubuntu environment
2. Sets a timeout of 20 minutes
3. Uses a matrix strategy to test with Python versions 3.8 and 3.9
### `jobs.build.steps`
#### Description
Defines the sequence of steps to be executed in the build job.
#### Internal Logic
1. Checks out the repository
2. Sets up Python environment
3. Caches pip dependencies
4. Installs project dependencies
5. Runs tests with pytest and generates coverage report
6. Uploads coverage data to Codecov
7. Lists installed packages
## Dependencies
| Dependency | Purpose |
| :---------- | :------------------------------------------------------------------------ |
| pytest-cov | For running tests with coverage reporting |
| codecov | For uploading coverage data |
| .\[spatial] | Project-specific dependency (likely the main package with spatial extras) |
## Configuration
| Option | Type | Default | Description |
| :-------------- | :------ | :---------- | :------------------------------ |
| python-version | list | \[3.8, 3.9] | Python versions to test against |
| timeout-minutes | integer | 20 | Maximum duration for the job |
## Error Handling
The workflow does not implement specific error handling beyond the default behavior of GitHub Actions, which will mark the workflow as failed if any step exits with a non-zero status code.
## Logging
The workflow uses default GitHub Actions logging. The `-vv` flag in the pytest command enables verbose output for test results.
## TODOs
There are no explicit TODOs in the workflow file.
# test.yml
Source: https://demo.agenticlabs.com/detailed-code-docs/github_workflows/test.yml
## High-level description
This GitHub Actions workflow, named "cassiopeia", is designed to automate the testing and linting process for a Python project. It runs on push events to the master branch and pull requests targeting the master branch, using multiple Python versions to ensure compatibility.
## Code Structure
The workflow is defined as a single job named "build" with multiple steps that set up the environment, install dependencies, run tests, and report code coverage.
## Symbols
### `name`
#### Description
Defines the name of the GitHub Actions workflow as "cassiopeia".
### `on`
#### Description
Specifies the events that trigger the workflow: pushes to the master branch and pull requests targeting the master branch.
### `jobs.build`
#### Description
Defines the main job of the workflow, named "build".
#### Internal Logic
1. Runs on the latest Ubuntu environment
2. Sets a timeout of 20 minutes
3. Uses a matrix strategy to test with Python versions 3.8 and 3.9
### `jobs.build.steps`
#### Description
Defines the sequence of steps to be executed in the build job.
#### Internal Logic
1. Checks out the repository
2. Sets up Python environment
3. Caches pip dependencies
4. Installs project dependencies
5. Runs tests with pytest and generates coverage report
6. Uploads coverage data to Codecov
7. Lists installed packages
## Dependencies
| Dependency | Purpose |
| :---------- | :------------------------------------------------------------------------ |
| pytest-cov | For running tests with coverage reporting |
| codecov | For uploading coverage data |
| .\[spatial] | Project-specific dependency (likely the main package with spatial extras) |
## Configuration
| Option | Type | Default | Description |
| :-------------- | :------ | :---------- | :------------------------------ |
| python-version | list | \[3.8, 3.9] | Python versions to test against |
| timeout-minutes | integer | 20 | Maximum duration for the job |
## Error Handling
The workflow does not implement specific error handling beyond the default behavior of GitHub Actions, which will mark the workflow as failed if any step exits with a non-zero status code.
## Logging
The workflow uses default GitHub Actions logging. The `-vv` flag in the pytest command enables verbose output for test results.
## TODOs
There are no explicit TODOs in the workflow file.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/project_root
## High-level description
This directory contains the core components of the Cassiopeia project, a comprehensive framework for single-cell lineage tracing and phylogenetic analysis. The project includes tools for data preprocessing, tree reconstruction, simulation, visualization, and various analytical utilities.
## What does it do?
Cassiopeia provides an end-to-end pipeline for analyzing single-cell lineage tracing experiments:
1. Preprocessing: Converts raw sequencing data into formats suitable for phylogenetic analysis, including filtering, error correction, and allele calling.
2. Tree Reconstruction: Implements various algorithms (greedy, distance-based, spectral, and integer linear programming) to reconstruct phylogenetic trees from mutation data.
3. Simulation: Generates synthetic phylogenetic trees and associated data for testing and validating analysis methods.
4. Visualization: Offers both local and cloud-based options for visualizing phylogenetic trees and associated data.
5. Analysis Tools: Provides utilities for parameter estimation, fitness calculation, evolutionary coupling analysis, and various tree metrics.
6. Tree Comparison: Implements methods for comparing phylogenetic trees, such as Robinson-Foulds distance and triplets correct accuracy.
## Entry points
The main entry points for the Cassiopeia package are:
1. `cassiopeia/preprocess`: For preprocessing raw sequencing data.
2. `cassiopeia/solver`: For reconstructing phylogenetic trees from processed data.
3. `cassiopeia/simulator`: For generating synthetic phylogenetic data.
4. `cassiopeia/plotting`: For visualizing phylogenetic trees and associated data.
5. `cassiopeia/tools`: For various analytical tools and utilities.
6. `cassiopeia/critique`: For comparing and analyzing phylogenetic trees.
The `cassiopeia/__init__.py` file serves as the main interface, importing and exposing key functionalities from these submodules.
## Key Files
1. `cassiopeia/data/CassiopeiaTree.py`: Defines the core data structure for representing phylogenetic trees with associated mutation data.
2. `cassiopeia/solver/CassiopeiaSolver.py`: Provides the base class for tree reconstruction algorithms.
3. `cassiopeia/simulator/TreeSimulator.py`: Offers the base class for tree simulation models.
4. `cassiopeia/plotting/local.py` and `cassiopeia/plotting/itol_utilities.py`: Implement local and cloud-based tree visualization.
5. `cassiopeia/tools/parameter_estimators.py`: Contains functions for estimating key parameters from tree data.
6. `cassiopeia/critique/compare.py`: Implements tree comparison algorithms.
7. `build.py`: Handles the building and compilation of Cython extensions for performance-critical components.
8. `README.md`: Provides an overview of the project, installation instructions, and links to documentation and tutorials.
## Dependencies
Cassiopeia relies on several external libraries:
1. numpy and pandas: For data manipulation and numerical computations.
2. networkx: For graph operations and tree manipulations.
3. scipy: For various scientific computing tasks.
4. ete3: For phylogenetic tree manipulation and visualization.
5. matplotlib and plotly: For local plotting.
6. pysam: For handling sequencing data formats.
7. cvxpy and gurobipy: For optimization problems in some solvers.
8. Cython: For compiling performance-critical components.
## Configuration
Cassiopeia uses various configuration methods:
1. Preprocessing pipeline: Configured using an INI-format file specifying parameters for each preprocessing step.
2. Solvers: Customizable through parameters passed during initialization.
3. Simulators: Configurable via parameters set during object creation.
4. Visualization: Customizable through function parameters for both local and cloud-based plotting.
5. `pyproject.toml`: Defines project metadata, dependencies, and build settings.
6. `.readthedocs.yml`: Configures the documentation build process on Read the Docs.
7. `codecov.yml`: Sets coverage requirements for the project.
The project also includes a comprehensive test suite in the `test` directory, ensuring the reliability and accuracy of the framework across its various modules and functionalities.
In summary, Cassiopeia provides a robust and flexible framework for analyzing single-cell lineage tracing data, offering tools for every step from raw data processing to advanced phylogenetic analysis and visualization.
# pyproject.toml
Source: https://demo.agenticlabs.com/detailed-code-docs/pyproject.toml
## High-level description
This `pyproject.toml` file defines the configuration for the Cassiopeia project, a Python package for single-cell lineage reconstruction. It specifies project metadata, dependencies, build settings, and optional features for development, documentation, and spatial analysis.
## Code Structure
The file is structured into several sections using TOML format, including tool configurations for Poetry (a Python dependency management tool), project metadata, dependencies, build settings, and extras for different use cases.
## Symbols
### \[tool.poetry]
#### Description
This section defines the project metadata for the Cassiopeia package.
#### Internal Logic
* Specifies authors, classifiers, description, documentation URL, homepage, keywords, license, package name, readme file, repository URL, and version.
* Includes file paths for wheel format distribution.
* Defines the main package to be included.
### \[tool.poetry.dependencies]
#### Description
Lists all the dependencies required for the Cassiopeia package.
#### Internal Logic
* Specifies required Python packages and their version constraints.
* Some dependencies are marked as optional and are tied to specific features (e.g., development, documentation, spatial analysis).
### \[tool.poetry.build]
#### Description
Configures the build process for the package.
#### Internal Logic
* Disables automatic setup file generation.
* Specifies a custom build script (`build.py`).
### \[build-system]
#### Description
Defines the build system requirements for the package.
#### Internal Logic
* Specifies Poetry as the build backend.
* Lists required packages for building, including Cython and numpy.
### \[tool.poetry.scripts]
#### Description
Defines command-line entry points for the package.
#### Internal Logic
* Maps the `cassiopeia-preprocess` command to the main function in the `cassiopeia.preprocess.cassiopeia_preprocess` module.
### \[tool.poetry.extras]
#### Description
Defines optional feature sets that can be installed separately.
#### Internal Logic
* `dev`: Includes development tools like black, pytest, and pre-commit.
* `docs`: Includes documentation generation tools like Sphinx.
* `spatial`: Includes packages for spatial analysis features.
## Dependencies
The project has numerous dependencies, including:
* Core scientific libraries: numpy, scipy, pandas, scikit-learn
* Bioinformatics tools: Biopython, pysam
* Visualization libraries: matplotlib, plotly
* Machine learning: scikit-learn
* Documentation: Sphinx and related packages
* Development tools: black, pytest, pre-commit
## Configuration
The file itself serves as a configuration for the project, defining various aspects of the package structure, dependencies, and build process.
## API/Interface Reference
The file defines one command-line interface:
| Endpoint | Method | Description |
| :-------------------- | :------ | :---------------------------------------------------- |
| cassiopeia-preprocess | Command | Entry point for the Cassiopeia preprocessing pipeline |
This command likely corresponds to the `main()` function in the `cassiopeia/preprocess/cassiopeia_preprocess.py` file, which is referenced in the related code snippets.
# setup.py
Source: https://demo.agenticlabs.com/detailed-code-docs/setup.py
## High-level description
This file, `setup.py`, is a minimal Python setup script that serves as a shim for package detection on GitHub. Its primary purpose is to enable GitHub to recognize the package, while the actual build process is managed using Poetry.
## Symbols
### `setuptools.setup()`
#### Description
This function is called to provide basic package information to setuptools. In this case, it only specifies the package name "cassiopeia".
#### Inputs
| Name | Type | Description |
| :--- | :--- | :------------------------------------------- |
| name | str | The name of the package, set to "cassiopeia" |
#### Outputs
This function doesn't explicitly return a value, but it sets up the package configuration for setuptools.
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------ |
| setuptools | Used to provide basic package setup functionality |
## Configuration
| Option | Type | Default | Description |
| :----- | :--- | :----------- | :-------------------------------- |
| name | str | "cassiopeia" | Specifies the name of the package |
## Additional Notes
1. The script includes a shebang line (`#!/usr/bin/env python`) allowing it to be executed directly on Unix-like systems.
2. The actual build process is not handled by this script but is instead managed by Poetry, as indicated in the comment.
3. The `if __name__ == "__main__":` block ensures that the `setuptools.setup()` function is only called when the script is run directly, not when it's imported as a module.
This minimal setup script is likely used as a workaround to make the project detectable by GitHub's package detection mechanisms, while the actual package management and build process are handled separately using Poetry.
# compare_tree_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/critique_tests/compare_tree_test.py
## High-level description
This file contains unit tests for the `cassiopeia.critique.compare` module, specifically focusing on the comparison of phylogenetic trees. It uses various tree structures, including balanced trees, multifurcating trees, and rake-shaped trees, to test the accuracy of the comparison metrics.
## Code Structure
The `TestTreeComparisons` class contains multiple test methods, each designed to test a specific aspect of the tree comparison functionality. The `setUp` method initializes various tree structures used throughout the tests. The test methods then utilize these trees to calculate and validate different comparison metrics.
## Symbols
### `TestTreeComparisons`
#### Description
A class containing unit tests for tree comparison functions in the `cassiopeia.critique.compare` module.
#### Inputs
None
#### Outputs
None (test results are outputted to the console)
#### Internal Logic
This class utilizes the `unittest` framework in Python. Each test method within this class is executed independently and checks for specific outcomes from the tree comparison functions. Assertions are used to validate the expected results.
##### `setUp`
###### Description
Initializes various `CassiopeiaTree` objects with different tree structures for testing purposes.
###### Inputs
None
###### Outputs
None (sets up instance variables)
##### `test_out_group`
###### Description
Tests the `get_outgroup` function, which identifies the outlier node in a triplet.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `get_outgroup`)
##### `test_same_tree_gives_perfect_triplets_correct`
###### Description
Tests the `triplets_correct` function when comparing a tree to itself, expecting perfect agreement.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_different_trees`
###### Description
Tests the `triplets_correct` function with different trees, expecting variations in agreement.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_multifurcating_same_tree`
###### Description
Tests the `triplets_correct` function with a multifurcating tree compared to itself, considering unresolvable triplets.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_multifurcating_different_trees`
###### Description
Tests the `triplets_correct` function with different multifurcating trees, considering unresolvable triplets.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_rake_tree`
###### Description
Tests the `triplets_correct` function with a rake-shaped tree, which has all nodes at the same depth.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_robinson_foulds_same_tree_bifurcating`
###### Description
Tests the `robinson_foulds` function, which calculates the Robinson-Foulds distance between trees.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `robinson_foulds`)
## References
This code references the following symbols from other files:
* `cassiopeia.critique.critique_utilities.get_outgroup`
* `cassiopeia.critique.triplets_correct`
* `cassiopeia.critique.robinson_foulds`
* `cassiopeia.data.CassiopeiaTree`
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------------------------------------------ |
| unittest | Provides the framework for writing and running unit tests. |
| networkx | Used for creating and manipulating graph structures, representing phylogenetic trees. |
| numpy | Used for numerical operations and array manipulation. |
| pandas | Used for data manipulation and analysis, particularly with DataFrames. |
| cassiopeia | The core library for single-cell lineage tracing analysis. |
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/critique_tests/overview
## High-level description
This file contains unit tests for the `cassiopeia.critique.compare` module, specifically focusing on the comparison of phylogenetic trees. It uses various tree structures, including balanced trees, multifurcating trees, and rake-shaped trees, to test the accuracy of the comparison metrics.
## Code Structure
The `TestTreeComparisons` class contains multiple test methods, each designed to test a specific aspect of the tree comparison functionality. The `setUp` method initializes various tree structures used throughout the tests. The test methods then utilize these trees to calculate and validate different comparison metrics.
## Symbols
### `TestTreeComparisons`
#### Description
A class containing unit tests for tree comparison functions in the `cassiopeia.critique.compare` module.
#### Inputs
None
#### Outputs
None (test results are outputted to the console)
#### Internal Logic
This class utilizes the `unittest` framework in Python. Each test method within this class is executed independently and checks for specific outcomes from the tree comparison functions. Assertions are used to validate the expected results.
##### `setUp`
###### Description
Initializes various `CassiopeiaTree` objects with different tree structures for testing purposes.
###### Inputs
None
###### Outputs
None (sets up instance variables)
##### `test_out_group`
###### Description
Tests the `get_outgroup` function, which identifies the outlier node in a triplet.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `get_outgroup`)
##### `test_same_tree_gives_perfect_triplets_correct`
###### Description
Tests the `triplets_correct` function when comparing a tree to itself, expecting perfect agreement.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_different_trees`
###### Description
Tests the `triplets_correct` function with different trees, expecting variations in agreement.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_multifurcating_same_tree`
###### Description
Tests the `triplets_correct` function with a multifurcating tree compared to itself, considering unresolvable triplets.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_triplets_correct_multifurcating_different_trees`
###### Description
Tests the `triplets_correct` function with different multifurcating trees, considering unresolvable triplets.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_rake_tree`
###### Description
Tests the `triplets_correct` function with a rake-shaped tree, which has all nodes at the same depth.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `triplets_correct`)
##### `test_robinson_foulds_same_tree_bifurcating`
###### Description
Tests the `robinson_foulds` function, which calculates the Robinson-Foulds distance between trees.
###### Inputs
None
###### Outputs
None (assertions are used to validate the output of `robinson_foulds`)
## References
This code references the following symbols from other files:
* `cassiopeia.critique.critique_utilities.get_outgroup`
* `cassiopeia.critique.triplets_correct`
* `cassiopeia.critique.robinson_foulds`
* `cassiopeia.data.CassiopeiaTree`
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------------------------------------------ |
| unittest | Provides the framework for writing and running unit tests. |
| networkx | Used for creating and manipulating graph structures, representing phylogenetic trees. |
| numpy | Used for numerical operations and array manipulation. |
| pandas | Used for data manipulation and analysis, particularly with DataFrames. |
| cassiopeia | The core library for single-cell lineage tracing analysis. |
# cassiopeia_tree_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/data_tests/cassiopeia_tree_test.py
Here's a high-level description and documentation of the provided code:
## High-level description
This file contains unit tests for the utilities stored in `cassiopeia/data/utilities.py`. It tests various functions related to bootstrapping character matrices and allele tables, converting trees to Newick format, computing phylogenetic weight matrices, and calculating inter-cluster distances.
## Code Structure
The code defines a `TestDataUtilities` class that inherits from `unittest.TestCase`. This class contains multiple test methods, each testing a specific functionality of the data utilities module.
## Symbols
### TestDataUtilities
#### Description
A test class containing various unit tests for the data utilities module.
#### Internal Logic
The class sets up test data in the `setUp` method and defines multiple test methods to verify the functionality of different utility functions.
### test\_bootstrap\_character\_matrices\_no\_priors
#### Description
Tests the `sample_bootstrap_character_matrices` function without providing priors.
### test\_bootstrap\_character\_matrices\_with\_priors
#### Description
Tests the `sample_bootstrap_character_matrices` function with provided priors.
### test\_bootstrap\_allele\_tables
#### Description
Tests the `sample_bootstrap_allele_tables` function for standard allele tables.
### test\_bootstrap\_allele\_tables\_non\_cassiopeia\_allele\_table
#### Description
Tests the `sample_bootstrap_allele_tables` function for non-standard allele tables.
### test\_bootstrap\_allele\_tables\_priors
#### Description
Tests the `sample_bootstrap_allele_tables` function with provided indel priors.
### test\_to\_newick\_no\_branch\_lengths
#### Description
Tests the `to_newick` function without including branch lengths.
### test\_to\_newick\_branch\_lengths
#### Description
Tests the `to_newick` function including branch lengths.
### test\_lca\_characters
#### Description
Tests the `get_lca_characters` function for finding the lowest common ancestor characters.
### test\_lca\_characters\_ambiguous
#### Description
Tests the `get_lca_characters` function with ambiguous character states.
### test\_lca\_characters\_ambiguous2
#### Description
Another test for the `get_lca_characters` function with different ambiguous character states.
### test\_lca\_characters\_ambiguous\_and\_missing
#### Description
Tests the `get_lca_characters` function with both ambiguous and missing character states.
### test\_resolve\_most\_abundant
#### Description
Tests the `resolve_most_abundant` function for resolving the most common state in an ambiguous character.
### test\_simple\_phylogenetic\_weights\_matrix
#### Description
Tests the `compute_phylogenetic_weight_matrix` function for a simple tree.
### test\_simple\_phylogenetic\_weights\_matrix\_inverse
#### Description
Tests the `compute_phylogenetic_weight_matrix` function with inverse weights.
### test\_phylogenetic\_weights\_matrix\_inverse\_fn
#### Description
Tests the `compute_phylogenetic_weight_matrix` function with a custom inverse function.
### test\_net\_relatedness\_index
#### Description
Tests the `net_relatedness_index` function for calculating the Net Relatedness Index.
### test\_inter\_cluster\_distance\_basic
#### Description
Tests the `compute_inter_cluster_distances` function for basic inter-cluster distance calculation.
### test\_inter\_cluster\_distance\_custom\_input
#### Description
Tests the `compute_inter_cluster_distances` function with custom input data.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia.data.CassiopeiaTree
* cassiopeia.data.utilities
* cassiopeia.mixins.errors.CassiopeiaError
* cassiopeia.preprocess.utilities
This test suite ensures that the data utility functions in the Cassiopeia package are working correctly and handling various edge cases appropriately.
# data_utilities_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/data_tests/data_utilities_test.py
Here's a high-level description and documentation of the provided code:
## High-level description
This file contains unit tests for the data utilities stored in `cassiopeia/data/utilities.py`. It tests various functions related to character matrix operations, bootstrap sampling, dissimilarity map computations, and other utility functions used in the Cassiopeia lineage tracing framework.
## Code Structure
The code is structured as a single test class `TestDataUtilities` that inherits from `unittest.TestCase`. This class contains multiple test methods, each testing a specific functionality of the data utilities module.
## Symbols
### TestDataUtilities
#### Description
A test class containing various unit tests for the data utilities module.
#### Internal Logic
1. Sets up test data in the `setUp` method, including character matrices, priors, and allele tables.
2. Defines multiple test methods, each focusing on a specific utility function or feature.
### test\_bootstrap\_character\_matrices\_no\_priors
#### Description
Tests the bootstrap sampling of character matrices without priors.
### test\_bootstrap\_character\_matrices\_with\_priors
#### Description
Tests the bootstrap sampling of character matrices with priors.
### test\_bootstrap\_allele\_tables
#### Description
Tests the bootstrap sampling of allele tables.
### test\_bootstrap\_allele\_tables\_non\_cassiopeia\_allele\_table
#### Description
Tests the bootstrap sampling of allele tables with non-standard column names.
### test\_bootstrap\_allele\_tables\_priors
#### Description
Tests the bootstrap sampling of allele tables with priors.
### test\_to\_newick\_no\_branch\_lengths
#### Description
Tests the conversion of a tree to Newick format without branch lengths.
### test\_to\_newick\_branch\_lengths
#### Description
Tests the conversion of a tree to Newick format with branch lengths.
### test\_lca\_characters
#### Description
Tests the computation of least common ancestor (LCA) characters.
### test\_lca\_characters\_ambiguous
#### Description
Tests the computation of LCA characters with ambiguous states.
### test\_lca\_characters\_ambiguous2
#### Description
Tests another case of LCA character computation with ambiguous states.
### test\_lca\_characters\_ambiguous\_and\_missing
#### Description
Tests the computation of LCA characters with both ambiguous and missing states.
### test\_resolve\_most\_abundant
#### Description
Tests the resolution of ambiguous states by selecting the most abundant state.
### test\_simple\_phylogenetic\_weights\_matrix
#### Description
Tests the computation of a simple phylogenetic weight matrix.
### test\_simple\_phylogenetic\_weights\_matrix\_inverse
#### Description
Tests the computation of an inverse simple phylogenetic weight matrix.
### test\_phylogenetic\_weights\_matrix\_inverse\_fn
#### Description
Tests the computation of a phylogenetic weight matrix with a custom inverse function.
### test\_net\_relatedness\_index
#### Description
Tests the computation of the net relatedness index.
### test\_inter\_cluster\_distance\_basic
#### Description
Tests the computation of inter-cluster distances.
### test\_inter\_cluster\_distance\_custom\_input
#### Description
Tests the computation of inter-cluster distances with custom input.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia.data
* cassiopeia.mixins.errors
* cassiopeia.preprocess.utilities
The test suite can be run by executing this file directly using Python.
# layers_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/data_tests/layers_test.py
Here's a detailed documentation of the provided code:
## High-level description
This file contains unit tests for the Layers class in the data module of the Cassiopeia project. It tests various functionalities related to character matrices, tree topologies, and layer operations in the context of phylogenetic analysis.
## Code Structure
The code defines a TestLayers class that inherits from unittest.TestCase. It contains several test methods that check different aspects of the Layers functionality, including basic character matrix operations, adding layers, reconstructing trees with layers, and handling different types of character matrices.
## Symbols
### TestLayers
#### Description
A test class that contains various unit tests for the Layers functionality in the Cassiopeia data module.
#### Internal Logic
The class sets up a basic character matrix and tree topology in the setUp method, which is used across multiple test cases. It then defines several test methods to check different aspects of the Layers functionality.
### setUp
#### Description
Initializes the test environment with a basic character matrix and tree topology.
#### Internal Logic
* Creates a character matrix using pandas DataFrame
* Defines a tree topology using networkx DiGraph
* Initializes a CassiopeiaTree object with the character matrix and topology
### test\_basic\_character\_matrix
#### Description
Tests if the basic character matrix is correctly initialized in the CassiopeiaTree object.
#### Internal Logic
* Retrieves the character matrix from the CassiopeiaTree object
* Compares it with the expected character matrix using pandas testing utilities
### test\_add\_layer
#### Description
Tests the functionality of adding a new layer to the CassiopeiaTree object.
#### Internal Logic
* Creates a modified character matrix
* Adds it as a new layer to the CassiopeiaTree object
* Verifies that the new layer is correctly added and can be accessed
* Checks that modifying the new layer doesn't affect the base character matrix
### test\_reconstruct\_tree\_with\_layers
#### Description
Tests the tree reconstruction process using different layers.
#### Internal Logic
* Reconstructs the tree using the base character matrix
* Verifies the reconstructed tree structure
* Adds a modified layer and reconstructs the tree again
* Compares the new tree structure with the expected topology
### test\_add\_layer\_with\_incorrect\_number\_of\_cells
#### Description
Tests error handling when adding a layer with an incorrect number of cells.
#### Internal Logic
* Attempts to add a layer with an extra cell
* Verifies that a ValueError is raised
### test\_add\_layer\_with\_different\_number\_of\_characters
#### Description
Tests adding a layer with a different number of characters than the base matrix.
#### Internal Logic
* Adds a layer with an extra character column
* Verifies that the new layer is correctly added and can be accessed
* Checks that the character states are updated correctly when using the new layer
## Dependencies
The code relies on several external libraries and modules:
* unittest for test framework
* ete3 for tree manipulation
* networkx for graph operations
* numpy and pandas for data manipulation
* cassiopeia for the main functionality being tested
## Error Handling
The code uses assertions and unittest methods to check for expected behavior and raise errors when unexpected results occur.
## Performance Considerations
The tests involve tree reconstruction and manipulation, which can be computationally expensive for large trees. However, as these are unit tests, they use small sample data and are not intended for performance testing.
This documentation provides a comprehensive overview of the test file, explaining its structure, main components, and the specific aspects of the Layers functionality being tested.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/data_tests/overview
## High-level description
This directory contains unit tests for the data module of the Cassiopeia project, which is likely a framework for phylogenetic analysis and lineage tracing. The tests cover various functionalities related to character matrices, tree topologies, bootstrapping, and utility functions used in phylogenetic analysis.
## What does it do?
These tests ensure that the core data handling and manipulation functions of Cassiopeia work correctly. They verify:
1. Character matrix operations: Creating, modifying, and analyzing matrices that represent genetic or cellular states.
2. Tree topology handling: Converting trees to Newick format, reconstructing trees from character matrices, and manipulating tree structures.
3. Bootstrapping: Sampling techniques for character matrices and allele tables to assess statistical confidence.
4. Phylogenetic calculations: Computing least common ancestor characters, phylogenetic weight matrices, and inter-cluster distances.
5. Layer operations: Adding and managing different layers of data in the phylogenetic analysis.
These tests help maintain the reliability and accuracy of the Cassiopeia framework, ensuring that researchers can trust the results of their lineage tracing and phylogenetic analyses.
## Key Files
1. cassiopeia\_tree\_test.py
* Tests the CassiopeiaTree class, which is likely the core data structure for representing phylogenetic trees in the framework.
* Covers operations like bootstrapping character matrices and allele tables, converting trees to Newick format, and calculating phylogenetic weights.
2. data\_utilities\_test.py
* Tests various utility functions in the data module, including character matrix operations, bootstrap sampling, and dissimilarity map computations.
* Ensures the correct functioning of helper functions that support the main phylogenetic analysis tasks.
3. layers\_test.py
* Tests the Layers class, which appears to handle multiple layers of character matrices or other data associated with phylogenetic trees.
* Verifies operations like adding new layers, reconstructing trees with different layers, and handling various types of character matrices.
## Dependencies
The test files rely on several external libraries and modules:
* unittest: The standard Python testing framework.
* networkx: Used for graph operations and tree topology manipulation.
* numpy and pandas: For data manipulation and analysis.
* ete3: A Python framework for phylogenetic tree manipulation.
* cassiopeia: The main package being tested, including its various modules like data, mixins, and preprocess.
## Configuration
The tests use sample data and configurations set up within each test class. For example:
```python theme={null}
def setUp(self):
self.character_matrix = pd.DataFrame({
"r1": ["0", "1", "0"],
"r2": ["1", "0", "1"]
}, index=["cell1", "cell2", "cell3"])
self.tree = nx.DiGraph()
self.tree.add_edges_from([("root", "cell1"), ("root", "cell2"), ("root", "cell3")])
self.ctree = CassiopeiaTree(character_matrix=self.character_matrix, tree=self.tree)
```
This setup creates a basic character matrix and tree topology used across multiple test cases in the `TestLayers` class.
The tests cover a wide range of scenarios and edge cases, ensuring robust functionality of the Cassiopeia framework. They include checks for error handling, such as raising appropriate exceptions when invalid inputs are provided:
```python theme={null}
def test_add_layer_with_incorrect_number_of_cells(self):
modified_cm = self.character_matrix.copy()
modified_cm.loc["cell4"] = ["0", "1"]
with self.assertRaises(ValueError):
self.ctree.add_layer(modified_cm, "test_layer")
```
Overall, these tests play a crucial role in maintaining the reliability and accuracy of the Cassiopeia framework for phylogenetic analysis and lineage tracing.
# mixin_utilities_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/mixin_tests/mixin_utilities_test.py
## High-level description
This file contains unit tests for the utility functions defined in `cassiopeia/mixins/utilities.py`. It tests the functionality of various utility functions related to handling ambiguous states in character matrices and finding duplicate groups.
## Symbols
### TestMixinUtilities
#### Description
This is a test class that inherits from `unittest.TestCase`. It contains several test methods to verify the functionality of utility functions from the `cassiopeia.mixins.utilities` module.
#### Internal Logic
The class defines three test methods:
1. `test_is_ambiguous_state`: Tests the `is_ambiguous_state` function.
2. `test_unravel_states`: Tests the `unravel_ambiguous_states` function.
3. `test_find_duplicated_character_states`: Tests the `find_duplicate_groups` function.
### test\_is\_ambiguous\_state
#### Description
This method tests the `is_ambiguous_state` function from the utilities module.
#### Internal Logic
* Asserts that `(1, 2)` is recognized as an ambiguous state.
* Asserts that `1` is not recognized as an ambiguous state.
### test\_unravel\_states
#### Description
This method tests the `unravel_ambiguous_states` function from the utilities module.
#### Internal Logic
* Tests the function with an array containing both ambiguous and non-ambiguous states.
* Tests the function with an array containing only non-ambiguous states.
* Asserts that the output matches the expected unraveled list in both cases.
### test\_find\_duplicated\_character\_states
#### Description
This method tests the `find_duplicate_groups` function from the utilities module.
#### Internal Logic
* Creates a sample character matrix using pandas DataFrame.
* Calls the `find_duplicate_groups` function on the character matrix.
* Checks if the returned duplicate mappings contain the expected entries.
* Verifies that the keys and values in the duplicate mappings match the expected groups.
## Dependencies
| Dependency | Purpose |
| :-------------------------- | :--------------------------------------------------- |
| unittest | Provides the testing framework |
| pandas | Used to create and manipulate the character matrix |
| cassiopeia.mixins.utilities | Module containing the utility functions being tested |
## Error Handling
The test methods use assertions to verify the correctness of the utility functions. If any assertion fails, an `AssertionError` will be raised, indicating a test failure.
## Notes
* The test file uses the `unittest` framework, which is a standard Python testing library.
* The tests cover various scenarios, including ambiguous states and duplicate character states in matrices.
* The `test_find_duplicated_character_states` method uses a complex character matrix with ambiguous states to thoroughly test the `find_duplicate_groups` function.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/mixin_tests/overview
## High-level description
This file contains unit tests for the utility functions defined in `cassiopeia/mixins/utilities.py`. It tests the functionality of various utility functions related to handling ambiguous states in character matrices and finding duplicate groups.
## Symbols
### TestMixinUtilities
#### Description
This is a test class that inherits from `unittest.TestCase`. It contains several test methods to verify the functionality of utility functions from the `cassiopeia.mixins.utilities` module.
#### Internal Logic
The class defines three test methods:
1. `test_is_ambiguous_state`: Tests the `is_ambiguous_state` function.
2. `test_unravel_states`: Tests the `unravel_ambiguous_states` function.
3. `test_find_duplicated_character_states`: Tests the `find_duplicate_groups` function.
### test\_is\_ambiguous\_state
#### Description
This method tests the `is_ambiguous_state` function from the utilities module.
#### Internal Logic
* Asserts that `(1, 2)` is recognized as an ambiguous state.
* Asserts that `1` is not recognized as an ambiguous state.
### test\_unravel\_states
#### Description
This method tests the `unravel_ambiguous_states` function from the utilities module.
#### Internal Logic
* Tests the function with an array containing both ambiguous and non-ambiguous states.
* Tests the function with an array containing only non-ambiguous states.
* Asserts that the output matches the expected unraveled list in both cases.
### test\_find\_duplicated\_character\_states
#### Description
This method tests the `find_duplicate_groups` function from the utilities module.
#### Internal Logic
* Creates a sample character matrix using pandas DataFrame.
* Calls the `find_duplicate_groups` function on the character matrix.
* Checks if the returned duplicate mappings contain the expected entries.
* Verifies that the keys and values in the duplicate mappings match the expected groups.
## Dependencies
| Dependency | Purpose |
| :-------------------------- | :--------------------------------------------------- |
| unittest | Provides the testing framework |
| pandas | Used to create and manipulate the character matrix |
| cassiopeia.mixins.utilities | Module containing the utility functions being tested |
## Error Handling
The test methods use assertions to verify the correctness of the utility functions. If any assertion fails, an `AssertionError` will be raised, indicating a test failure.
## Notes
* The test file uses the `unittest` framework, which is a standard Python testing library.
* The tests cover various scenarios, including ambiguous states and duplicate character states in matrices.
* The `test_find_duplicated_character_states` method uses a complex character matrix with ambiguous states to thoroughly test the `find_duplicate_groups` function.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/overview
## High-level description
This directory contains a comprehensive suite of unit tests for the Cassiopeia project, which is a framework for phylogenetic analysis and lineage tracing. The tests cover various modules and functionalities of the project, including data handling, preprocessing, simulation, solving algorithms, plotting, and utility functions.
## What does it do?
The test suite verifies the correctness and reliability of Cassiopeia's core functionalities:
1. Data handling: Tests for character matrices, tree topologies, and data structures used in phylogenetic analysis.
2. Preprocessing: Validates sequence alignment, allele calling, lineage group assignment, and UMI collapsing.
3. Simulation: Ensures correct functionality of various simulators for tree generation and spatial data.
4. Solving algorithms: Tests different phylogenetic tree reconstruction methods, including Neighbor Joining, ILP, Greedy, and Spectral solvers.
5. Plotting: Verifies 2D and 3D plotting capabilities, including integration with iTOL for online visualization.
6. Utility functions: Tests helper functions for tasks like dissimilarity calculations and parameter estimation.
These tests help maintain the integrity of the Cassiopeia framework, ensuring that researchers can rely on its results for lineage tracing and phylogenetic analyses.
## Entry points
The main entry points for developers are the individual test directories and files:
1. critique\_tests: Tests for tree comparison functions.
2. data\_tests: Tests for data handling and manipulation.
3. mixin\_tests: Tests for utility functions in mixins.
4. plotting\_tests: Tests for various plotting functionalities.
5. preprocess\_tests: Tests for the preprocessing pipeline.
6. simulator\_tests: Tests for tree and data simulation.
7. solver\_tests: Tests for phylogenetic tree reconstruction algorithms.
8. tools\_tests: Tests for various analytical tools and algorithms.
Each directory contains multiple test files focusing on specific aspects of the respective module.
## Key Files
While all test files are important, some key files that cover core functionality include:
1. data\_tests/cassiopeia\_tree\_test.py: Tests the CassiopeiaTree class, a core data structure.
2. preprocess\_tests/align\_sequence\_test.py: Tests sequence alignment functionality.
3. solver\_tests/neighborjoining\_solver\_test.py: Tests the widely-used Neighbor Joining algorithm.
4. plotting\_tests/local\_test.py: Tests local 2D plotting functionality.
5. tools\_tests/tree\_metrics\_test.py: Tests various tree-based metrics and calculations.
## Dependencies
The test suite relies on several external libraries and frameworks:
1. unittest: The standard Python testing framework.
2. networkx: For graph and tree operations.
3. numpy and pandas: For data manipulation and analysis.
4. scipy: For scientific computing tasks.
5. matplotlib and plotly: For plotting backends.
6. ete3: For phylogenetic tree manipulation.
7. pysam: For reading and manipulating SAM/BAM files.
8. gurobipy (optional): For ILP solving in some tests.
## Configuration
Most tests use in-memory configurations or sample data created within the test methods. Some specific configurations include:
1. iTOL plotting tests may require a configuration file (\~/.itolconfig) for credentials.
2. Some tests use random seeds for reproducibility.
3. CCPhylo tests check for a config.ini file and a ccphylo\_path setting.
4. ILP solver tests may be skipped if Gurobi is not installed.
Overall, this comprehensive test suite ensures the reliability and accuracy of the Cassiopeia framework across its various modules and functionalities.
# itol_plotting_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/plotting_tests/itol_plotting_test.py
## High-level description
This file contains unit tests for the iTOL plotting utilities in the `cassiopeia.plotting.itol_utilities` module. It tests various functions related to generating files for iTOL visualization, including gradient files, colorbar files, and allele heatmaps. The tests also cover the integrated pipeline for uploading and exporting trees to iTOL.
## Code Structure
The code is structured as a single test class `TestITOLPlotting` that inherits from `unittest.TestCase`. It contains several test methods, each testing a specific functionality of the iTOL plotting utilities. The class uses a `setUp` method to initialize common test data and a `tearDown` method to clean up temporary files.
## Symbols
### TestITOLPlotting
#### Description
A test class that contains unit tests for iTOL plotting utilities.
#### Internal Logic
1. Sets up test data in `setUp` method, including a temporary directory, random state, indel priors, allele table, and a test tree.
2. Implements various test methods to check different functionalities.
3. Cleans up temporary files in `tearDown` method.
### test\_generate\_gradient\_file
#### Description
Tests the creation of a gradient file for iTOL visualization.
#### Internal Logic
1. Calls `itol_utilities.create_gradient_from_df` with test data.
2. Verifies that the file is created and its content matches the expected output.
### test\_generate\_colorbar\_file
#### Description
Tests the creation of a colorbar file for iTOL visualization without a legend.
#### Internal Logic
1. Defines a color map for test data.
2. Calls `itol_utilities.create_colorbar` with test data and color map.
3. Verifies that the file is created and its content matches the expected output.
### test\_generate\_colorbar\_file\_with\_legend
#### Description
Tests the creation of a colorbar file for iTOL visualization with a legend.
#### Internal Logic
Similar to `test_generate_colorbar_file`, but sets `create_legend=True` and verifies the legend content in the output file.
### test\_create\_allele\_heatmap\_specified\_colors
#### Description
Tests the creation of an allele heatmap with specified colors.
#### Internal Logic
1. Defines indel colors.
2. Calls `itol_utilities.create_indel_heatmap` with test data and specified colors.
3. Verifies that the correct number of files are created and that they exist.
### test\_create\_allele\_heatmap\_specified\_priors
#### Description
Tests the creation of an allele heatmap with specified indel priors.
#### Internal Logic
Similar to `test_create_allele_heatmap_specified_colors`, but uses indel priors instead of specified colors.
### test\_create\_allele\_heatmap\_no\_priors\_or\_colormap
#### Description
Tests the creation of an allele heatmap without specifying priors or a color map.
#### Internal Logic
Calls `itol_utilities.create_indel_heatmap` with only the basic required parameters and verifies the output files.
### test\_integrated\_pipeline\_simple\_tree
#### Description
Tests the integrated pipeline for uploading and exporting a simple tree to iTOL.
#### Internal Logic
1. Calls `cas.pl.upload_and_export_itol` with a simple tree.
2. Verifies that the exported file exists.
### test\_integrated\_pipeline\_tree\_with\_allele\_heatmap
#### Description
Tests the integrated pipeline for uploading and exporting a tree with an allele heatmap to iTOL.
#### Internal Logic
Similar to `test_integrated_pipeline_simple_tree`, but includes an allele table in the parameters.
### test\_integrated\_pipeline\_tree\_with\_meta\_data
#### Description
Tests the integrated pipeline for uploading and exporting a tree with metadata to iTOL.
#### Internal Logic
Similar to `test_integrated_pipeline_simple_tree`, but includes metadata parameters.
## Dependencies
* unittest
* os
* shutil
* tempfile
* networkx
* numpy
* pandas
* cassiopeia
## Configuration
The tests use a configuration file `~/.itolconfig` for iTOL credentials. Some tests are skipped if this file doesn't exist.
## Error Handling
The tests use `unittest` assertions to check for expected outcomes and file existence.
## Performance Considerations
The tests create and delete temporary files and directories, which may impact performance on slower file systems.
## TODOs
None explicitly mentioned in the code.
# local_3d_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/plotting_tests/local_3d_test.py
## High-level description
This file contains unit tests for the `local_3d` module in the Cassiopeia package. It tests various functions and classes related to 3D plotting of tree structures, including coordinate interpolation, color manipulation, label generation, and the main `Tree3D` class.
## Code Structure
The main test class `TestLocal3DPlotting` inherits from `unittest.TestCase`. It sets up a simulated tree and spatial data in the `setUp` method, which is then used across multiple test methods. Each test method focuses on a specific function or aspect of the `local_3d` module.
## Symbols
### TestLocal3DPlotting
#### Description
A test class that contains multiple unit tests for the `local_3d` module.
#### Internal Logic
1. Sets up a simulated tree and spatial data in `setUp`.
2. Tests individual functions like `interpolate_branch`, `polyline_from_points`, `average_mixing`, etc.
3. Tests the `labels_from_coordinates` function with various inputs and edge cases.
4. Tests the initialization of the `Tree3D` class.
### test\_interpolate\_branch
#### Description
Tests the `interpolate_branch` function from the `local_3d` module.
#### Internal Logic
Checks if the function correctly interpolates a branch between two 3D points.
### test\_polyline\_from\_points
#### Description
Tests the `polyline_from_points` function from the `local_3d` module.
#### Internal Logic
Verifies if the function correctly creates a polyline from a set of 3D points.
### test\_average\_mixing
#### Description
Tests the `average_mixing` function from the `local_3d` module.
#### Internal Logic
Checks if the function correctly averages multiple colors.
### test\_highlight
#### Description
Tests the `highlight` function from the `local_3d` module.
#### Internal Logic
Verifies if the function correctly brightens a given color.
### test\_lowlight
#### Description
Tests the `lowlight` function from the `local_3d` module.
#### Internal Logic
Checks if the function correctly dims a given color.
### test\_labels\_from\_coordinates
#### Description
Tests the `labels_from_coordinates` function from the `local_3d` module.
#### Internal Logic
1. Tests various invalid inputs (shape, attribute key).
2. Checks if the function correctly generates labels and updates tree metadata.
3. Tests with non-square shapes and dense spatial positions.
### test\_Tree3D
#### Description
Tests the initialization of the `Tree3D` class from the `local_3d` module.
#### Internal Logic
Initializes `Tree3D` objects with and without labels and calls the `plot` method to ensure no errors occur.
## Dependencies
The test file depends on the following modules:
| Dependency | Purpose |
| :------------ | :--------------------------------------------------------------------- |
| unittest | Provides the testing framework |
| unittest.mock | Used for mocking objects in tests |
| networkx | Graph library (imported but not used in the provided code) |
| numpy | Used for numerical operations and assertions |
| pandas | Data manipulation library (imported but not used in the provided code) |
| cassiopeia | The main package being tested |
## Error Handling
The test methods use various `assert` statements and `self.assertRaises` to check for expected behaviors and error conditions.
# local_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/plotting_tests/local_test.py
Here's a comprehensive documentation for the `test/plotting_tests/local_test.py` file:
## High-level description
This file contains unit tests for the local plotting functionality in the Cassiopeia package. It tests various aspects of the local plotting module, including color strip creation, indel heatmap generation, and the main plotting functions for both matplotlib and plotly.
## Code Structure
The main class `TestLocalPlotting` inherits from `unittest.TestCase` and contains multiple test methods. Each method tests a specific functionality of the local plotting module. The class uses a `setUp` method to create a sample `CassiopeiaTree` and allele table for testing purposes.
## Symbols
### TestLocalPlotting
#### Description
A test class that contains unit tests for the local plotting functionality in Cassiopeia.
#### Internal Logic
1. Sets up a sample `CassiopeiaTree` and allele table in the `setUp` method.
2. Contains multiple test methods, each testing a specific aspect of the local plotting module.
### setUp
#### Description
Initializes the test environment by creating a sample allele table, graph, cell metadata, and a `CassiopeiaTree` object.
### test\_compute\_colorstrip\_size
#### Description
Tests the `compute_colorstrip_size` function for various locations (up, right, left, polar).
### test\_create\_categorical\_colorstrip
#### Description
Tests the `create_categorical_colorstrip` function for creating a color strip based on categorical data.
### test\_create\_continous\_colorstrip
#### Description
Tests the `create_continuous_colorstrip` function for creating a color strip based on continuous data. Also tests for a warning when invalid vmin and vmax are provided.
### test\_create\_indel\_heatmap
#### Description
Tests the `create_indel_heatmap` function for generating a heatmap of indels.
### test\_plot\_matplotlib
#### Description
Tests the `plot_matplotlib` function with various parameters and checks for error handling with invalid metadata.
### test\_plot\_plotly
#### Description
Tests the `plot_plotly` function with various parameters.
### test\_create\_clade\_colors
#### Description
Tests the `create_clade_colors` function for generating node and branch colors based on clades.
## Dependencies
* unittest
* mock
* networkx
* numpy
* pandas
* cassiopeia
## Error Handling
The test methods include assertions to check for expected behavior and error handling. For example, `test_plot_matplotlib` checks for `PlottingError` when invalid metadata is provided.
## Logging
The test file doesn't implement any specific logging mechanisms, but it uses assertions and unittest's built-in reporting for test results.
## TODOs
There are no explicit TODOs in the code.
This test file is crucial for ensuring the correctness and robustness of the local plotting functionality in the Cassiopeia package. It covers various aspects of the plotting module, including color generation, heatmap creation, and the main plotting functions for both matplotlib and plotly backends.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/plotting_tests/overview
## High-level description
This directory contains unit tests for the plotting functionality in the Cassiopeia project. It includes tests for various plotting utilities, local 2D and 3D plotting, and iTOL (Interactive Tree of Life) plotting. The tests cover a wide range of functions and classes related to tree visualization, color manipulation, coordinate transformations, and data representation.
## What does it do?
The test suite in this directory ensures the correctness and reliability of Cassiopeia's plotting capabilities. It verifies:
1. Basic utility functions for color manipulation, coordinate transformations, and tree placement.
2. Generation of color strips and indel heatmaps for tree visualization.
3. 2D plotting functionality using both matplotlib and plotly backends.
4. 3D tree plotting and spatial data representation.
5. Integration with iTOL for online tree visualization and export.
These tests help maintain the integrity of Cassiopeia's plotting features, ensuring that researchers can reliably visualize and analyze their phylogenetic data.
## Key Files
### itol\_plotting\_test.py
This file tests the iTOL plotting utilities. It verifies the creation of gradient files, colorbar files, and allele heatmaps for iTOL visualization. It also tests the integrated pipeline for uploading and exporting trees to iTOL.
Key tests include:
* Generation of gradient and colorbar files
* Creation of allele heatmaps with various color specifications
* Integrated pipeline for uploading and exporting trees to iTOL
### local\_3d\_test.py
This file contains tests for 3D plotting of tree structures. It verifies functions related to coordinate interpolation, color manipulation, label generation, and the main `Tree3D` class.
Key tests include:
* Interpolation of branch coordinates
* Color mixing and manipulation functions
* Generation of labels from spatial coordinates
* Initialization and basic functionality of the `Tree3D` class
### local\_test.py
This file tests the local 2D plotting functionality. It covers color strip creation, indel heatmap generation, and the main plotting functions for both matplotlib and plotly backends.
Key tests include:
* Computation of color strip sizes
* Creation of categorical and continuous color strips
* Generation of indel heatmaps
* Plotting using matplotlib and plotly
* Creation of clade colors for nodes and branches
### utilities\_test.py
This file tests various utility functions used in plotting trees and other visualizations. It covers functions for coordinate transformations, color generation, and tree placement.
Key tests include:
* Conversion between degrees and radians
* Conversion between polar and Cartesian coordinates
* Generation of random colors
* Placement of tree nodes and branches
* Color conversion between hex and RGB formats
* Generation of indel colors based on priors
## Dependencies
The test suite relies on the following main dependencies:
* unittest: For the testing framework
* networkx: For graph and tree operations
* numpy: For numerical computations
* pandas: For data manipulation
* matplotlib and plotly: For plotting backends
* cassiopeia: The main package being tested
## Configuration
Some tests, particularly those related to iTOL integration, may require a configuration file (`~/.itolconfig`) for iTOL credentials. Tests that depend on this configuration are skipped if the file is not present.
The tests use various fixtures and setup methods to create sample data, including:
* Random number generation with fixed seeds for reproducibility
* Sample trees and allele tables
* Indel priors and color maps
These configurations ensure consistent and reproducible test results across different environments.
# utilities_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/plotting_tests/utilities_test.py
Here's a comprehensive documentation for the provided test file:
## High-level description
This file contains unit tests for the utility functions used in plotting trees and other visualizations in the Cassiopeia project. It tests various functions from the `cassiopeia.plotting.utilities` module, ensuring their correctness and expected behavior.
## Code Structure
The main class `TestPlottingUtilities` inherits from `unittest.TestCase` and contains multiple test methods, each focusing on a specific utility function or feature. The `setUp` method initializes common test data used across multiple test cases.
## Symbols
### TestPlottingUtilities
#### Description
A test class that contains unit tests for various plotting utility functions in the Cassiopeia project.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Implements multiple test methods, each targeting a specific utility function.
### setUp
#### Description
Initializes test data used across multiple test cases.
#### Internal Logic
1. Sets a random state for reproducibility.
2. Creates a sample indel priors DataFrame.
3. Constructs a basic tree structure using NetworkX.
4. Sets attributes on the tree nodes.
### test\_degrees\_to\_radians
#### Description
Tests the `degrees_to_radians` function for correct conversion.
#### Internal Logic
Asserts that 180 degrees equals π radians and 90 degrees equals π/2 radians.
### test\_polar\_to\_cartesian
#### Description
Tests the `polar_to_cartesian` function for correct conversion.
#### Internal Logic
Checks conversion for 0 and 90 degrees with a radius of 1.
### test\_generate\_random\_color
#### Description
Tests the `generate_random_color` function for generating random colors within specified ranges.
#### Internal Logic
1. Defines color intervals.
2. Generates a random color using the function.
3. Asserts that the generated color components are within expected ranges.
### test\_place\_tree
#### Description
Tests the `place_tree` function for correct placement of tree nodes and branches.
#### Internal Logic
1. Tests tree placement with default settings.
2. Tests tree placement without extending branches.
3. Tests tree placement using a custom depth key.
4. Tests tree placement with branch extension and custom depth key.
5. Tests tree placement in polar coordinates.
### test\_place\_colorstrip
#### Description
Tests the `place_colorstrip` function for correct placement of color strips.
#### Internal Logic
Tests color strip placement in different directions (right, left, up, down) and checks the resulting coordinates.
### test\_generate\_indel\_colors\_from\_priors
#### Description
Tests the `get_indel_colors` function for generating colors based on indel priors.
#### Internal Logic
1. Generates indel colors using the function.
2. Compares the generated colors with expected values.
### test\_color\_converters
#### Description
Tests the hex to RGB and RGB to hex color conversion functions.
#### Internal Logic
1. Tests conversion from hex to RGB.
2. Tests conversion from RGB to hex.
### test\_generate\_random\_indel\_colors
#### Description
Tests the `get_random_indel_colors` function for generating random colors for indels.
#### Internal Logic
1. Creates a sample lineage profile.
2. Generates random indel colors using the function.
3. Compares the generated colors with expected values.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia
## Error Handling
The test cases use various assertion methods from the `unittest` module to check for expected outcomes and raise errors if the results don't match the expectations.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# align_sequence_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/align_sequence_test.py
Here's a detailed explanation of the `align_sequence_test.py` file:
## High-level description
This file contains unit tests for the sequence alignment functionality in the Cassiopeia preprocessing pipeline. It tests various aspects of the `align_sequences` function, including the structure of the output, the behavior with different alignment parameters, and the correctness of alignments for specific input sequences.
## Code Structure
The main class `TestAlignSequence` inherits from `unittest.TestCase` and contains several test methods. Each method tests a different aspect of the sequence alignment functionality.
## Symbols
### TestAlignSequence
#### Description
This class contains all the unit tests for the sequence alignment functionality.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Defines several test methods, each focusing on a specific aspect of the alignment function.
### setUp
#### Description
Initializes the test data used across all test methods.
#### Internal Logic
1. Creates a DataFrame `self.queries` with sample sequence data.
2. Sets a reference sequence `self.reference`.
### test\_alignment\_dataframe\_structure
#### Description
Tests the structure of the output DataFrame from the `align_sequences` function.
#### Internal Logic
1. Calls `align_sequences` with test data.
2. Checks if the output DataFrame has the correct number of rows and expected columns.
3. Verifies that all cell barcodes from the input are present in the output.
### test\_extremely\_large\_gap\_open\_penalty
#### Description
Tests the alignment behavior when using an extremely large gap open penalty.
#### Internal Logic
1. Calls `align_sequences` with a very high gap open penalty (255).
2. Checks that no gaps (insertions or deletions) are present in the resulting alignments.
### test\_default\_alignment\_works
#### Description
Tests the correctness of alignments using default parameters.
#### Internal Logic
1. Calls `align_sequences` with default parameters.
2. Compares the resulting CIGAR strings and alignment scores with expected values for each input sequence.
### test\_global\_alignment
#### Description
Tests the global alignment mode of the `align_sequences` function.
#### Internal Logic
1. Calls `align_sequences` with the `method` parameter set to "global".
2. Compares the resulting CIGAR strings and alignment scores with expected values for global alignment.
## Dependencies
The test file depends on the following modules:
* `unittest`: For creating and running unit tests.
* `numpy`: For numerical operations.
* `pandas`: For handling DataFrames.
* `cassiopeia`: The main package being tested.
## Error Handling
The tests use assertions to check for expected outcomes. If any assertion fails, the test will raise an AssertionError.
## Performance Considerations
These tests are not specifically designed to test performance, but rather the correctness of the alignment function. However, the use of different parameters (like gap penalties and alignment methods) can affect the performance of the alignment algorithm.
In conclusion, this test file provides comprehensive coverage of the sequence alignment functionality in the Cassiopeia preprocessing pipeline, ensuring that the alignment function works correctly under various conditions and with different input parameters.
# call_alleles_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/call_alleles_test.py
## High-level description
This file contains unit tests for the allele calling functionality in the Cassiopeia library, specifically testing the `alignment_utilities.py` and `pipeline.py` modules. The tests cover various scenarios of CIGAR string parsing, indel detection, and allele calling for both basic and complex cases.
## Code Structure
The code is structured as a single test class `TestCallAlleles` that inherits from `unittest.TestCase`. It contains multiple test methods, each focusing on different aspects of allele calling and CIGAR string parsing. The `setUp` method initializes test data used across multiple test cases.
## Symbols
### TestCallAlleles
#### Description
A test class that contains multiple unit tests for the allele calling functionality in Cassiopeia.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Implements various test methods to cover different scenarios of CIGAR string parsing and allele calling.
3. Uses assertions to verify the correctness of the results.
### setUp
#### Description
Initializes test data used across multiple test cases.
#### Internal Logic
1. Sets up basic and long reference sequences.
2. Defines barcode intervals and cutsite locations.
3. Creates test alignment dataframes.
### test\_basic\_cigar\_string\_match
#### Description
Tests parsing of a basic CIGAR string with only matches.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a simple match case.
2. Asserts the correctness of the returned intBC and indels.
### test\_basic\_cigar\_string\_deletion
#### Description
Tests parsing of a CIGAR string with a deletion.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a deletion case.
2. Asserts the correctness of the returned intBC and indels.
### test\_basic\_cigar\_string\_deletion\_with\_context
#### Description
Tests parsing of a CIGAR string with a deletion, including context information.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a deletion case and context enabled.
2. Asserts the correctness of the returned intBC and indels, including context.
### test\_basic\_cigar\_string\_insertion
#### Description
Tests parsing of a CIGAR string with an insertion.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with an insertion case.
2. Asserts the correctness of the returned intBC and indels.
### test\_basic\_cigar\_string\_insertion\_with\_context
#### Description
Tests parsing of a CIGAR string with an insertion, including context information.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with an insertion case and context enabled.
2. Asserts the correctness of the returned intBC and indels, including context.
### test\_long\_cigar\_parsing\_no\_context
#### Description
Tests parsing of a long CIGAR string without context.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a long CIGAR string.
2. Asserts the correctness of the returned intBC and indels for multiple cutsites.
### test\_long\_cigar\_parsing\_with\_context
#### Description
Tests parsing of a long CIGAR string with context.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a long CIGAR string and context enabled.
2. Asserts the correctness of the returned intBC and indels for multiple cutsites, including context.
### test\_intersite\_deletion\_parsing
#### Description
Tests parsing of a CIGAR string with an intersite deletion.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a CIGAR string containing an intersite deletion.
2. Asserts the correctness of the returned intBC and indels for multiple cutsites.
### test\_complex\_cigar\_parsing\_intersite\_deletion
#### Description
Tests parsing of a complex CIGAR string with multiple intersite deletions.
#### Internal Logic
1. Calls `alignment_utilities.parse_cigar` with a complex CIGAR string containing multiple intersite deletions.
2. Asserts the correctness of the returned intBC and indels for multiple cutsites.
### test\_call\_alleles\_function
#### Description
Tests the `call_alleles` function from the Cassiopeia preprocessing module.
#### Internal Logic
1. Calls `cassiopeia.pp.call_alleles` with test alignment data.
2. Verifies the structure of the returned molecule table.
3. Asserts the correctness of the called alleles and intBCs for each read.
### test\_missing\_data\_in\_allele\_throws\_warning
#### Description
Tests that a warning is raised when there is missing data in the allele calling process.
#### Internal Logic
1. Calls `cassiopeia.pp.call_alleles` with alignment data containing missing information.
2. Asserts that a `PreprocessWarning` is raised.
## Dependencies
| Dependency | Purpose |
| :--------- | :----------------------------- |
| unittest | Provides the testing framework |
| numpy | Used for numerical operations |
| pandas | Used for handling dataframes |
| cassiopeia | The main library being tested |
## Error Handling
The code uses assertions to verify the correctness of the results. It also tests for the raising of a `PreprocessWarning` in case of missing data.
This test suite provides comprehensive coverage for the allele calling functionality in Cassiopeia, ensuring that various scenarios of CIGAR string parsing and indel detection are handled correctly.
# call_lineage_groups_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/call_lineage_groups_test.py
## High-level description
This file contains unit tests for the `call_lineage_groups` function in the Cassiopeia preprocessing pipeline. The tests verify the functionality of lineage group assignment, including basic grouping, doublet handling, reassignment, and filtering.
## Code Structure
The main class `TestCallLineageGroup` contains several test methods, each testing different aspects of the `call_lineage_groups` function. The `setUp` method initializes test data for various scenarios.
## Symbols
### TestCallLineageGroup
#### Description
A unittest.TestCase subclass containing tests for the `call_lineage_groups` function.
#### Internal Logic
1. Sets up test data in `setUp` method
2. Defines multiple test methods to check different aspects of lineage group assignment
### setUp
#### Description
Initializes test data for various scenarios, including basic grouping, reassignment, filtering, and doublet handling.
#### Internal Logic
1. Creates multiple pandas DataFrames with test data
2. Sets up different scenarios for lineage group assignment testing
### test\_format
#### Description
Tests if the output of `call_lineage_groups` has the expected columns.
#### Internal Logic
1. Calls `call_lineage_groups` with test data
2. Checks if the output DataFrame contains all expected columns
### test\_basic\_grouping
#### Description
Tests basic lineage group assignment functionality.
#### Internal Logic
1. Calls `call_lineage_groups` with basic test data
2. Verifies if the assigned lineage groups and UMI counts match expected values
### test\_doublet
#### Description
Tests lineage group assignment with potential doublets.
#### Internal Logic
1. Calls `call_lineage_groups` with doublet test data
2. Checks if doublets are correctly identified and assigned
### test\_reassign
#### Description
Tests reassignment of cells to lineage groups.
#### Internal Logic
1. Calls `call_lineage_groups` with reassignment test data
2. Verifies if cells are correctly reassigned to appropriate lineage groups
### test\_filter\_reassign
#### Description
Tests filtering and reassignment of lineage groups.
#### Internal Logic
1. Calls `call_lineage_groups` with filter and reassign test data
2. Checks if filtering and reassignment are performed correctly
### test\_filter\_lineage\_group\_to\_allele\_table\_single\_lineage
#### Description
Tests the conversion of filtered lineage groups to an allele table for a single lineage.
#### Internal Logic
1. Calls `filtered_lineage_group_to_allele_table` with test data
2. Verifies if the resulting allele table is correct for a single lineage
## Dependencies
* unittest
* numpy
* pandas
* cassiopeia (specifically the pipeline and lineage\_utils modules)
The test file relies on the `cassiopeia` package, particularly the `pipeline` and `lineage_utils` modules, to perform lineage group assignments and related operations.
# character_matrix_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/character_matrix_test.py
Here's a high-level description and documentation of the provided code:
## High-level description
This file contains unit tests for the character matrix formation functionality in the Cassiopeia preprocessing pipeline. It tests various aspects of converting allele tables to character matrices and lineage profiles, including handling of missing data, conflicts, and different input formats.
## Code Structure
The code defines a TestCharacterMatrixFormation class that inherits from unittest.TestCase. This class contains multiple test methods, each focusing on a specific aspect of the character matrix formation process. The setUp method initializes test data used across multiple test cases.
## Symbols
### TestCharacterMatrixFormation
#### Description
A test class containing multiple unit tests for the character matrix formation functionality.
#### Internal Logic
1. Sets up test data in the setUp method.
2. Defines multiple test methods, each testing a specific aspect of the character matrix formation process.
3. Uses assertions to verify the correctness of the output.
### setUp
#### Description
Initializes test data used across multiple test cases.
#### Internal Logic
1. Creates sample allele tables with various configurations.
2. Sets up mutation priors for testing.
### test\_basic\_character\_matrix\_formation
#### Description
Tests the basic functionality of converting an allele table to a character matrix.
#### Internal Logic
1. Calls convert\_alleletable\_to\_character\_matrix with basic input.
2. Verifies the shape and content of the resulting character matrix.
### test\_character\_matrix\_formation\_custom\_missing\_data
#### Description
Tests character matrix formation with custom missing data handling.
#### Internal Logic
1. Modifies the input data to include custom missing data.
2. Calls convert\_alleletable\_to\_character\_matrix with custom missing data parameters.
3. Verifies the correct handling of missing data in the output.
### test\_character\_matrix\_formation\_with\_conflicts
#### Description
Tests character matrix formation when there are conflicting alleles.
#### Internal Logic
1. Uses an allele table with conflicting data.
2. Calls convert\_alleletable\_to\_character\_matrix.
3. Verifies that conflicts are correctly represented in the output.
### test\_ignore\_intbc
#### Description
Tests the ability to ignore specific integration barcodes (intBCs) during character matrix formation.
#### Internal Logic
1. Calls convert\_alleletable\_to\_character\_matrix with specific intBCs to ignore.
2. Verifies that the specified intBCs are not included in the output.
### test\_filter\_out\_low\_diversity\_intbcs
#### Description
Tests filtering out integration barcodes with low diversity.
#### Internal Logic
1. Calls convert\_alleletable\_to\_character\_matrix with a high allele representation threshold.
2. Verifies that low-diversity intBCs are removed from the output.
### test\_mutation\_prior\_formation
#### Description
Tests the formation of mutation priors during character matrix conversion.
#### Internal Logic
1. Calls convert\_alleletable\_to\_character\_matrix with mutation priors.
2. Verifies that the resulting prior probabilities are correct.
### test\_indel\_state\_mapping\_formation
#### Description
Tests the formation of indel state mappings during character matrix conversion.
#### Internal Logic
1. Calls convert\_alleletable\_to\_character\_matrix with mutation priors.
2. Verifies that the resulting indel state mappings are correct.
### test\_alleletable\_to\_lineage\_profile
#### Description
Tests the conversion of an allele table to a lineage profile.
#### Internal Logic
1. Calls convert\_alleletable\_to\_lineage\_profile.
2. Verifies that the resulting lineage profile is correct.
### test\_lineage\_profile\_to\_character\_matrix\_no\_priors
#### Description
Tests the conversion of a lineage profile to a character matrix without priors.
#### Internal Logic
1. Converts an allele table to a lineage profile.
2. Calls convert\_lineage\_profile\_to\_character\_matrix without priors.
3. Verifies the correctness of the resulting character matrix.
### test\_lineage\_profile\_to\_character\_matrix\_with\_priors
#### Description
Tests the conversion of a lineage profile to a character matrix with priors.
#### Internal Logic
1. Converts an allele table to a lineage profile.
2. Calls convert\_lineage\_profile\_to\_character\_matrix with priors.
3. Verifies the correctness of the resulting character matrix and priors.
### test\_compute\_empirical\_indel\_probabilities
#### Description
Tests the computation of empirical indel probabilities.
#### Internal Logic
1. Calls compute\_empirical\_indel\_priors.
2. Verifies that the resulting probabilities are correct.
### test\_noncanonical\_cut\_sites\_allele\_table\_to\_character\_matrix
#### Description
Tests character matrix formation with non-canonical cut site names.
#### Internal Logic
1. Uses an allele table with non-standard cut site column names.
2. Calls convert\_alleletable\_to\_character\_matrix with custom cut site names.
3. Verifies that the resulting character matrix is correct.
## Dependencies
* unittest: For defining and running unit tests
* numpy: For numerical operations
* pandas: For data manipulation and analysis
* cassiopeia: The main package being tested
This test suite provides comprehensive coverage of the character matrix formation functionality in Cassiopeia, ensuring that it handles various edge cases and input formats correctly.
# collapse_umi_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/collapse_umi_test.py
Here's a detailed documentation of the target file `test/preprocess_tests/collapse_umi_test.py`:
## High-level description
This file contains unit tests for the UMI Collapsing module in the Cassiopeia preprocessing pipeline. It tests various aspects of UMI collapsing, including sorting BAM files, forming collapsed clusters, and converting BAM files to DataFrames.
## Code Structure
The main class `TestCollapseUMIs` inherits from `unittest.TestCase` and contains several test methods. The `setUp` method prepares the test environment by creating test files and running UMI collapsing operations. The test methods then verify the correctness of these operations.
## Symbols
### TestCollapseUMIs
#### Description
A test class that contains unit tests for the UMI Collapsing functionality.
#### Internal Logic
1. Sets up test files and performs UMI collapsing operations in `setUp`.
2. Tests various aspects of UMI collapsing in individual test methods.
### setUp
#### Description
Prepares the test environment by creating test files and running UMI collapsing operations.
#### Internal Logic
1. Creates test directories and files.
2. Sorts BAM files.
3. Performs UMI collapsing using different methods and parameters.
### test\_sort\_bam
#### Description
Tests the sorting of BAM files.
#### Internal Logic
1. Opens the sorted BAM file.
2. Extracts cell barcodes and UMIs.
3. Verifies the correctness of specific entries.
### test\_sort\_bam\_uncorrected
#### Description
Tests the sorting of uncorrected BAM files.
#### Internal Logic
Similar to `test_sort_bam`, but uses uncorrected test files.
### test\_collapse\_bam
#### Description
Tests the collapsing of BAM files.
#### Internal Logic
1. Opens the collapsed BAM file.
2. Extracts various fields (cell barcodes, UMIs, read counts, cluster IDs, qualities).
3. Verifies the correctness of specific entries and counts.
### test\_collapse\_bam\_bayesian
#### Description
Tests the Bayesian collapsing of BAM files.
#### Internal Logic
Similar to `test_collapse_bam`, but uses Bayesian collapsed files.
### test\_collapse\_bam\_uncorrected
#### Description
Tests the collapsing of uncorrected BAM files.
#### Internal Logic
Similar to `test_collapse_bam`, but uses uncorrected collapsed files.
### test\_bam2DF
#### Description
Tests the conversion of BAM files to DataFrames.
#### Internal Logic
1. Converts BAM to DataFrame.
2. Verifies specific entries in the resulting DataFrame.
3. Compares with a separately loaded DataFrame from a text file.
### test\_collapsing\_passes\_header
#### Description
Tests that the collapsing process correctly passes header information.
#### Internal Logic
1. Opens the collapsed BAM file.
2. Verifies that the alignment has non-None query sequence and qualities.
## Dependencies
| Dependency | Purpose |
| :-------------------- | :------------------------------------ |
| os | File and directory operations |
| unittest | Unit testing framework |
| pandas | Data manipulation and analysis |
| pathlib | Object-oriented filesystem paths |
| pysam | Reading/writing SAM/BAM/VCF/BCF files |
| cassiopeia.preprocess | UMI collapsing functionality |
## Error Handling
The tests use assertions to verify the correctness of operations. If any assertion fails, an `AssertionError` will be raised, indicating a test failure.
This file is crucial for ensuring the correctness of the UMI collapsing functionality in the Cassiopeia preprocessing pipeline. It covers various scenarios and edge cases, helping to maintain the reliability of the UMI collapsing process.
# config_parser_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/config_parser_test.py
## High-level description
This file contains unit tests for the config parsing functionality of the cassiopeia\_preprocess pipeline. It tests the parsing of configuration files, the creation of pipeline stages, and the handling of various configuration scenarios.
## Code Structure
The main class `TestCollapseUMIs` inherits from `unittest.TestCase` and contains several test methods. The `setUp` method initializes test data, including different configuration scenarios. The test methods then use this data to verify the correct behavior of the config parsing and pipeline setup functions.
## Symbols
### TestCollapseUMIs
#### Description
A test class that inherits from `unittest.TestCase`, containing methods to test the configuration parsing and pipeline setup for the cassiopeia\_preprocess pipeline.
#### Internal Logic
1. Sets up test data in the `setUp` method, including various configuration scenarios.
2. Tests the parsing of a valid configuration file.
3. Tests the handling of an unspecified configuration parameter.
4. Tests the correct setup of the full pipeline.
5. Tests the correct setup of a subset pipeline.
### TestCollapseUMIs.setUp
#### Description
Initializes test data for the test cases, including different configuration scenarios.
#### Internal Logic
1. Creates a basic configuration dictionary and its string representation.
2. Creates a subset configuration dictionary and its string representation.
3. Creates a failure configuration dictionary and its string representation.
### TestCollapseUMIs.test\_read\_good\_config
#### Description
Tests the parsing of a valid configuration file.
#### Internal Logic
1. Parses the basic configuration string.
2. Checks that default parameters are set correctly.
3. Verifies that specified parameters are updated correctly.
4. Ensures that output directories are set for various pipeline stages.
### TestCollapseUMIs.test\_unspecified\_config\_raises\_error
#### Description
Tests that an error is raised when an unspecified configuration parameter is encountered.
#### Internal Logic
1. Attempts to parse the failure configuration string.
2. Asserts that a `UnspecifiedConfigParameterError` is raised.
### TestCollapseUMIs.test\_pipeline\_setup\_correct
#### Description
Tests the correct setup of the full pipeline.
#### Internal Logic
1. Parses the basic configuration string.
2. Creates the pipeline using the entry and exit points from the configuration.
3. Compares the created pipeline stages with the expected stages.
### TestCollapseUMIs.test\_subset\_pipeline\_setup\_correct
#### Description
Tests the correct setup of a subset pipeline.
#### Internal Logic
1. Parses the subset configuration string.
2. Creates the pipeline using the entry and exit points from the configuration.
3. Compares the created pipeline stages with the expected subset of stages.
## Dependencies
| Dependency | Purpose |
| :------------------------------------------- | :------------------------------------------------------------------ |
| unittest | Provides the testing framework |
| pandas | Used for data manipulation (imported but not used in the test file) |
| cassiopeia.preprocess.cassiopeia\_preprocess | Contains the main pipeline logic |
| cassiopeia.preprocess.constants | Contains constants used in the pipeline |
| cassiopeia.preprocess.setup\_utilities | Provides utility functions for setting up the pipeline |
## Error Handling
The test cases check for proper error handling, specifically testing that a `UnspecifiedConfigParameterError` is raised when an unspecified configuration parameter is encountered.
This test file ensures that the configuration parsing and pipeline setup functionality of the cassiopeia\_preprocess pipeline works correctly under various scenarios, including full pipeline setup, subset pipeline setup, and handling of configuration errors.
# convert_fastqs_to_unmapped_bam_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/convert_fastqs_to_unmapped_bam_test.py
Here's a detailed documentation of the target file `test/preprocess_tests/convert_fastqs_to_unmapped_bam_test.py`:
## High-level description
This file contains unit tests for the `convert_fastqs_to_unmapped_bam` function in the `cassiopeia.preprocess.pipeline` module. The tests verify the correct conversion of FASTQ files to unmapped BAM files for different sequencing chemistries.
## Code Structure
The main class `TestConvertFastqsToUnmappedBam` contains several test methods, each testing a different sequencing chemistry. The tests use sample FASTQ files and verify the correct conversion to BAM format, including proper tag assignments.
## Symbols
### TestConvertFastqsToUnmappedBam
#### Description
A unittest.TestCase subclass that contains test methods for the `convert_fastqs_to_unmapped_bam` function.
#### Internal Logic
* Sets up test FASTQ file paths in the `setUp` method
* Implements test methods for different sequencing chemistries (dropseq, 10xv2, 10xv3, indropsv3, slideseq2)
* Each test method:
1. Calls `convert_fastqs_to_unmapped_bam` with appropriate parameters
2. Reads the resulting BAM file
3. Verifies the correct number of alignments, query names, sequences, and qualities
4. Checks the correct assignment of BAM tags (UR, UY, CR, CY, RG)
### setUp
#### Description
Initializes the test environment by setting up file paths for test FASTQ files.
### test\_dropseq
#### Description
Tests the conversion of FASTQ files to BAM for the Drop-seq chemistry.
### test\_10xv2
#### Description
Tests the conversion of FASTQ files to BAM for the 10x Genomics v2 chemistry.
### test\_10xv3
#### Description
Tests the conversion of FASTQ files to BAM for the 10x Genomics v3 chemistry.
### test\_indropsv3
#### Description
Tests the conversion of FASTQ files to BAM for the inDrops v3 chemistry.
### test\_slideseq2
#### Description
Tests the conversion of FASTQ files to BAM for the Slide-seq v2 chemistry.
## Dependencies
| Dependency | Purpose |
| :----------------------------- | :------------------------------------------ |
| os | File path operations |
| unittest | Unit testing framework |
| tempfile | Creating temporary directories |
| pysam | Reading and manipulating BAM files |
| ngs\_tools | FASTQ file handling |
| cassiopeia.preprocess.pipeline | Module containing the function being tested |
## Error Handling
The tests use assertions to verify the correctness of the BAM conversion. If any assertion fails, the test will raise an AssertionError, indicating a problem with the conversion process.
This test file is crucial for ensuring the correct functionality of the FASTQ to BAM conversion process in the Cassiopeia preprocessing pipeline, covering various sequencing chemistries and verifying the proper assignment of BAM tags.
# error_correct_cellbcs_to_whitelist_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/error_correct_cellbcs_to_whitelist_test.py
Here's a detailed documentation of the target file:
## High-level description
This file contains unit tests for the `error_correct_cellbcs_to_whitelist` function in the `cassiopeia.preprocess.pipeline` module. The tests verify the functionality of correcting raw cell barcodes to a whitelist for different sequencing chemistries (10x Genomics v3 and Slide-seq v2).
## Code Structure
The main class `TestErrorCorrectCellBCsToWhitelist` contains setup methods and test cases. It uses the `unittest` framework and relies on the `pysam` library for BAM file operations.
## Symbols
### TestErrorCorrectCellBCsToWhitelist
#### Description
A test class that inherits from `unittest.TestCase`. It contains methods to set up test data and run tests for the `error_correct_cellbcs_to_whitelist` function.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Defines test methods for different scenarios:
* 10x v3 chemistry with file-based whitelist
* 10x v3 chemistry with list-based whitelist
* Slide-seq v2 chemistry with file-based whitelist
* Slide-seq v2 chemistry with list-based whitelist
### setUp
#### Description
Initializes test data and file paths for the test cases.
#### Internal Logic
1. Determines the directory path of the test file.
2. Sets up file paths for test BAM files and whitelists.
3. Defines whitelist sequences for 10x v3 and Slide-seq v2 chemistries.
### test\_10xv3
#### Description
Tests the `error_correct_cellbcs_to_whitelist` function for 10x v3 chemistry using a file-based whitelist.
#### Internal Logic
1. Calls `error_correct_cellbcs_to_whitelist` with 10x v3 BAM file and whitelist file.
2. Verifies the number of alignments and their corrected cell barcodes.
### test\_10xv3\_whitelist\_list
#### Description
Tests the `error_correct_cellbcs_to_whitelist` function for 10x v3 chemistry using a list-based whitelist.
#### Internal Logic
1. Calls `error_correct_cellbcs_to_whitelist` with 10x v3 BAM file and whitelist list.
2. Verifies the number of alignments and their corrected cell barcodes.
### test\_slideseq2
#### Description
Tests the `error_correct_cellbcs_to_whitelist` function for Slide-seq v2 chemistry using a file-based whitelist.
#### Internal Logic
1. Calls `error_correct_cellbcs_to_whitelist` with Slide-seq v2 BAM file and whitelist file.
2. Verifies the number of alignments, presence of CB tags, and the corrected cell barcode.
### test\_slideseq2\_whitelist\_list
#### Description
Tests the `error_correct_cellbcs_to_whitelist` function for Slide-seq v2 chemistry using a list-based whitelist.
#### Internal Logic
1. Calls `error_correct_cellbcs_to_whitelist` with Slide-seq v2 BAM file and whitelist list.
2. Verifies the number of alignments, presence of CB tags, and the corrected cell barcode.
## Dependencies
* unittest: Python's built-in unit testing framework
* os: For file and directory operations
* tempfile: For creating temporary directories
* pysam: For BAM file operations
* ngs\_tools: A custom library for NGS data processing
* cassiopeia.preprocess.pipeline: The module containing the function being tested
## Error Handling
The test cases use assertions to verify the expected outcomes. If any assertion fails, an `AssertionError` will be raised, indicating a test failure.
# error_correct_intbcs_to_whitelist_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/error_correct_intbcs_to_whitelist_test.py
## High-level description
This file contains unit tests for the `error_correct_intbcs_to_whitelist` function in the Cassiopeia package. The tests verify the functionality of correcting integration barcodes (intBCs) to a whitelist, using both a file path and a list as input for the whitelist.
## Code Structure
The code defines a single test class `TestErrorCorrectIntBCstoWhitelist` with a `setUp` method to initialize test data and two test methods to verify the correction functionality.
## Symbols
### TestErrorCorrectIntBCstoWhitelist
#### Description
A test class that inherits from `unittest.TestCase`, containing setup and test methods for the `error_correct_intbcs_to_whitelist` function.
#### Internal Logic
1. Sets up test data in the `setUp` method, including a whitelist file path, a whitelist list, and a DataFrame with sample data.
2. Defines two test methods to verify the correction functionality.
### setUp
#### Description
Initializes test data, including file paths, whitelist, and a sample DataFrame.
#### Internal Logic
1. Sets up file paths for test files.
2. Creates a whitelist of integration barcodes.
3. Generates a sample DataFrame with various columns including cellBC, UMI, readCount, Seq, intBC, etc.
4. Adds 'readName' and 'allele' columns to the DataFrame.
5. Defines a dictionary of expected corrections for intBCs.
### test\_correct
#### Description
Tests the `error_correct_intbcs_to_whitelist` function using a file path for the whitelist.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_intbcs_to_whitelist` with the sample DataFrame and whitelist file path.
2. Creates an expected DataFrame by applying corrections to the intBC column.
3. Compares the result with the expected DataFrame using `pd.testing.assert_frame_equal`.
### test\_correct\_whitelist\_list
#### Description
Tests the `error_correct_intbcs_to_whitelist` function using a list for the whitelist.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_intbcs_to_whitelist` with the sample DataFrame and whitelist list.
2. Creates an expected DataFrame by applying corrections to the intBC column.
3. Compares the result with the expected DataFrame using `pd.testing.assert_frame_equal`.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------- |
| os | File path operations |
| unittest | Unit testing framework |
| numpy | Not directly used in the code |
| pandas | Data manipulation and testing |
| cassiopeia | Package being tested |
## Error Handling
The code relies on the default error handling provided by the unittest framework. No specific error handling mechanisms are implemented in this test file.
# error_correct_umi_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/error_correct_umi_test.py
Here's a detailed documentation of the provided code:
## High-level description
This file contains unit tests for the error correction of UMI sequences in the Cassiopeia pipeline. It tests various scenarios of UMI error correction, including handling of conflicts and different distance thresholds.
## Code Structure
The code defines a single test class `TestErrorCorrectUMISequence` that inherits from `unittest.TestCase`. This class contains multiple test methods, each testing different aspects of the UMI error correction functionality.
## Symbols
### `TestErrorCorrectUMISequence`
#### Description
A test class that contains multiple test methods for validating the UMI error correction functionality in the Cassiopeia pipeline.
#### Internal Logic
The class sets up test data in the `setUp` method and defines several test methods to check different aspects of UMI error correction.
### `setUp`
#### Description
Initializes test data for use in the test methods.
#### Internal Logic
Creates two pandas DataFrames:
1. `self.multi_case`: A complex case with multiple cells and UMIs.
2. `self.ambiguous`: A case with ambiguous alleles.
Both DataFrames are populated with sample data including cell barcodes, UMIs, read counts, sequences, and other relevant information.
### `test_format`
#### Description
Tests if the output of the error correction function has the expected format.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_umis` on `self.multi_case`.
2. Checks if the resulting DataFrame has all the expected columns.
### `test_zero_dist`
#### Description
Tests the behavior of the error correction function when the maximum UMI distance is set to 0.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_umis` with `max_umi_distance=0`.
2. Verifies that the output has the same shape as the input.
3. Checks if all original cell barcodes are present in the output.
### `test_error_correct_two_dist`
#### Description
Tests the error correction function with a maximum UMI distance of 2.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_umis` with `max_umi_distance=2`.
2. Checks if the read counts in the output match the expected values for specific read names.
### `test_error_correct_allow_conflicts`
#### Description
Tests the error correction function when allowing allele conflicts.
#### Internal Logic
1. Calls `cassiopeia.pp.error_correct_umis` with `allow_allele_conflicts=True`.
2. Verifies if the read counts in the output match the expected values for specific read names.
## Dependencies
* unittest
* numpy
* pandas
* cassiopeia
## Error Handling
The test methods use assertions to check if the output of the error correction function matches the expected results. If any assertion fails, the test will raise an AssertionError.
## Notes
* The tests cover various scenarios including zero distance, two-distance error correction, and handling of allele conflicts.
* The test data includes cases with multiple cells, UMIs, and potential conflicts to ensure robust testing of the error correction functionality.
# filter_bam_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/filter_bam_test.py
## High-level description
This file contains unit tests for the `filter_bam` function from the `cassiopeia.preprocess.pipeline` module. The tests verify that the function correctly filters BAM files based on a specified read quality threshold.
## Symbols
### `TestFilterBam`
#### Description
This is a test class that inherits from `unittest.TestCase`. It contains test methods to verify the functionality of the `filter_bam` function.
#### Internal Logic
1. Sets up the test environment by defining the path to a test BAM file.
2. Implements a test method `test_filter` to check the filtering functionality with different quality thresholds.
### `setUp`
#### Description
This method is called before each test method to set up the test environment.
#### Internal Logic
1. Determines the directory path of the current file.
2. Constructs the path to the test files directory.
3. Sets the path to a 10x Genomics v3 unmapped BAM file for testing.
### `test_filter`
#### Description
This method tests the `filter_bam` function with different quality thresholds.
#### Internal Logic
1. Calls `filter_bam` with a quality threshold of 10 and verifies that 2 alignments pass the filter.
2. Calls `filter_bam` with a quality threshold of 20 and verifies that 0 alignments pass the filter.
3. Uses `pysam.AlignmentFile` to read the filtered BAM files and count the number of alignments.
4. Uses `assertEqual` to check if the number of filtered alignments matches the expected count.
## Dependencies
| Dependency | Purpose |
| :----------------------------- | :-------------------------------------------------------- |
| os | File path operations |
| unittest | Unit testing framework |
| tempfile | Creating temporary directories |
| pysam | Reading BAM files |
| ngs\_tools | Not directly used in the test file, but imported as `ngs` |
| cassiopeia.preprocess.pipeline | Module containing the `filter_bam` function being tested |
## Error Handling
The test class uses the `unittest` framework's assertion methods to check for expected outcomes. If any assertion fails, the test will raise an exception, indicating a test failure.
## Notes
1. The test file uses a specific 10x Genomics v3 unmapped BAM file for testing, located in the "test\_files" directory.
2. The `filter_bam` function is called with temporary directories created using `tempfile.mkdtemp()` to store the output filtered BAM files.
3. The tests check two scenarios: one where some reads pass the quality filter (threshold 10) and another where no reads pass the filter (threshold 20).
4. The `pysam` library is used to read the filtered BAM files and count the number of alignments.
5. The `ngs_tools` module is imported but not used directly in this test file.
# filter_molecule_table_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/filter_molecule_table_test.py
Here's a detailed explanation of the `test/preprocess_tests/filter_molecule_table_test.py` file:
## High-level description
This file contains unit tests for the `filter_molecule_table` function in the Cassiopeia preprocessing pipeline. It tests various aspects of the function, including filtering based on UMI and cell barcode counts, handling of doublets, error correction of integration barcodes (intBCs), and allowing for allele conflicts.
## Code Structure
The main class `TestFilterMolculeTable` inherits from `unittest.TestCase` and contains several test methods. Each test method focuses on a specific aspect of the `filter_molecule_table` function's behavior.
## Symbols
### TestFilterMolculeTable
#### Description
This is the main test class that contains all the unit tests for the `filter_molecule_table` function.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Defines various test methods to check different aspects of the `filter_molecule_table` function.
3. Cleans up temporary directories in the `tearDown` method.
### setUp
#### Description
Initializes test data for use in the test methods.
#### Internal Logic
1. Creates a base case DataFrame (`self.base_filter_case`) with sample data.
2. Creates a DataFrame for testing doublet handling (`self.doublets_case`).
3. Creates a DataFrame for testing intBC error correction (`self.intBC_case`).
4. Sets up a temporary directory for output files.
### test\_format
#### Description
Tests if the output DataFrame from `filter_molecule_table` has the expected columns.
#### Internal Logic
1. Calls `filter_molecule_table` with the base case data.
2. Checks if the resulting DataFrame contains all the expected columns.
### test\_umi\_and\_cellbc\_filter
#### Description
Tests the UMI and cell barcode filtering functionality of `filter_molecule_table`.
#### Internal Logic
1. Calls `filter_molecule_table` with specific filtering parameters.
2. Checks if the resulting DataFrame contains only the expected alignments after filtering.
### test\_doublet\_and\_map
#### Description
Tests the doublet handling and mapping functionality of `filter_molecule_table`.
#### Internal Logic
1. Calls `filter_molecule_table` with doublet-specific test data and parameters.
2. Verifies if the resulting DataFrame contains the expected alleles after doublet handling.
### test\_error\_correct\_intBC
#### Description
Tests the integration barcode (intBC) error correction functionality of `filter_molecule_table`.
#### Internal Logic
1. Calls `filter_molecule_table` with intBC-specific test data.
2. Checks if the resulting DataFrame contains the expected corrected intBCs.
### test\_filter\_allow\_conflicts
#### Description
Tests the `allow_allele_conflicts` parameter of `filter_molecule_table`.
#### Internal Logic
1. Calls `filter_molecule_table` with `allow_allele_conflicts=True`.
2. Verifies if the resulting DataFrame retains the expected allele conflicts.
### tearDown
#### Description
Cleans up the temporary directory created during the tests.
#### Internal Logic
Removes the temporary directory and its contents.
## Dependencies
* unittest: Python's built-in unit testing framework
* shutil: For file and directory operations
* tempfile: For creating temporary directories
* numpy: For numerical operations
* pandas: For data manipulation and analysis
* cassiopeia: The main package being tested
This test file is crucial for ensuring the correct functionality of the `filter_molecule_table` function, which is an important part of the Cassiopeia preprocessing pipeline. It covers various edge cases and scenarios that the function might encounter when processing real data.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/overview
## High-level description
This directory contains unit tests for the preprocessing module of the Cassiopeia library. The tests cover various aspects of the preprocessing pipeline, including sequence alignment, allele calling, lineage group assignment, character matrix formation, UMI collapsing, and configuration parsing.
## What does it do?
The tests in this directory verify the correctness of different preprocessing steps in the Cassiopeia pipeline. These steps include:
1. Aligning sequences to a reference
2. Calling alleles from aligned sequences
3. Assigning lineage groups to cells
4. Converting allele tables to character matrices
5. Collapsing UMIs (Unique Molecular Identifiers)
6. Parsing configuration files for the pipeline
7. Converting FASTQ files to unmapped BAM files
8. Error-correcting cell barcodes and integration barcodes
9. Filtering BAM files and molecule tables
10. Resolving UMI sequences
Each test file focuses on a specific aspect of the preprocessing pipeline, ensuring that the functions and methods perform as expected under various input conditions and edge cases.
## Entry points
The main entry points for developers working on the preprocessing module tests are:
1. `align_sequence_test.py`: Tests for sequence alignment functionality
2. `call_alleles_test.py`: Tests for allele calling from aligned sequences
3. `call_lineage_groups_test.py`: Tests for lineage group assignment
4. `character_matrix_test.py`: Tests for converting allele tables to character matrices
5. `collapse_umi_test.py`: Tests for UMI collapsing functionality
6. `config_parser_test.py`: Tests for parsing configuration files
These files contain the core tests for the main preprocessing steps. Other test files focus on more specific functionalities or edge cases within the preprocessing pipeline.
## Key Files
1. `align_sequence_test.py`: Tests sequence alignment with different parameters and methods.
2. `call_alleles_test.py`: Verifies correct allele calling from CIGAR strings and aligned sequences.
3. `call_lineage_groups_test.py`: Checks lineage group assignment, including handling of doublets and reassignment.
4. `character_matrix_test.py`: Tests conversion of allele tables to character matrices and lineage profiles.
5. `collapse_umi_test.py`: Verifies UMI collapsing for different sequencing chemistries.
6. `config_parser_test.py`: Ensures correct parsing of configuration files and pipeline setup.
7. `convert_fastqs_to_unmapped_bam_test.py`: Tests conversion of FASTQ files to unmapped BAM files.
8. `error_correct_cellbcs_to_whitelist_test.py`: Checks error correction of cell barcodes against a whitelist.
9. `error_correct_intbcs_to_whitelist_test.py`: Verifies error correction of integration barcodes.
10. `error_correct_umi_test.py`: Tests UMI error correction functionality.
11. `filter_bam_test.py`: Checks filtering of BAM files based on read quality.
12. `filter_molecule_table_test.py`: Tests filtering of molecule tables based on various criteria.
13. `resolve_umi_sequence_test.py`: Verifies UMI sequence resolution and cell filtering.
## Dependencies
The test files rely on the following main dependencies:
1. `unittest`: Python's built-in unit testing framework
2. `numpy`: For numerical operations
3. `pandas`: For data manipulation and analysis
4. `pysam`: For reading and manipulating SAM/BAM files
5. `cassiopeia`: The main package being tested
Additional dependencies include:
* `os`, `shutil`, `tempfile`: For file and directory operations
* `pathlib`: For handling file paths
* `ngs_tools`: For FASTQ file handling
## Configuration
Many test files use configuration parameters to set up test scenarios. These configurations are typically defined within the test methods or in the `setUp` methods of test classes. Some tests also read configuration files to verify the correct parsing of pipeline settings.
The tests cover various configuration scenarios, including:
* Different sequencing chemistries (e.g., 10x Genomics v2/v3, Drop-seq, inDrops v3, Slide-seq v2)
* Various alignment parameters and methods
* Different filtering thresholds for UMIs, cell barcodes, and read counts
* Error correction settings for cell barcodes, integration barcodes, and UMIs
By testing these different configurations, the test suite ensures that the Cassiopeia preprocessing pipeline can handle a wide range of input data and user-defined settings.
# resolve_umi_sequence_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/preprocess_tests/resolve_umi_sequence_test.py
Here's a detailed documentation of the target file `test/preprocess_tests/resolve_umi_sequence_test.py`:
## High-level description
This file contains unit tests for the UMI Resolution module in the Cassiopeia preprocessing pipeline. It tests the functionality of resolving UMI sequences and filtering cells based on UMI and read count thresholds.
## Code Structure
The main class `TestResolveUMISequence` inherits from `unittest.TestCase` and contains several test methods. It uses a sample collapsed UMI table to test the `resolve_umi_sequence` function from the `pipeline` module.
## Symbols
### `TestResolveUMISequence`
#### Description
A test case class that contains methods to test the UMI resolution functionality.
#### Internal Logic
1. Sets up a sample collapsed UMI table in the `setUp` method.
2. Contains test methods to verify different aspects of UMI resolution.
### `setUp`
#### Description
Initializes the test environment by creating a sample collapsed UMI table and setting up a temporary directory.
### `test_resolve_umi`
#### Description
Tests the basic functionality of the `resolve_umi_sequence` function.
#### Internal Logic
1. Calls `resolve_umi_sequence` with the sample data.
2. Checks if the correct sequence was selected for cell1-UMIA.
3. Verifies that cell2 was filtered out.
4. Ensures cell3 retained both UMIs.
5. Checks the expected read counts for each cell.
### `test_filter_by_reads`
#### Description
Tests the filtering of cells based on the average number of reads per UMI.
#### Internal Logic
1. Calls `resolve_umi_sequence` with a higher `min_avg_reads_per_umi` threshold.
2. Verifies that only the expected cells (cell3) remain after filtering.
3. Checks that the expected removed cells (cell1, cell2) are not in the result.
### `tearDown`
#### Description
Cleans up the temporary directory after each test.
## Dependencies
* `unittest`: Python's built-in unit testing framework
* `os`: For file path operations
* `shutil`: For directory removal
* `tempfile`: For creating temporary directories
* `pandas`: For data manipulation
* `cassiopeia.preprocess.pipeline`: The module being tested
## Configuration
The tests use a predefined collapsed UMI table and various parameters for the `resolve_umi_sequence` function.
## Error Handling
The tests use assertions to verify the expected outcomes. If any assertion fails, the test will raise an `AssertionError`.
## Notes
* The tests use a small, predefined dataset to verify the functionality of the UMI resolution process.
* The `plot` parameter is set to `False` in `test_resolve_umi` and `True` in `test_filter_by_reads`, demonstrating testing with and without plot generation.
* The temporary directory is used to store any output files generated during the tests and is cleaned up after each test.
This test file ensures that the UMI resolution functionality in the Cassiopeia preprocessing pipeline works as expected, particularly in handling different cell and UMI filtering scenarios.
# birth_death_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/birth_death_simulator_test.py
Here's a documentation of the target file:
## High-level description
This file contains unit tests for the BirthDeathFitnessSimulator class in Cassiopeia. It tests various aspects of the simulator, including error handling, basic functionality, and more complex scenarios involving birth, death, and fitness mutations.
## Code Structure
The main class `BirthDeathSimulatorTest` inherits from `unittest.TestCase` and contains multiple test methods. Each method tests a specific aspect or scenario of the BirthDeathFitnessSimulator.
## Symbols
### `BirthDeathSimulatorTest`
#### Description
A test class that contains various unit tests for the BirthDeathFitnessSimulator class.
#### Internal Logic
The class sets up a character matrix and priors in the `setUp` method, which are used in some of the tests. It then defines multiple test methods, each focusing on a specific aspect of the simulator's functionality.
### `test_bad_waiting_distributions`
#### Description
Tests that the simulator raises appropriate errors when invalid waiting time distributions are provided.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with invalid parameters and asserts that TreeSimulatorError is raised in each case.
### `test_bad_stopping_conditions`
#### Description
Ensures that the simulator raises errors when invalid stopping conditions are given.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with various invalid stopping conditions and asserts that TreeSimulatorError is raised in each case.
### `test_dead_at_start`
#### Description
Tests that the simulator raises an error when all lineages die on the first event.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with parameters that would cause immediate death of all lineages and asserts that TreeSimulatorError is raised.
### `test_dead_before_end`
#### Description
Ensures that the simulator raises an error when all lineages die before the stopping condition is met.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with parameters that would cause all lineages to die before the stopping condition and asserts that TreeSimulatorError is raised.
### `test_single_lineage`
#### Description
Tests the base case where stopping conditions are met before any divisions occur.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with parameters that result in a single lineage and verifies the resulting tree structure.
### `test_constant_yule`
#### Description
Tests a small case without death and with constant waiting times.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with constant birth rates and no death, and verifies the resulting tree structure.
### `test_nonconstant_yule`
#### Description
Tests a case without death but with variable waiting times.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with variable birth rates and no death, and verifies the resulting tree structure.
### `test_nonconstant_birth_death`
#### Description
Tests a case with variable birth and death waiting times, including pruning of dead lineages and collapsing of unifurcations.
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with variable birth and death rates, and verifies the resulting tree structure, including the absence of certain nodes due to pruning and collapsing.
### `test_nonconstant_birth_death_no_unifurcation_collapsing`
#### Description
Tests a case with variable birth and death waiting times, but without collapsing unifurcations.
#### Internal Logic
Similar to the previous test, but with `collapse_unifurcations=False`, and verifies that certain nodes are present in the resulting tree.
### `test_nonconstant_birth_death_both_stopping_conditions`
#### Description
Tests a case with variable birth and death waiting times, using both stopping conditions (number of extant nodes and experiment time).
#### Internal Logic
Creates BirthDeathFitnessSimulator instances with both stopping conditions and verifies the resulting tree structure under different scenarios.
### `test_nonconstant_yule_with_predictable_fitness`
#### Description
Tests a case with birth and death with constant fitness.
#### Internal Logic
Creates a BirthDeathFitnessSimulator with predictable fitness mutations and verifies that the fitness values in the resulting tree match the expected values.
### `test_nonconstant_birth_death_with_variable_fitness`
#### Description
Tests a case with variable birth and death waiting times, as well as variable fitness evolution.
#### Internal Logic
Creates a BirthDeathFitnessSimulator with variable fitness mutations and verifies the resulting tree structure.
### `test_no_initial_birth_scale`
#### Description
Tests the simulator's behavior when initializing with a tree that doesn't have default initial birth scales.
#### Internal Logic
Creates an initial tree without birth scales, initializes a BirthDeathFitnessSimulator with this tree, and verifies that the resulting tree has the correct number of leaves and birth scales.
### `test_birth_scale`
#### Description
Tests that the birth scales are correctly maintained when initializing the simulator with an existing tree.
#### Internal Logic
Creates an initial tree using the simulator, then uses this tree to initialize another simulation, and verifies that the birth scales are correctly maintained in the final tree.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia.data.CassiopeiaTree
* cassiopeia.mixins.TreeSimulatorError
* cassiopeia.simulator.BirthDeathFitnessSimulator
* cassiopeia.data.utilities
## Error Handling
The test cases use `assertRaises` to check for expected exceptions in various scenarios, such as invalid input parameters or unexpected simulation outcomes.
## TODOs
There are no explicit TODOs in the code.
# brownian_spatial_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/brownian_spatial_simulator_test.py
## High-level description
This file contains unit tests for the `BrownianSpatialDataSimulator` class, which is part of the Cassiopeia package. The tests verify the initialization of the simulator and its ability to overlay spatial data onto a `CassiopeiaTree` object, both with and without existing cell metadata.
## Code Structure
The code defines a single test class `TestBrownianSpatialDataSimulator` that inherits from `unittest.TestCase`. It contains a `setUp` method to initialize test data and several test methods to verify different aspects of the `BrownianSpatialDataSimulator` class.
## Symbols
### TestBrownianSpatialDataSimulator
#### Description
A test class for the `BrownianSpatialDataSimulator` class, containing multiple test methods to verify its functionality.
#### Internal Logic
1. Sets up test data in the `setUp` method, including a basic tree and a tree with cell metadata.
2. Tests the initialization of the simulator with various parameters.
3. Tests the overlay of spatial data onto a basic tree and verifies the results.
4. Tests the overlay of spatial data without scaling to unit area.
5. Tests the overlay of spatial data onto a tree with existing cell metadata.
### setUp
#### Description
Initializes test data, including a basic tree and a tree with cell metadata.
#### Internal Logic
1. Creates a directed graph representing a tree topology.
2. Initializes a `CassiopeiaTree` object with the topology.
3. Sets node times for the tree.
4. Creates a DataFrame with cell metadata for some nodes.
5. Initializes another `CassiopeiaTree` object with the topology and cell metadata.
### test\_init
#### Description
Tests the initialization of the `BrownianSpatialDataSimulator` with various parameters.
#### Internal Logic
1. Attempts to initialize the simulator with invalid parameters (dimension 0 and negative diffusion coefficient) and expects `DataSimulatorError` to be raised.
2. Initializes the simulator with valid parameters and verifies the attribute values.
### test\_overlay\_data
#### Description
Tests the `overlay_data` method of the `BrownianSpatialDataSimulator` on a basic tree.
#### Internal Logic
1. Initializes the simulator with 2 dimensions and a diffusion coefficient of 1.
2. Overlays spatial data onto the basic tree.
3. Verifies the generated spatial coordinates for each node.
4. Checks that the cell metadata is correctly updated with spatial coordinates.
### test\_overlay\_data\_without\_scale
#### Description
Tests the `overlay_data` method without scaling the coordinates to unit area.
#### Internal Logic
1. Initializes the simulator with `scale_unit_area=False`.
2. Overlays spatial data onto the basic tree.
3. Verifies the generated spatial coordinates for each node.
4. Checks that the cell metadata is correctly updated with spatial coordinates.
### test\_overlay\_data\_with\_existing\_cell\_meta
#### Description
Tests the `overlay_data` method on a tree with existing cell metadata.
#### Internal Logic
1. Initializes the simulator with 2 dimensions and a diffusion coefficient of 1.
2. Overlays spatial data onto the tree with existing cell metadata.
3. Verifies the generated spatial coordinates for each node.
4. Checks that the cell metadata is correctly updated with spatial coordinates while preserving existing metadata.
## Dependencies
| Dependency | Purpose |
| :--------- | :-------------------------------------------------- |
| unittest | Provides the testing framework |
| networkx | Used for creating and manipulating graph structures |
| numpy | Used for numerical operations and testing |
| pandas | Used for handling DataFrames and testing |
| cassiopeia | The package being tested |
## Error Handling
The tests use `self.assertRaises` to verify that the `BrownianSpatialDataSimulator` raises `DataSimulatorError` when initialized with invalid parameters.
## Notes
* The tests use a fixed random seed (0) to ensure reproducibility of the generated spatial coordinates.
* The expected coordinates in the tests are hardcoded, which might make the tests brittle if the underlying implementation changes.
# cas9_lineage_tracing_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/cas9_lineage_tracing_simulator_test.py
Here's a high-level description and documentation for the target file:
## High-level description
This file contains unit tests for the Cas9-based lineage tracing data simulator in the Cassiopeia package. It tests various aspects of the simulator, including basic setup, error handling, and data generation functionality.
## Code Structure
The main class `TestCas9LineageTracingDataSimulator` contains multiple test methods that check different aspects of the `Cas9LineageTracingDataSimulator` class. The tests cover basic setup, error handling, and data generation functionality.
## Symbols
### TestCas9LineageTracingDataSimulator
#### Description
A unittest.TestCase class that contains test methods for the Cas9LineageTracingDataSimulator.
#### Internal Logic
The class sets up a basic tree structure and several instances of Cas9LineageTracingDataSimulator with different configurations in the `setUp` method. It then tests various aspects of the simulator in separate test methods.
### setUp
#### Description
Sets up the test environment by creating a basic tree structure and initializing several Cas9LineageTracingDataSimulator instances with different configurations.
### test\_basic\_setup
#### Description
Tests the basic setup of the Cas9LineageTracingDataSimulator, including the number of characters, mutation rates, and priors.
### test\_setup\_errors
#### Description
Tests various error conditions in the setup of Cas9LineageTracingDataSimulator, such as invalid input parameters.
### test\_get\_cassettes
#### Description
Tests the `get_cassettes` method of the Cas9LineageTracingDataSimulator.
### test\_introduce\_states
#### Description
Tests the `introduce_states` method of the Cas9LineageTracingDataSimulator.
### test\_silence\_cassettes
#### Description
Tests the `silence_cassettes` method of the Cas9LineageTracingDataSimulator.
### test\_collapse\_sites
#### Description
Tests the `collapse_sites` method of the Cas9LineageTracingDataSimulator.
### test\_simulator\_basic
#### Description
Tests the basic functionality of the Cas9LineageTracingDataSimulator by overlaying data on a tree and checking the resulting character matrix.
### test\_no\_collapse
#### Description
Tests the Cas9LineageTracingDataSimulator without collapsing sites.
### test\_no\_resection
#### Description
Tests the Cas9LineageTracingDataSimulator without resection.
### test\_simulator\_with\_state\_generating\_distribution
#### Description
Tests the Cas9LineageTracingDataSimulator with a state-generating distribution.
### test\_simulator\_with\_per\_character\_priors
#### Description
Tests the Cas9LineageTracingDataSimulator with per-character priors.
### test\_simulator\_with\_per\_character\_rates
#### Description
Tests the Cas9LineageTracingDataSimulator with per-character mutation rates.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia
## Error Handling
The test methods use assertions to check for expected behavior and raise exceptions for unexpected results.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# clonal_spatial_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/clonal_spatial_simulator_test.py
## High-level description
This code defines a test suite for the `ClonalSpatialDataSimulator` class, which is responsible for simulating spatial data with clonal spatial autocorrelation constraints. The tests cover the initialization of the simulator and the overlay of spatial data onto a `CassiopeiaTree` object.
## Code Structure
The main class `TestClonalSpatialDataSimulator` contains several test methods that check different aspects of the `ClonalSpatialDataSimulator` functionality. The `setUp` method initializes common objects used across multiple tests.
## Symbols
### TestClonalSpatialDataSimulator
#### Description
A test class that inherits from `unittest.TestCase` to test the `ClonalSpatialDataSimulator` class.
#### Internal Logic
1. Sets up a basic tree structure and cell metadata for testing.
2. Tests the initialization of the simulator with different parameters.
3. Tests the overlay of spatial data onto a tree without existing cell metadata.
4. Tests the overlay of spatial data onto a tree with existing cell metadata.
### setUp
#### Description
Initializes the test environment by creating a basic tree structure and cell metadata.
#### Internal Logic
1. Sets a random seed for reproducibility.
2. Creates a directed graph representing a tree topology.
3. Initializes a `CassiopeiaTree` object with the created topology.
4. Creates a DataFrame with cell metadata for some leaves of the tree.
### test\_init
#### Description
Tests the initialization of the `ClonalSpatialDataSimulator` with different parameters.
#### Internal Logic
1. Checks that initializing without parameters raises a `DataSimulatorError`.
2. Tests initialization with a 2D space, verifying the dimension and space shape.
3. Tests initialization with a 3D space, verifying the dimension and space shape.
### test\_overlay\_data
#### Description
Tests the `overlay_data` method of `ClonalSpatialDataSimulator` on a basic tree without existing cell metadata.
#### Internal Logic
1. Initializes a simulator with a 2D space.
2. Overlays spatial data onto the basic tree.
3. Checks that spatial coordinates are assigned to all nodes in the tree.
4. Verifies that the cell metadata is correctly updated with spatial coordinates for leaf nodes.
### test\_overlay\_data\_with\_space
#### Description
Tests the `overlay_data` method with a custom space parameter.
#### Internal Logic
Similar to `test_overlay_data`, but initializes the simulator with a custom space parameter.
### test\_overlay\_data\_with\_existing\_cell\_meta
#### Description
Tests the `overlay_data` method on a tree with existing cell metadata.
#### Internal Logic
1. Initializes a simulator with a 2D space.
2. Overlays spatial data onto the tree with existing cell metadata.
3. Verifies that spatial coordinates are assigned to all nodes.
4. Checks that the existing cell metadata is preserved and new spatial coordinates are added correctly.
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------------------------------------------------------------ |
| unittest | Provides the testing framework |
| networkx | Used for creating and manipulating graph structures |
| numpy | Used for numerical operations and testing array equality |
| pandas | Used for handling DataFrames (cell metadata) |
| cassiopeia | The main package being tested, provides CassiopeiaTree and ClonalSpatialDataSimulator |
## Error Handling
The test cases use `self.assertRaises` to check for expected exceptions, particularly `DataSimulatorError` when initializing the simulator incorrectly.
## Notes
* The tests use fixed random seeds to ensure reproducibility of spatial coordinate generation.
* The expected coordinate values in the tests are hardcoded, which might make the tests brittle if the underlying simulation algorithm changes.
# complete_binary_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/complete_binary_simulator_test.py
## High-level description
This file contains unit tests for the `CompleteBinarySimulator` class, which is responsible for simulating complete binary trees. The tests verify the initialization of the simulator and the structure of the simulated tree, including its nodes, edges, and branch lengths.
## Symbols
### TestCompleteBinarySimulator
#### Description
This is a test class that inherits from `unittest.TestCase`. It contains two test methods to verify the functionality of the `CompleteBinarySimulator` class.
### test\_init
#### Description
This method tests the initialization of the `CompleteBinarySimulator` class.
#### Internal Logic
1. It checks that the simulator raises a `TreeSimulatorError` when initialized without arguments.
2. It verifies that the simulator raises an error when initialized with an invalid number of cells (not a power of 2).
3. It ensures that the simulator raises an error when initialized with a depth of 0.
4. It confirms that the simulator correctly calculates the depth when initialized with a valid number of cells.
### test\_simulate\_tree
#### Description
This method tests the `simulate_tree` method of the `CompleteBinarySimulator` class.
#### Internal Logic
1. It creates a simulator with a depth of 2 and simulates a tree.
2. It verifies the correct set of nodes in the simulated tree.
3. It checks the correct set of leaf nodes.
4. It confirms the correct set of edges in the tree.
5. It validates the branch lengths (times) for each node in the tree.
## Dependencies
| Dependency | Purpose |
| :------------------- | :------------------------------------------------------------------- |
| numpy | Used for numerical operations (not explicitly used in the test file) |
| unittest | Provides the testing framework |
| cassiopeia.mixins | Imports the `TreeSimulatorError` exception |
| cassiopeia.simulator | Imports the `CompleteBinarySimulator` class |
## Error Handling
The test cases check for proper error handling in the `CompleteBinarySimulator` class by asserting that `TreeSimulatorError` is raised under specific conditions.
## References
* `CompleteBinarySimulator`: The main class being tested, imported from `cassiopeia.simulator`.
* `TreeSimulatorError`: An exception class used for error handling, imported from `cassiopeia.mixins`.
This test file ensures that the `CompleteBinarySimulator` class correctly initializes and generates complete binary trees with the expected structure and properties.
# ecdna_birth_death_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/ecdna_birth_death_simulator_test.py
Here's a documentation for the target file `test/simulator_tests/ecdna_birth_death_simulator_test.py`:
## High-level description
This file contains unit tests for the `ecDNABirthDeathSimulator` class in the Cassiopeia package. It tests various aspects of the simulator, including ecDNA splitting, lineage sampling, tree population, and cosegregation of ecDNA species.
## Code Structure
The main class `ecDNABirthDeathSimulatorTest` inherits from `unittest.TestCase` and contains several test methods. These methods test different functionalities of the `ecDNABirthDeathSimulator` class, such as ecDNA splitting, initial sample events, lineage sampling, tree population, and cosegregation.
## References
* `cassiopeia.simulator.ecDNABirthDeathSimulator.ecDNABirthDeathSimulator`
* `cassiopeia.data.CassiopeiaTree`
* `cassiopeia.data.utilities`
## Symbols
### `node_name_generator()`
#### Description
A generator function that yields unique node names for the tree.
#### Outputs
| Name | Type | Description |
| :--------- | :--- | :----------------- |
| node\_name | str | A unique node name |
### `extract_tree_statistics(tree)`
#### Description
A helper function for testing simulated trees. It extracts various statistics from the given tree.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :------------------ |
| tree | CassiopeiaTree | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :--------------- | :----------- | :--------------------------------------------- |
| times | List\[float] | Total lived time for each extant lineage |
| num\_leaves | int | Number of extant lineages |
| correct\_degrees | bool | Whether the tree has the expected node degrees |
### `ecDNABirthDeathSimulatorTest`
#### Description
The main test class containing various test methods for the `ecDNABirthDeathSimulator`.
#### Test Methods
1. `test_ecdna_splitting`: Tests the ecDNA splitting functionality.
2. `test_initial_sample_event`: Tests the initial sample event in the simulation.
3. `test_basic_sample_lineage_events`: Checks that sample\_lineage\_event behaves as expected under edge cases.
4. `test_populate_tree_from_simulation`: Tests the population of a tree from simulation results.
5. `test_basic_cosegregation`: Tests basic cosegregation of ecDNA species.
6. `test_perfect_cosegregation`: Tests perfect cosegregation of ecDNA species.
7. `test_low_capture_efficiency`: Tests the simulator with low capture efficiency.
## Dependencies
* unittest
* heapq
* typing
* queue
* networkx
* numpy
* pandas
* cassiopeia.data.CassiopeiaTree
* cassiopeia.simulator.ecDNABirthDeathSimulator
* cassiopeia.data.utilities
## Error Handling
The tests use assertions to check for expected behavior and raise `unittest.TestCase` assertion errors if the behavior doesn't match the expected outcome.
## TODOs
None explicitly mentioned in the code.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/overview
## High-level description
This directory contains unit tests for various simulators and data generators in the Cassiopeia project. The tests cover different aspects of tree simulation, spatial data simulation, lineage tracing, and leaf subsampling. These tests ensure the correct functionality of the simulators and help maintain the integrity of the Cassiopeia package.
## What does it do?
The test suite in this directory verifies the following functionalities:
1. Birth-death tree simulation
2. Brownian spatial data simulation
3. CRISPR/Cas9 lineage tracing simulation
4. Clonal spatial simulation
5. Complete binary tree simulation
6. ecDNA birth-death simulation
7. Sequential lineage tracing simulation
8. Simple fit subclone simulation
9. Spatial leaf subsampling
10. Supercellular sampling
11. Uniform leaf subsampling
Each test file contains multiple test cases that check for correct initialization, error handling, and output generation for the respective simulators and data generators.
## Entry points
The main entry points for these tests are the individual test files, each corresponding to a specific simulator or data generator. Some key files include:
1. `birth_death_simulator_test.py`: Tests for the birth-death tree simulator
2. `brownian_spatial_simulator_test.py`: Tests for the Brownian spatial data simulator
3. `cas9_lineage_tracing_simulator_test.py`: Tests for the CRISPR/Cas9 lineage tracing simulator
4. `ecdna_birth_death_simulator_test.py`: Tests for the ecDNA birth-death simulator
5. `sequential_lineage_tracing_simulator_test.py`: Tests for the sequential lineage tracing simulator
6. `spatial_leaf_subsampler_test.py`: Tests for the spatial leaf subsampler
These test files can be run individually or as part of a larger test suite to ensure the correct functionality of the Cassiopeia package.
## Dependencies
The test suite relies on the following main dependencies:
1. unittest: The Python unit testing framework
2. numpy: For numerical operations and random number generation
3. networkx: For graph operations and tree structures
4. pandas: For handling data frames and character matrices
5. cassiopeia: The main package being tested, including its various modules and classes
## Configuration
The tests do not require specific configuration files. However, they often use random seeds to ensure reproducibility of test results. For example:
```python theme={null}
np.random.seed(41)
```
This ensures that random number generation produces consistent results across test runs.
In summary, this test suite provides comprehensive coverage for the various simulators and data generators in the Cassiopeia project, ensuring their correct functionality and helping maintain the overall quality of the package.
# sequential_lineage_tracing_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/sequential_lineage_tracing_simulator_test.py
## High-level description
This file contains unit tests for the `SequentialLineageTracingDataSimulator` class in the Cassiopeia library. It tests various aspects of the simulator, including basic setup, error handling, data overlay functionality, and specific scenarios like handling missing data and multiple edits on a branch.
## Code Structure
The code defines a test class `TestSequentialLineageTracingDataSimulator` that inherits from `unittest.TestCase`. It contains several test methods, each focusing on a specific aspect of the simulator's functionality. The `setUp` method initializes common objects used across multiple tests.
## Symbols
### TestSequentialLineageTracingDataSimulator
#### Description
This class contains unit tests for the `SequentialLineageTracingDataSimulator` class.
#### Internal Logic
1. Sets up a basic tree structure and simulator instances in the `setUp` method.
2. Tests basic setup, error handling, and data overlay functionality.
3. Verifies the simulator's behavior with and without missing data.
4. Checks for correct handling of multiple edits on a branch.
### setUp
#### Description
Initializes the test environment by creating a basic tree structure and two simulator instances.
#### Internal Logic
1. Creates a directed graph representing a tree topology.
2. Sets up a `CassiopeiaTree` with the topology and node times.
3. Initializes two `SequentialLineageTracingDataSimulator` instances, one without missing data and one with missing data.
### test\_basic\_setup
#### Description
Tests the basic setup of the simulator, verifying that the initialization parameters are correctly set.
#### Internal Logic
Checks various attributes of the simulator, such as the number of characters, silencing rates, state priors, and initiation/continuation rates.
### test\_setup\_errors
#### Description
Tests error handling for invalid initialization parameters of the simulator.
#### Internal Logic
Attempts to create simulator instances with invalid parameters and checks that the appropriate `DataSimulatorError` is raised.
### test\_simulator\_basic
#### Description
Tests the basic functionality of the simulator by overlaying data on the tree and verifying the results.
#### Internal Logic
1. Overlays data on the basic tree.
2. Compares the resulting character matrix with an expected matrix.
3. Checks inheritance patterns in the tree after reconstructing ancestral characters.
### test\_simulator\_with\_missing
#### Description
Tests the simulator's behavior when dealing with missing data (i.e., when heritable and stochastic silencing rates are non-zero).
#### Internal Logic
Overlays data on the basic tree using the simulator with missing data and compares the resulting character matrix with an expected matrix containing missing values (-1).
### test\_branch\_multiple\_edits
#### Description
Tests the simulator's ability to handle multiple edits on a single branch.
#### Internal Logic
1. Creates a simple tree with a long branch.
2. Overlays data using a simulator with specific parameters.
3. Verifies that the resulting character matrix contains multiple edits along the branch.
## Dependencies
The code relies on the following external libraries and modules:
| Dependency | Purpose |
| :--------- | :-------------------------------------------------------------------- |
| unittest | Provides the testing framework |
| networkx | Used for creating and manipulating graph structures |
| numpy | Not directly used in the test file, but likely used in the simulator |
| pandas | Used for creating and comparing DataFrames |
| cassiopeia | The library being tested, including its simulator and data structures |
## Error Handling
The code tests for various error conditions in the `test_setup_errors` method, ensuring that the simulator raises appropriate `DataSimulatorError` exceptions when given invalid input parameters.
# simple_fit_subclone_simulator_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/simple_fit_subclone_simulator_test.py
## High-level description
This file contains unit tests for the SimpleFitSubcloneSimulator class, which simulates a clonal population that develops one fit subclone. The tests verify the correctness of the simulated phylogenetic tree structure and branch lengths in both deterministic and stochastic scenarios.
## Symbols
### TestSimpleFitSubcloneSimulator
#### Description
A unittest.TestCase subclass that contains test methods for the SimpleFitSubcloneSimulator class.
### test\_deterministic
#### Description
Tests the SimpleFitSubcloneSimulator with deterministic branch lengths to ensure the generated phylogeny is correct.
#### Internal Logic
1. Creates a SimpleFitSubcloneSimulator instance with specific parameters.
2. Simulates a tree using the simulator.
3. Asserts that the generated tree has the expected structure:
* Correct node names
* Correct edge connections
* Correct node times
### test\_stochastic
#### Description
Tests the SimpleFitSubcloneSimulator with stochastic branch lengths to ensure non-deterministic behavior.
#### Internal Logic
1. Sets a random seed for reproducibility.
2. Defines two functions for generating random branch lengths:
* `branch_length_neutral`: Returns an exponentially distributed value with mean 1.0
* `branch_length_fit`: Returns an exponentially distributed value with mean 0.5
3. Creates a SimpleFitSubcloneSimulator instance with the random branch length functions.
4. Simulates a tree using the simulator.
5. Extracts branch lengths for non-leaf nodes.
6. Asserts that all branch lengths are distinct, confirming non-determinism.
## Dependencies
| Dependency | Purpose |
| :---------------------------------------------- | :----------------------------------------------------- |
| numpy | Used for random number generation and array operations |
| unittest | Provides the testing framework |
| cassiopeia.simulator.SimpleFitSubcloneSimulator | The class being tested |
## Notes
* The tests cover both deterministic and stochastic scenarios, ensuring the simulator works correctly in different conditions.
* The stochastic test uses a fixed random seed to ensure reproducibility while still testing random behavior.
* The tests focus on verifying the tree structure, node naming, and branch length generation, which are key aspects of the SimpleFitSubcloneSimulator's functionality.
# spatial_leaf_subsampler_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/spatial_leaf_subsampler_test.py
Here's a detailed documentation of the `test/simulator_tests/spatial_leaf_subsampler_test.py` file:
## High-level description
This file contains unit tests for the `SpatialLeafSubsampler` class in the Cassiopeia package. It tests various functionalities of the subsampler, including initialization, parameter validation, and leaf subsampling in both 2D and 3D spaces.
## Code Structure
The main class `SpatialLeafSubsamplerTest` inherits from `unittest.TestCase` and contains multiple test methods. The `setUp` method initializes test data, including 2D and 3D trees and spaces.
## Symbols
### `SpatialLeafSubsamplerTest`
#### Description
A test class for the `SpatialLeafSubsampler` class.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Tests various aspects of the `SpatialLeafSubsampler` class through individual test methods.
### `setUp`
#### Description
Initializes test data for the test cases.
#### Internal Logic
1. Creates a balanced tree with spatial attributes.
2. Sets up 2D and 3D versions of the tree.
3. Defines 2D and 3D bounding boxes and spaces for testing.
### `test_bad_init_parameters`
#### Description
Tests error handling for invalid initialization parameters.
#### Internal Logic
Checks for `LeafSubsamplerError` in various scenarios:
1. Both ratio and number of leaves provided
2. Neither space nor bounding box provided
3. Both space and bounding box provided
4. Negative number of leaves
5. Ratio > 1
6. Merge cells without space
### `test_bad_subsample_parameters`
#### Description
Tests error handling for invalid subsampling parameters.
#### Internal Logic
Checks for `LeafSubsamplerError` in various scenarios:
1. Tree without spatial attributes
2. Incompatible space dimensions
3. Incompatible bounding box dimensions
4. Leaf coordinates outside space
5. No leaves in region
6. Number of leaves too large
### `test_bad_scale`
#### Description
Tests warning generation for potentially problematic scale issues.
#### Internal Logic
1. Creates a tree with very small spatial coordinates.
2. Checks for `LeafSubsamplerWarning` when subsampling.
### `test_subsample_3d_tree`
#### Description
Tests subsampling of a 3D tree.
#### Internal Logic
1. Tests subsampling using a bounding box.
2. Tests subsampling using a space.
3. Verifies the resulting tree structure.
### `test_subsample_2d_tree`
#### Description
Tests subsampling of a 2D tree.
#### Internal Logic
Similar to `test_subsample_3d_tree`, but for 2D trees.
### `test_keep_singular_root_edge`
#### Description
Tests the `keep_singular_root_edge` parameter.
#### Internal Logic
1. Creates a tree with a singular root edge.
2. Tests subsampling with `keep_singular_root_edge=True` and `False`.
3. Verifies the resulting tree structure in both cases.
### `test_subsample_with_ratio`
#### Description
Tests subsampling using a ratio.
#### Internal Logic
1. Sets up a subsampler with a ratio of 0.7.
2. Verifies the resulting tree structure after subsampling.
### `test_subsample_with_number`
#### Description
Tests subsampling using a specific number of leaves.
#### Internal Logic
1. Sets up a subsampler with `number_of_leaves=2`.
2. Verifies the resulting tree structure after subsampling.
### `test_merge_cells`
#### Description
Tests the cell merging functionality.
#### Internal Logic
1. Tests merging without a character matrix.
2. Tests merging with a character matrix.
3. Tests merging without collapsing duplicates.
4. Verifies the resulting tree structure and attributes after merging.
## Dependencies
* `unittest`: For creating and running unit tests.
* `networkx`: For creating and manipulating graph structures.
* `numpy`: For numerical operations and random number generation.
* `pandas`: For data manipulation and analysis.
* `cassiopeia.data.CassiopeiaTree`: The main tree structure being tested.
* `cassiopeia.mixins`: For custom errors and warnings.
* `cassiopeia.simulator.SpatialLeafSubsampler`: The class being tested.
## Error Handling
The test cases extensively check for proper error handling in the `SpatialLeafSubsampler` class, using `assertRaises` to verify that appropriate exceptions are raised for invalid inputs or operations.
## Logging
This test file does not implement any specific logging mechanisms.
## TODOs
There are no explicit TODOs in this test file.
# supercellular_sampler_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/supercellular_sampler_test.py
Here's a detailed documentation of the `test/simulator_tests/supercellular_sampler_test.py` file:
## High-level description
This file contains unit tests for the SupercellularSampler class, which is a subclass of LeafSubsampler. The tests verify the functionality of the SupercellularSampler, including its initialization, error handling, and leaf subsampling behavior.
## Code Structure
The main class `SupercellularSamplerTest` inherits from `unittest.TestCase` and contains several test methods. The `setUp` method initializes a test network and character matrix used in multiple tests.
## Symbols
### SupercellularSamplerTest
#### Description
A test class that contains unit tests for the SupercellularSampler class.
#### Internal Logic
The class sets up a test network and character matrix in the `setUp` method, which are used across multiple test cases.
### setUp
#### Description
Initializes the test environment by creating a test network and character matrix.
#### Internal Logic
* Creates a directed graph (`self.test_network`) representing a tree structure.
* Defines a character matrix (`self.character_matrix`) with sample data for testing.
### test\_bad\_parameters
#### Description
Tests the SupercellularSampler initialization with invalid parameters.
#### Internal Logic
* Attempts to create a SupercellularSampler with both ratio and number\_of\_merges specified, which should raise a LeafSubsamplerError.
* Attempts to create a SupercellularSampler with neither ratio nor number\_of\_merges specified, which should also raise a LeafSubsamplerError.
### test\_bad\_number\_of\_samples
#### Description
Tests the subsample\_leaves method with invalid inputs.
#### Internal Logic
* Creates a CassiopeiaTree with the test network and character matrix.
* Attempts to subsample leaves with invalid number\_of\_merges values (10, 0, and 2 for a tree without a character matrix), which should raise appropriate errors.
### test\_subsample\_balanced\_tree
#### Description
Tests the subsample\_leaves method with valid inputs and verifies the output.
#### Internal Logic
* Creates a CassiopeiaTree with the test network and character matrix.
* Subsamples leaves using SupercellularSampler with number\_of\_merges=2.
* Verifies the resulting character matrix and edges against expected values.
* Repeats the test with collapse\_duplicates=False and verifies the results.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia.data.CassiopeiaTree
* cassiopeia.simulator.LeafSubsampler
* cassiopeia.simulator.SupercellularSampler
* cassiopeia.data.utilities
## Error Handling
The tests check for various error conditions, including:
* Incorrect initialization parameters for SupercellularSampler
* Invalid number of merges in subsample\_leaves method
* Attempting to subsample leaves on a tree without a character matrix
## TODOs
There are no explicit TODOs in this file.
This test file ensures that the SupercellularSampler class behaves correctly under various conditions and produces expected outputs when subsampling leaves from a CassiopeiaTree.
# unifom_leaf_subsampler_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/simulator_tests/unifom_leaf_subsampler_test.py
Here's a detailed documentation of the target file:
## High-level description
This file contains unit tests for the UniformLeafSubsampler class, which is responsible for uniformly subsampling leaves from a CassiopeiaTree. The tests cover various scenarios including parameter validation, error handling, and the correctness of the subsampling process.
## Symbols
### `UniformLeafSubsamplerTest`
#### Description
A test class that inherits from `unittest.TestCase`, containing multiple test methods to verify the functionality of the UniformLeafSubsampler class.
#### Internal Logic
The class contains three main test methods:
1. `test_bad_parameters`: Tests error handling for invalid initialization parameters.
2. `test_bad_number_of_samples`: Tests error handling for invalid sample sizes.
3. `test_subsample_balanced_tree`: Tests the subsampling process on a balanced tree.
4. `test_subsample_custom_tree`: Tests the subsampling process on a custom tree.
### `test_bad_parameters`
#### Description
Tests that the UniformLeafSubsampler raises appropriate errors when initialized with invalid parameters.
#### Internal Logic
1. Attempts to create a UniformLeafSubsampler with both ratio and number\_of\_leaves specified, expecting a LeafSubsamplerError.
2. Attempts to create a UniformLeafSubsampler with neither ratio nor number\_of\_leaves specified, expecting a LeafSubsamplerError.
### `test_bad_number_of_samples`
#### Description
Tests that the UniformLeafSubsampler raises appropriate errors when attempting to subsample with invalid sample sizes.
#### Internal Logic
1. Creates a balanced tree using networkx.
2. Attempts to subsample with 0 leaves, expecting a LeafSubsamplerError.
3. Attempts to subsample with a very small ratio (0.0001), expecting a LeafSubsamplerError.
### `test_subsample_balanced_tree`
#### Description
Tests the subsampling process on a balanced tree, verifying the correctness of the resulting subsampled tree.
#### Internal Logic
1. Creates a balanced tree using networkx.
2. Performs subsampling with a fixed number of leaves (3) and verifies the resulting tree structure.
3. Performs subsampling with a ratio (0.65) and verifies the resulting tree structure.
### `test_subsample_custom_tree`
#### Description
Tests the subsampling process on a custom tree, verifying the correctness of the resulting subsampled tree, including branch lengths and node times.
#### Internal Logic
1. Creates a custom tree using networkx.
2. Performs subsampling with a ratio (0.5) and verifies the resulting tree structure, branch lengths, and node times.
3. Performs subsampling with a fixed number of leaves (6) and verifies the resulting tree structure.
## Dependencies
| Dependency | Purpose |
| :----------------------------------------- | :--------------------------------------------------------- |
| unittest | Provides the testing framework |
| networkx | Used to create and manipulate graph structures for testing |
| numpy | Used for random number generation in the tests |
| cassiopeia.data.CassiopeiaTree | The main tree structure being tested |
| cassiopeia.simulator.LeafSubsampler | Provides the LeafSubsamplerError |
| cassiopeia.simulator.UniformLeafSubsampler | The class being tested |
## Error Handling
The tests specifically check for proper error handling in the following cases:
1. Invalid initialization parameters for UniformLeafSubsampler.
2. Attempting to subsample with invalid sample sizes (0 or very small ratio).
## Notes
* The tests use fixed random seeds (np.random.seed(10) and np.random.seed(11)) to ensure reproducibility of the random subsampling process.
* The tests cover both the case of subsampling with a fixed number of leaves and with a ratio of the total leaves.
* The tests verify not only the structure of the resulting subsampled tree but also the correctness of branch lengths and node times in the custom tree test case.
# ccphylo_solver_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/ccphylo_solver_test.py
## High-level description
This file contains unit tests for the CCPhylo solver implementations in the Cassiopeia package. It compares the results of CCPhylo solvers (NJ, DNJ, HNJ, and UPGMA) against standard Neighbor Joining (NJ) and UPGMA implementations for phylogenetic tree reconstruction.
## Code Structure
The code defines a test class `TestCCPhyloSolver` that inherits from `unittest.TestCase`. It includes several test methods to compare different solver implementations and their behavior under various conditions.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :------------- | :----------------------------------- |
| triplet | tuple | A tuple containing three node labels |
| T | networkx.Graph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Find ancestors for each node in the triplet
2. Calculate the number of common ancestors for each pair
3. Determine the structure based on which pair has the most common ancestors
### `delta_fn`
#### Description
This function calculates the dissimilarity between two arrays.
#### Inputs
| Name | Type | Description |
| :------------- | :--------------------------------------- | :------------------------------ |
| x | np.array | First array |
| y | np.array | Second array |
| missing\_state | int | Value representing missing data |
| priors | Optional\[Dict\[int, Dict\[int, float]]] | Prior probabilities |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :----------------------------------------------- |
| d | int | The number of differences between the two arrays |
### `TestCCPhyloSolver`
#### Description
This is the main test class containing various test methods for CCPhylo solvers.
#### Internal Logic
1. Sets up test data and solver instances in the `setUp` method
2. Defines test methods for different solver implementations and scenarios
3. Compares the results of CCPhylo solvers with standard implementations
### Test Methods
1. `test_ccphylo_invalid_input`: Tests error handling for invalid solver inputs
2. `test_ccphylo_nj_solver`: Compares CCPhylo NJ solver with standard NJ solver
3. `test_ccphylo_dnj_solver`: Compares CCPhylo DNJ solver with standard NJ solver
4. `test_ccphylo_hnj_solver`: Compares CCPhylo HNJ solver with standard NJ solver
5. `test_ccphylo_upgma_solver`: Compares CCPhylo UPGMA solver with standard UPGMA solver
6. `test_collapse_mutationless_edges_ccphylo`: Tests collapsing mutationless edges in CCPhylo NJ solver
7. `test_duplicate_sample_ccphylo`: Tests CCPhylo NJ solver with duplicate samples
## Dependencies
| Dependency | Purpose |
| :----------- | :---------------------------------- |
| unittest | For creating and running unit tests |
| mock | For mocking objects in tests |
| configparser | For reading configuration files |
| itertools | For generating combinations |
| networkx | For working with graph structures |
| numpy | For numerical operations |
| pandas | For data manipulation |
| cassiopeia | The main package being tested |
## Configuration
The tests check for the existence of a `config.ini` file and a `ccphylo_path` setting to determine if CCPhylo is configured.
## Error Handling
The code uses `unittest.skipUnless` to skip tests if CCPhylo is not configured. It also tests for `DistanceSolverError` when invalid inputs are provided to solvers.
# dissimilarity_functions_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/dissimilarity_functions_test.py
Here's a comprehensive documentation for the target file `test/solver_tests/dissimilarity_functions_test.py`:
## High-level description
This file contains unit tests for the dissimilarity functions used in the DistanceSolver module of the Cassiopeia project. It tests various dissimilarity and similarity functions, as well as utility functions related to prior transformations and dissimilarity map saving.
## Code Structure
The main class `TestDissimilarityFunctions` inherits from `unittest.TestCase` and contains multiple test methods. Each method tests a specific function from the `dissimilarity_functions` or `solver_utilities` modules.
## Symbols
### `TestDissimilarityFunctions`
#### Description
A test class that contains unit tests for various dissimilarity functions and related utilities.
#### Internal Logic
1. Sets up test data in the `setUp` method.
2. Tests different dissimilarity and similarity functions with various inputs and configurations.
3. Tests prior transformation functions.
4. Tests the function for saving dissimilarity maps as phylip files.
### `setUp`
#### Description
Initializes test data for use in the test methods.
#### Internal Logic
1. Creates sample arrays and lists for testing.
2. Sets up prior probabilities and transforms them using different methods.
### `test_bad_prior_transformations`
#### Description
Tests that invalid prior transformations raise the expected error.
### `test_negative_log_prior_transformations`
#### Description
Verifies the correct transformation of priors using the negative log method.
### `test_inverse_prior_transformations`
#### Description
Verifies the correct transformation of priors using the inverse method.
### `test_sq_inverse_prior_transformations`
#### Description
Verifies the correct transformation of priors using the square root inverse method.
### `test_weighted_hamming_distance_identical`
#### Description
Tests the weighted Hamming distance function with identical inputs.
### `test_weighted_hamming_distance_no_priors`
#### Description
Tests the weighted Hamming distance function without using priors.
### `test_weighted_hamming_distance_priors_negative_log`
#### Description
Tests the weighted Hamming distance function using negative log transformed priors.
### `test_weighted_hamming_distance_priors_inverse`
#### Description
Tests the weighted Hamming distance function using inverse transformed priors.
### `test_weighted_hamming_distance_priors_sq_inverse`
#### Description
Tests the weighted Hamming distance function using square root inverse transformed priors.
### `test_weighted_hamming_distance_all_missing`
#### Description
Tests the weighted Hamming distance function with all missing values.
### `test_hamming_similarity_without_missing_identical`
#### Description
Tests the Hamming similarity function without missing values for identical inputs.
### `test_hamming_similarity_without_missing_no_priors`
#### Description
Tests the Hamming similarity function without missing values and without priors.
### `test_hamming_similarity_without_missing_priors`
#### Description
Tests the Hamming similarity function without missing values but with priors.
### `test_hamming_similarity_without_missing_all_missing`
#### Description
Tests the Hamming similarity function with all missing values.
### `test_hamming_similarity_normalized_identical`
#### Description
Tests the normalized Hamming similarity function with identical inputs.
### `test_hamming_similarity_normalized_no_priors`
#### Description
Tests the normalized Hamming similarity function without priors.
### `test_hamming_similarity_normalized_priors`
#### Description
Tests the normalized Hamming similarity function with priors.
### `test_hamming_similarity_normalized_all_missing`
#### Description
Tests the normalized Hamming similarity function with all missing values.
### `test_weighted_hamming_similarity_identical`
#### Description
Tests the weighted Hamming similarity function with identical inputs.
### `test_weighted_hamming_similarity_no_priors`
#### Description
Tests the weighted Hamming similarity function without priors.
### `test_weighted_hamming_similarity_priors`
#### Description
Tests the weighted Hamming similarity function with priors.
### `test_weighted_hamming_similarity_all_missing`
#### Description
Tests the weighted Hamming similarity function with all missing values.
### `test_cluster_dissimilarity`
#### Description
Tests the cluster dissimilarity function.
### `test_cluster_dissimilarity_weighted_hamming_distance_min_linkage`
#### Description
Tests the cluster dissimilarity function using weighted Hamming distance with minimum linkage.
### `test_hamming_distance`
#### Description
Tests the Hamming distance function.
### `test_hamming_distance_ignore_missing`
#### Description
Tests the Hamming distance function with the option to ignore missing values.
### `test_save_dissimilarity_as_phylip`
#### Description
Tests the function for saving dissimilarity maps as phylip files.
#### Internal Logic
1. Creates a sample dissimilarity map.
2. Mocks the file writing process.
3. Calls the function to save the dissimilarity map.
4. Verifies that the correct content is written to the file.
## Dependencies
* unittest
* unittest.mock
* numpy
* pandas
* cassiopeia.solver.dissimilarity\_functions
* cassiopeia.solver.solver\_utilities
## Error Handling
The test class uses assertions provided by `unittest.TestCase` to verify the correctness of function outputs and to check for expected exceptions.
## Performance Considerations
Some of the tested functions, particularly those involving dissimilarity calculations, may have performance implications for large datasets. However, this test file focuses on correctness rather than performance testing.
# greedy_variants_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/greedy_variants_test.py
## High-level description
This file contains unit tests for the `SpectralGreedySolver` and `MaxCutGreedySolver` classes, which are variants of the greedy solver algorithm used in the Cassiopeia project for phylogenetic tree reconstruction. The tests verify the correctness of the solvers' behavior in various scenarios, including handling of sparse data, base cases, and different weighting schemes.
## Code Structure
The main class `GreedyVariantsTest` contains multiple test methods, each focusing on a specific aspect of the solvers' functionality. The tests create sample character matrices, instantiate solver objects, and compare the resulting tree structures against expected outcomes.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet of nodes in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :--------------------------- |
| triplet | tuple | A tuple of three node labels |
| T | networkx.DiGraph | The tree graph |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Find ancestors for each node in the triplet
2. Calculate the number of common ancestors for each pair
3. Determine the structure based on which pair has the most common ancestors
### `GreedyVariantsTest`
#### Description
A test class containing multiple test methods for the `SpectralGreedySolver` and `MaxCutGreedySolver` classes.
#### Test Methods
1. `test_raises_error_on_ambiguous`: Verifies that the solver raises an error for ambiguous data.
2. `test_spectral_sparse_case`: Tests the `SpectralGreedySolver` with sparse data.
3. `test_spectral_base_case`: Tests the `SpectralGreedySolver` with a base case scenario.
4. `test_spectral_base_case_weights_almost_one`: Tests the `SpectralGreedySolver` with weights close to 1.
5. `test_maxcut_base_case`: Tests the `MaxCutGreedySolver` with a base case scenario.
6. `test_maxcut_base_case_weights_trivial`: Tests the `MaxCutGreedySolver` with trivial weights.
#### Internal Logic
Each test method follows a similar pattern:
1. Create a character matrix
2. Instantiate a solver object
3. Solve the tree
4. Compare the resulting tree structure against expected outcomes using triplet structures
## Dependencies
* unittest
* itertools
* networkx
* pandas
* cassiopeia
## Error Handling
The tests use `assertRaises` to check for expected errors and `assertEqual` to compare results against expected values.
## Notes
* The tests focus on verifying the correctness of the solvers' output rather than performance optimization.
* The use of triplet structures for comparison allows for flexible testing of tree topologies.
* The tests cover various scenarios, including sparse data and different weighting schemes, to ensure robust solver behavior.
# hybrid_solver_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/hybrid_solver_test.py
Here's a comprehensive documentation for the target file `test/solver_tests/hybrid_solver_test.py`:
## High-level description
This file contains unit tests for the HybridSolver class in Cassiopeia's solver module. It tests various aspects of the hybrid solver, including its constructor, cutoff assessment, top-down splitting, and full solving capabilities for different scenarios (small trees, large trees, trees with missing data).
## Code Structure
The main class `TestHybridSolver` inherits from `unittest.TestCase` and contains multiple test methods. It sets up test data in the `setUp` method and cleans up in the `tearDown` method. The tests cover different aspects of the HybridSolver's functionality, including its initialization, cutoff assessment, top-down splitting, and full solving capabilities.
## Symbols
### TestHybridSolver
#### Description
A test class that inherits from `unittest.TestCase` and contains multiple test methods for the HybridSolver class.
#### Internal Logic
1. Sets up test data in `setUp` method
2. Defines various test methods to check different aspects of HybridSolver
3. Cleans up test files in `tearDown` method
### find\_triplet\_structure
#### Description
A helper function that finds the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :-------------------------- |
| triplet | tuple | A tuple of three node names |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Finds ancestors for each node in the triplet
2. Compares the number of common ancestors between pairs
3. Returns the structure based on which pair has the most common ancestors
### TestHybridSolver.setUp
#### Description
Sets up the test environment by creating test data and initializing HybridSolver instances.
### TestHybridSolver.test\_constructor
#### Description
Tests the constructor of the HybridSolver class.
### TestHybridSolver.test\_cutoff
#### Description
Tests the cutoff assessment functionality of the HybridSolver.
### TestHybridSolver.test\_top\_down\_split\_manual
#### Description
Manually tests the top-down splitting functionality of the HybridSolver.
### TestHybridSolver.test\_apply\_top\_solver\_small
#### Description
Tests the application of the top solver on a small dataset.
### TestHybridSolver.test\_apply\_top\_solver\_large
#### Description
Tests the application of the top solver on a large dataset.
### TestHybridSolver.test\_apply\_top\_solver\_missing
#### Description
Tests the application of the top solver on a dataset with missing data.
### TestHybridSolver.test\_full\_hybrid
#### Description
Tests the full hybrid solving process on a small dataset.
### TestHybridSolver.test\_full\_hybrid\_single\_thread
#### Description
Tests the full hybrid solving process using a single thread.
### TestHybridSolver.test\_full\_hybrid\_large
#### Description
Tests the full hybrid solving process on a large dataset.
### TestHybridSolver.test\_full\_hybrid\_maxcut
#### Description
Tests the full hybrid solving process using the MaxCut greedy solver.
### TestHybridSolver.test\_full\_hybrid\_missing
#### Description
Tests the full hybrid solving process on a dataset with missing data.
### TestHybridSolver.test\_greedy\_over\_greedy\_maxcut\_missing
#### Description
Tests the hybrid solver using a greedy solver over a greedy MaxCut solver on a dataset with missing data.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------- |
| unittest | Provides testing framework |
| itertools | Used for generating combinations |
| networkx | Used for graph operations |
| pandas | Used for data manipulation |
| pathlib | Used for file path operations |
| cassiopeia | The main package being tested |
| gurobipy | Optional dependency for ILP solving |
## Error Handling
The tests use assertions provided by the `unittest` framework to check for expected behaviors and results. Some tests are skipped if Gurobi is not installed using the `@unittest.skipUnless` decorator.
## Logging
The tests create log files during execution, which are cleaned up in the `tearDown` method.
## TODOs
None explicitly mentioned in the code.
This documentation provides a comprehensive overview of the test file for the HybridSolver in Cassiopeia. It covers the main test class, its methods, and the helper function used in the tests. The documentation also includes information about dependencies, error handling, and logging practices used in the test suite.
# ilp_solver_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/ilp_solver_test.py
Here's a comprehensive documentation for the `test/solver_tests/ilp_solver_test.py` file:
## High-level description
This file contains unit tests for the ILPSolver class in Cassiopeia's solver module. It tests various aspects of the ILP (Integer Linear Programming) solver, including its ability to handle perfect phylogenies, duplicates, missing data, and edge cases.
## Code Structure
The main class `TestILPSolver` inherits from `unittest.TestCase` and contains multiple test methods. It sets up test data in the `setUp` method and defines helper functions for specific test scenarios. The tests cover different aspects of the ILPSolver, from basic functionality to edge cases and error handling.
## Symbols
### TestILPSolver
#### Description
A test class that inherits from `unittest.TestCase`, containing various test methods for the ILPSolver.
#### Internal Logic
1. Sets up test data in `setUp` method.
2. Defines helper functions like `find_triplet_structure`.
3. Implements multiple test methods to cover different scenarios and functionalities of the ILPSolver.
### find\_triplet\_structure
#### Description
A helper function that determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :--------------------- |
| triplet | tuple | A tuple of three nodes |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
### setUp
#### Description
Sets up the test environment by creating test data and initializing the ILPSolver.
### test\_raises\_error\_on\_ambiguous
#### Description
Tests if the ILPSolver raises an error when given ambiguous data.
### test\_get\_lca\_cython
#### Description
Tests the functionality of the get\_lca\_characters\_cython function.
### test\_cython\_hamming\_dist
#### Description
Tests the Cython implementation of the Hamming distance function.
### test\_single\_sample\_ilp
#### Description
Tests the ILPSolver with a single sample input.
### test\_basic\_ilp\_constructor
#### Description
Tests the basic constructor of the ILPSolver.
### test\_get\_layer\_for\_potential\_graph
#### Description
Tests the functionality of inferring a layer of the potential graph.
### test\_simple\_potential\_graph\_inference
#### Description
Tests the inference of a simple potential graph.
### test\_post\_process\_steiner\_solution
#### Description
Tests the post-processing of a Steiner tree solution.
### test\_append\_sample\_nodes\_and\_remove\_spurious\_leaves
#### Description
Tests the functionality of appending sample nodes and removing spurious leaves.
### test\_ilp\_solver\_perfect\_phylogeny
#### Description
Tests the ILPSolver with a perfect phylogeny scenario.
### test\_potential\_graph\_inference\_with\_duplicates
#### Description
Tests the potential graph inference with duplicate samples.
### test\_ilp\_solver\_with\_duplicates
#### Description
Tests the ILPSolver with duplicate samples.
### test\_ilp\_solver\_missing\_data
#### Description
Tests the ILPSolver with missing data.
### test\_ilp\_throws\_error\_when\_potential\_graph\_is\_not\_found
#### Description
Tests if the ILPSolver throws an error when a potential graph cannot be found.
## Dependencies
| Dependency | Purpose |
| :--------- | :----------------------------------- |
| unittest | Provides testing framework |
| networkx | Used for graph operations |
| numpy | Used for numerical operations |
| pandas | Used for data manipulation |
| cassiopeia | The main package being tested |
| gurobipy | Used for ILP optimization (optional) |
## Error Handling
The tests include various assertions to check for expected behavior and raise appropriate errors. The `ILPSolverError` is used to handle specific errors related to the ILPSolver.
## Performance Considerations
Some tests may be skipped if Gurobi is not installed, as indicated by the `@unittest.skipUnless(GUROBI_INSTALLED, "Gurobi installation not found.")` decorator.
## TODOs
There are no explicit TODOs in the code.
This documentation provides a comprehensive overview of the `ilp_solver_test.py` file, covering its purpose, structure, and key components. It should help developers understand the testing suite for the ILPSolver in Cassiopeia.
# maxcut_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/maxcut_test.py
Here's a detailed documentation of the target file `test/solver_tests/maxcut_test.py`:
## High-level description
This file contains unit tests for the MaxCutSolver class, which is part of the Cassiopeia lineage reconstruction framework. The tests verify the functionality of various components of the MaxCutSolver, including graph construction, cut evaluation, and the overall solving process.
## Code Structure
The main class `MaxCutSolverTest` inherits from `unittest.TestCase` and contains several test methods. It uses sample character matrices and trees to test different aspects of the MaxCutSolver. The file also includes a helper function `find_triplet_structure` used in comparing tree topologies.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :--------------------------- |
| triplet | tuple | A tuple of three node labels |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Find ancestors for each node in the triplet.
2. Compare the number of common ancestors between pairs.
3. Return the structure based on which pair has the most common ancestors.
### `MaxCutSolverTest`
#### Description
This is the main test class containing various unit tests for the MaxCutSolver.
#### Methods
##### `setUp`
###### Description
Initializes test data and objects for use in the test methods.
###### Internal Logic
1. Creates sample character matrices.
2. Initializes CassiopeiaTrees and a MaxCutSolver.
3. Computes mutation frequencies.
##### `test_check_if_cut`
###### Description
Tests the `check_if_cut` function from graph\_utilities.
##### `test_evaluate_cut`
###### Description
Tests the `evaluate_cut` method of MaxCutSolver.
##### `test_graph_construction`
###### Description
Tests the construction of the connectivity graph.
##### `test_graph_construction_weights`
###### Description
Tests the construction of the connectivity graph with custom weights.
##### `test_hill_climb`
###### Description
Tests the hill-climbing procedure for improving cuts.
##### `test_polytomy_base_case`
###### Description
Tests the solver's handling of polytomies (unresolved relationships).
##### `test_simple_base_case`
###### Description
Tests the solver on a simple case without priors.
##### `test_simple_base_case_priors`
###### Description
Tests the solver on a simple case with priors.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------------- |
| unittest | Provides testing framework |
| itertools | Used for combinations in triplet analysis |
| networkx | Used for graph operations |
| pandas | Used for data manipulation |
| cassiopeia | The main package being tested |
## Error Handling
The tests use assertions provided by the unittest framework to check for expected outcomes. No explicit error handling is implemented in this test file.
## Performance Considerations
The tests involve graph operations and tree comparisons, which can be computationally expensive for large datasets. However, as these are unit tests with small sample data, performance is not a significant concern.
This documentation provides a comprehensive overview of the `maxcut_test.py` file, explaining its purpose, structure, and key components. The file is crucial for ensuring the correct functionality of the MaxCutSolver in the Cassiopeia framework.
# neighborjoining_solver_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/neighborjoining_solver_test.py
Here's a comprehensive documentation of the `test/solver_tests/neighborjoining_solver_test.py` file:
## High-level description
This file contains unit tests for the NeighborJoiningSolver in Cassiopeia's solver module. It tests various aspects of the neighbor-joining algorithm implementation, including basic functionality, error correction, and handling of different input scenarios.
## Code Structure
The main class `TestNeighborJoiningSolver` inherits from `unittest.TestCase` and contains multiple test methods. It uses mock objects and sample data to test different aspects of the NeighborJoiningSolver. The file also includes helper functions for creating test data and analyzing tree structures.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet of nodes in a tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :-------------------------- |
| triplet | tuple | A tuple of three node names |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
It compares the number of common ancestors between pairs of nodes to determine the closest pair.
### `delta_fn`
#### Description
A dissimilarity function used by the solvers to compute distances between sequences.
#### Inputs
| Name | Type | Description |
| :------------- | :--------------------------------------- | :------------------------------ |
| x | np.array | First sequence |
| y | np.array | Second sequence |
| missing\_state | int | Value representing missing data |
| priors | Optional\[Dict\[int, Dict\[int, float]]] | Prior probabilities |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :---------------------------------------------- |
| d | int | The number of differences between the sequences |
### `TestNeighborJoiningSolver`
#### Description
The main test class containing all the unit tests for the NeighborJoiningSolver.
#### Internal Logic
The `setUp` method initializes various test scenarios, including:
1. General Neighbor Joining
2. Lineage Tracing Neighbor Joining
3. Character Matrix with Duplicates
4. Neighbor Joining with modified hamming dissimilarity
Each test method focuses on a specific aspect of the NeighborJoiningSolver:
1. `test_constructor`: Tests the constructor and error handling
2. `test_compute_q`: Tests the Q-matrix computation
3. `test_find_cherry`: Tests the cherry-picking step
4. `test_update_dissimilarity_map`: Tests updating the dissimilarity map
5. `test_basic_solver`: Tests the basic solver functionality
6. `test_nj_solver_weights`: Tests the solver with weighted distances
7. `test_pp_solver`: Tests the solver with a different dissimilarity function
8. `test_duplicate_sample_neighbor_joining`: Tests handling of duplicate samples
9. `test_setup_root_finder_missing_dissimilarity_map`: Tests root finding with missing dissimilarity map
10. `test_setup_root_finder_existing_dissimilarity_map`: Tests root finding with existing dissimilarity map
## Dependencies
| Dependency | Purpose |
| :------------ | :---------------------------------- |
| unittest | For creating and running unit tests |
| typing | For type hinting |
| unittest.mock | For creating mock objects |
| itertools | For generating combinations |
| networkx | For graph operations |
| numpy | For numerical operations |
| pandas | For data manipulation |
| cassiopeia | The main package being tested |
## Error Handling
The tests use `assertRaises` to check if the solver raises appropriate exceptions in error cases.
## Performance Considerations
The tests don't explicitly address performance, but they do test the correctness of the algorithm on various input sizes and scenarios.
This documentation provides a comprehensive overview of the test file, its structure, and the various aspects of the NeighborJoiningSolver that it tests. The file ensures that the solver works correctly for different input scenarios and handles edge cases appropriately.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/overview
## High-level description
This directory contains unit tests for various solver algorithms and utility functions used in the Cassiopeia package for phylogenetic tree reconstruction. The tests cover a wide range of solvers, including Neighbor Joining, ILP, Greedy, Spectral, and UPGMA solvers, as well as dissimilarity functions and other utilities.
## What does it do?
The test suite in this directory verifies the correctness and robustness of different phylogenetic tree reconstruction algorithms implemented in Cassiopeia. It tests various aspects of these solvers, including:
1. Tree construction from character matrices
2. Handling of missing data and ambiguous states
3. Computation of dissimilarity/similarity measures
4. Graph construction and manipulation
5. Specific algorithm steps (e.g., cherry picking in Neighbor Joining)
6. Integration with different types of input data and configurations
These tests ensure that the solvers produce accurate phylogenetic trees under various scenarios and input conditions, maintaining the reliability of the Cassiopeia package for lineage reconstruction tasks.
## Entry points
The main entry points for this test suite are the individual test files, each focusing on a specific solver or utility:
1. `ccphylo_solver_test.py`: Tests CCPhylo solver implementations
2. `dissimilarity_functions_test.py`: Tests dissimilarity and similarity functions
3. `greedy_variants_test.py`: Tests variants of the greedy solver algorithm
4. `hybrid_solver_test.py`: Tests the hybrid solver combining multiple algorithms
5. `ilp_solver_test.py`: Tests the Integer Linear Programming solver
6. `maxcut_test.py`: Tests the MaxCut solver
7. `neighborjoining_solver_test.py`: Tests the Neighbor Joining solver
8. `percolation_test.py`: Tests the Percolation solver
9. `sharedmutationjoiner_test.py`: Tests the Shared Mutation Joining solver
10. `snj_solver_test.py`: Tests the Spectral Neighbor Joining solver
11. `spectral_test.py`: Tests the Spectral solver
12. `upgma_test.py`: Tests the UPGMA solver
13. `vanillagreedy_test.py`: Tests the Vanilla Greedy solver
Each test file contains a main test class that inherits from `unittest.TestCase` and includes multiple test methods covering different aspects of the respective solver or utility.
## Key Files
While all test files are important, some key files that cover core functionality include:
1. `neighborjoining_solver_test.py`: Tests the widely-used Neighbor Joining algorithm
2. `ilp_solver_test.py`: Tests the Integer Linear Programming solver, which can provide optimal solutions for smaller datasets
3. `dissimilarity_functions_test.py`: Tests critical dissimilarity and similarity functions used across multiple solvers
4. `hybrid_solver_test.py`: Tests the hybrid solver that combines multiple algorithms, representing a more advanced approach
## Dependencies
The test suite relies on several external libraries and frameworks:
1. unittest: Provides the testing framework
2. networkx: Used for graph operations and tree representation
3. numpy: Used for numerical operations and array manipulations
4. pandas: Used for data manipulation and character matrix representation
5. scipy: Used for scientific computing and optimization
6. mock: Used for creating mock objects in tests
7. gurobipy (optional): Used for ILP solving in some tests
The tests also depend on various modules from the Cassiopeia package itself.
## Configuration
Some tests may require specific configurations:
1. CCPhylo tests check for the existence of a `config.ini` file and a `ccphylo_path` setting.
2. ILP solver tests may be skipped if Gurobi is not installed.
The tests use various helper functions, such as `find_triplet_structure`, to assist in verifying tree structures and comparing results across different solvers.
In summary, this test suite provides comprehensive coverage of the phylogenetic tree reconstruction algorithms implemented in Cassiopeia, ensuring their correctness and reliability across a wide range of scenarios and input conditions.
# percolation_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/percolation_test.py
Here's a detailed documentation of the target file `test/solver_tests/percolation_test.py`:
## High-level description
This file contains unit tests for the PercolationSolver in Cassiopeia's solver module. It tests the solver's ability to construct phylogenetic trees from character matrices using different joining solvers and dissimilarity functions.
## Code Structure
The main class `PercolationSolverTest` inherits from `unittest.TestCase` and contains several test methods. These methods create character matrices, solve them using different configurations of the PercolationSolver, and compare the resulting trees to expected structures.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet of nodes in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :--------------------------- |
| triplet | tuple | A tuple of three node labels |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Find ancestors for each node in the triplet.
2. Count common ancestors between pairs of nodes.
3. Determine the structure based on which pair has the most common ancestors.
### `neg_hamming_similarity_without_missing`
#### Description
This function computes the negative Hamming similarity between two sequences, ignoring missing data.
#### Inputs
| Name | Type | Description |
| :------------------------ | :--------------------------------------- | :---------------------------- |
| s1 | np.array | First sequence |
| s2 | np.array | Second sequence |
| missing\_state\_indicator | int | Value indicating missing data |
| weights | Optional\[Dict\[int, Dict\[int, float]]] | Optional weights for states |
#### Outputs
| Name | Type | Description |
| :--------- | :---- | :-------------------------- |
| similarity | float | Negative Hamming similarity |
### `class PercolationSolverTest(unittest.TestCase)`
#### Description
This class contains unit tests for the PercolationSolver.
#### Test Methods
##### `test_NJ_negative_similarity`
Tests the PercolationSolver using NeighborJoiningSolver with negative Hamming similarity.
##### `test_NJ_weighted_hamming_distance`
Tests the PercolationSolver using NeighborJoiningSolver with weighted Hamming distance.
##### `test_Greedy`
Tests the PercolationSolver using VanillaGreedySolver.
##### `test_priors_case`
Tests the PercolationSolver with prior probabilities for mutations.
#### Internal Logic
For each test:
1. Create a character matrix.
2. Initialize a CassiopeiaTree with the character matrix.
3. Set up a joining solver (NeighborJoiningSolver or VanillaGreedySolver).
4. Create a PercolationSolver with the joining solver.
5. Solve the tree.
6. Compare the resulting tree structure with the expected structure using triplet comparisons.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------- |
| unittest | For creating and running unit tests |
| itertools | For generating combinations |
| networkx | For graph operations |
| numba | For JIT compilation of functions |
| numpy | For numerical operations |
| pandas | For data manipulation |
| cassiopeia | The main package being tested |
## Error Handling
The tests use assertions to check if the resulting tree structures match the expected structures. If any assertion fails, the test will raise an AssertionError.
## Logging
This file does not implement any explicit logging. However, the unittest framework will log test results.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# sharedmutationjoiner_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/sharedmutationjoiner_test.py
Here's a detailed documentation of the target file `test/solver_tests/sharedmutationjoiner_test.py`:
## High-level description
This file contains unit tests for the SharedMutationJoiningSolver in Cassiopeia's solver module. It tests various aspects of the solver, including initialization, cherry finding, similarity map updating, and tree solving for different scenarios.
## Code Structure
The main class `TestSharedMutationJoiningSolver` inherits from `unittest.TestCase` and contains multiple test methods. The `setUp` method initializes test data and solver instances used across different tests.
## Symbols
### `find_triplet_structure`
#### Description
A helper function that determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :-------------------------- |
| triplet | tuple | A tuple of three node names |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("-", "ab", "ac", or "bc") |
#### Internal Logic
1. Find ancestors for each node in the triplet
2. Count common ancestors between pairs
3. Determine the structure based on which pair has the most common ancestors
### `TestSharedMutationJoiningSolver`
#### Description
The main test class containing various test methods for the SharedMutationJoiningSolver.
#### Internal Logic
* `setUp`: Initializes test data and solver instances
* Test methods: Each method tests a specific aspect of the solver
### `TestSharedMutationJoiningSolver.setUp`
#### Description
Initializes test data and solver instances used across different tests.
#### Internal Logic
1. Creates character matrices and dissimilarity maps for different test scenarios
2. Initializes CassiopeiaTrees with the test data
3. Creates SharedMutationJoiningSolver instances with different configurations
### `TestSharedMutationJoiningSolver.test_init`
#### Description
Tests the initialization of the SharedMutationJoiningSolver.
#### Internal Logic
1. Checks if the similarity function is numbaized when possible
2. Checks if a warning is raised when the similarity function can't be numbaized
### `TestSharedMutationJoiningSolver.test_find_cherry`
#### Description
Tests the `find_cherry` method of the solver.
#### Internal Logic
1. Calls `find_cherry` on a test similarity map
2. Checks if the returned cherry is correct
### `TestSharedMutationJoiningSolver.test_create_similarity_map`
#### Description
Tests the creation of a similarity map with weighted priors.
#### Internal Logic
1. Creates a similarity map using test data and priors
2. Checks if the similarity values are correct
### `TestSharedMutationJoiningSolver.test_update_similarity_map_and_character_matrix`
#### Description
Tests the updating of the similarity map and character matrix after joining nodes.
#### Internal Logic
1. Updates the similarity map and character matrix for different joining scenarios
2. Checks if the updated maps and matrices are correct
### `TestSharedMutationJoiningSolver.test_basic_solver`
#### Description
Tests the basic functionality of the solver on a simple tree.
#### Internal Logic
1. Solves a basic tree
2. Checks if the original data is unaltered
3. Verifies the structure of the solved tree
### `TestSharedMutationJoiningSolver.test_solver_no_numba`
#### Description
Tests the solver functionality without Numba optimization.
#### Internal Logic
Similar to `test_basic_solver`, but uses a solver instance without Numba optimization.
### `TestSharedMutationJoiningSolver.test_smj_solver_weights`
#### Description
Tests the solver with weighted priors.
#### Internal Logic
1. Solves trees with weighted priors
2. Checks the structure of the solved trees
### `TestSharedMutationJoiningSolver.test_pp_solver`
#### Description
Tests the solver on a perfect phylogeny scenario.
#### Internal Logic
1. Solves a perfect phylogeny tree
2. Checks the structure of the solved tree
3. Tests with collapsing mutationless edges
### `TestSharedMutationJoiningSolver.test_duplicate`
#### Description
Tests the solver's behavior with duplicate and missing data.
#### Internal Logic
1. Solves a tree with duplicate and missing data
2. Checks the structure of the solved tree
## Dependencies
* unittest
* functools (partial)
* typing (Dict)
* itertools
* networkx (nx)
* numba
* numpy (np)
* pandas (pd)
* scipy
* cassiopeia.data (CassiopeiaTree, utilities)
* cassiopeia.solver (SharedMutationJoiningSolver, dissimilarity\_functions, solver\_utilities)
## Error Handling
The test methods use various assertions to check if the solver behaves correctly. If any assertion fails, the test will raise an AssertionError.
# snj_solver_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/snj_solver_test.py
Here's a detailed explanation of the `test/solver_tests/snj_solver_test.py` file:
## High-level description
This file contains unit tests for the SpectralNeighborJoiningSolver in Cassiopeia's solver module. It tests various aspects of the solver, including its constructor, core functionality, and integration with different types of phylogenetic trees.
## Code Structure
The main class `TestSpectralNeighborJoiningSolver` inherits from `unittest.TestCase` and contains multiple test methods. The `setUp` method initializes common test data and objects used across different tests. Helper functions like `find_triplet_structure` and `assertTripletCorrectness` are defined outside the class to assist in verifying tree structures.
## Symbols
### TestSpectralNeighborJoiningSolver
#### Description
This is the main test class that contains all the unit tests for the SpectralNeighborJoiningSolver.
#### Internal Logic
1. Initializes test data in the `setUp` method.
2. Defines various test methods to check different aspects of the solver.
3. Uses helper functions to verify tree structures.
### find\_triplet\_structure
#### Description
A helper function that identifies the two nodes with the most similar ancestry in a triplet of nodes.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :---------------------------- |
| triplet | tuple | A tuple of three node names |
| T | networkx.DiGraph | The tree containing the nodes |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
### assertTripletCorrectness
#### Description
A helper method to check if two trees are isomorphic by comparing their triplet structures.
#### Inputs
| Name | Type | Description |
| :------------- | :--------------- | :-------------------------------------- |
| self | TestCase | The test case instance |
| nodes | List\[str] | List of leaf nodes to get triplets from |
| expected\_tree | networkx.DiGraph | The expected tree structure |
| observed\_tree | networkx.DiGraph | The observed tree structure |
## Test Methods
### test\_constructor
Tests the constructor of the SpectralNeighborJoiningSolver and checks for proper initialization and error handling.
### test\_compute\_svd2\_pairwise
Tests the `_compute_svd2` method for pairwise comparisons of subsets.
### test\_compute\_svd2\_N3
Tests the `_compute_svd2` method when there are only 3 subsets left.
### test\_update\_dissimilarity\_map\_base
Tests the `update_dissimilarity_map` method for updating the lambda matrix.
### test\_update\_dissimilarity\_map\_N3
Tests the `update_dissimilarity_map` method when there are only 3 subsets left.
### test\_get\_dissimilarity\_map
Tests the `get_dissimilarity_map` method for outputting a lambda matrix.
### test\_find\_cherry
Tests the `find_cherry` method for identifying the closest pair of nodes.
### test\_basic\_solver
Tests the basic functionality of the solver on a root-specified input tree.
### test\_snj\_solver\_weights
Tests the solver with a perfect phylogenetic tree and priors.
### test\_pp\_solver
Integration test for a Perfect Phylogenetic tree.
### test\_duplicate\_sample
Tests the solving of a tree with duplicate leaves.
### test\_integration1
Integration test on a 7-leaf tree.
### test\_integration2
Integration test on a 10-leaf tree.
### test\_setup\_root\_finder\_missing\_dissimilarity\_map
Checks that the root is still set despite missing dissimilarity map.
### test\_setup\_root\_finder\_existing\_dissimilarity\_map
Checks that the root is still set with an existing dissimilarity map.
## Dependencies
The test file imports and uses several modules and classes:
* Standard Python libraries: typing, itertools, unittest, mock
* Third-party libraries: networkx, numpy, pandas
* Cassiopeia modules: cassiopeia as cas
## Error Handling
The tests use various assertions to check for expected behavior and raise exceptions when unexpected results occur.
This test suite provides comprehensive coverage of the SpectralNeighborJoiningSolver's functionality, ensuring its correctness and robustness across different scenarios and tree structures.
# spectral_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/spectral_test.py
Here's a detailed documentation of the target file `test/solver_tests/spectral_test.py`:
## High-level description
This file contains unit tests for the SpectralSolver class and related utility functions in the Cassiopeia package. It tests various aspects of the spectral solver algorithm, including similarity calculations, graph construction, and the overall solving process.
## Code Structure
The main class `SpectralSolverTest` inherits from `unittest.TestCase` and contains multiple test methods. These methods test different components of the SpectralSolver, such as similarity calculations, graph construction, and the complete solving process. The file also includes helper functions like `find_triplet_structure` to assist in verifying the correctness of the generated trees.
## Symbols
### `find_triplet_structure`
#### Description
This function determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :--------------------------- |
| triplet | tuple | A tuple of three node labels |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Find ancestors for each node in the triplet.
2. Count common ancestors between pairs of nodes.
3. Determine the structure based on which pair has the most common ancestors.
### `SpectralSolverTest`
#### Description
This class contains unit tests for the SpectralSolver and related utilities.
#### Test Methods
1. `test_similarity`
* Tests the hamming similarity calculation without missing data.
2. `test_similarity_weighted`
* Tests the weighted hamming similarity calculation.
3. `test_graph_construction`
* Tests the construction of the similarity graph.
4. `test_graph_construction_weighted`
* Tests the construction of the weighted similarity graph.
5. `test_hill_climb`
* Tests the hill-climbing procedure for improving the spectral cut.
6. `test_simple_base_case`
* Tests the complete solving process on a simple character matrix.
7. `test_simple_base_case_string`
* Tests the solving process with string-based cell barcodes.
8. `test_simple_base_case2`
* Tests another simple base case for the solver.
9. `test_simple_base_case2_priors`
* Tests the solver with prior probabilities for mutations.
## Dependencies
| Dependency | Purpose |
| :--------- | :------------------------------- |
| unittest | Provides the testing framework |
| itertools | Used for generating combinations |
| networkx | Used for graph operations |
| pandas | Used for data manipulation |
| cassiopeia | The main package being tested |
## Error Handling
The tests use assertions provided by the `unittest` framework to check for correctness. No explicit error handling is implemented in this test file.
## Logging
No explicit logging is implemented in this test file. The `unittest` framework handles test result reporting.
## Notes
* The tests cover various aspects of the SpectralSolver, including similarity calculations, graph construction, and the complete solving process.
* Some tests use predefined character matrices and expected tree structures to verify the correctness of the solver.
* The tests include cases with and without weighted similarities and priors.
* The `find_triplet_structure` function is used to compare the structure of the generated trees with expected structures.
This test file is crucial for ensuring the correctness and reliability of the SpectralSolver implementation in the Cassiopeia package.
# upgma_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/upgma_test.py
Here's a detailed documentation of the `test/solver_tests/upgma_test.py` file:
## High-level description
This file contains unit tests for the UPGMASolver class in Cassiopeia's solver module. It tests various aspects of the UPGMA (Unweighted Pair Group Method with Arithmetic Mean) algorithm implementation, including basic functionality, handling of lineage tracing data, and dealing with duplicates and missing data.
## Code Structure
The main class `TestUPGMASolver` inherits from `unittest.TestCase` and contains multiple test methods. The `setUp` method initializes various test scenarios, and each test method focuses on a specific aspect of the UPGMASolver.
## Symbols
### TestUPGMASolver
#### Description
A test class for the UPGMASolver, containing multiple test methods to verify the correctness of the UPGMA algorithm implementation.
#### Internal Logic
The class sets up various test scenarios in the `setUp` method and then tests different aspects of the UPGMASolver in separate test methods.
### setUp
#### Description
Initializes test data and solver instances for various scenarios.
#### Internal Logic
1. Sets up a basic character matrix and dissimilarity map
2. Creates a basic CassiopeiaTree
3. Initializes UPGMASolver instances
4. Sets up additional test scenarios for lineage tracing and handling duplicates/missing data
### test\_constructor
#### Description
Tests the initialization of the UPGMASolver.
#### Internal Logic
Checks that the dissimilarity function and dissimilarity map are properly set.
### test\_find\_cherry
#### Description
Tests the `find_cherry` method of UPGMASolver.
#### Internal Logic
1. Calls `find_cherry` on the basic dissimilarity map
2. Verifies that the returned cherry is either ("a", "b") or ("b", "a")
### test\_update\_dissimilarity\_map
#### Description
Tests the `update_dissimilarity_map` method of UPGMASolver.
#### Internal Logic
1. Finds a cherry in the basic dissimilarity map
2. Updates the dissimilarity map with the found cherry
3. Verifies the updated dissimilarity map against expected values
4. Repeats the process for a second update
### test\_basic\_solver
#### Description
Tests the basic functionality of the UPGMASolver.
#### Internal Logic
1. Solves the basic tree using UPGMASolver
2. Checks that all leaves exist in the tree
3. Verifies the number of edges in the tree
4. Compares the tree structure with an expected tree structure
5. Compares tree distances between leaves
### test\_upgma\_solver\_weights
#### Description
Tests the UPGMASolver with weighted Hamming distance and prior probabilities.
#### Internal Logic
1. Solves the tree with priors using UPGMASolver
2. Verifies the dissimilarity between specific nodes
3. Compares the resulting tree structure with an expected structure
4. Repeats the test with collapsing mutationless edges
### test\_pp\_solver
#### Description
Tests the UPGMASolver with a lineage tracing scenario.
#### Internal Logic
1. Solves the lineage tracing tree using UPGMASolver
2. Verifies the dissimilarity between specific nodes
3. Compares the resulting tree structure with an expected structure
### test\_duplicate
#### Description
Tests the UPGMASolver with a scenario containing duplicates and missing data.
#### Internal Logic
1. Solves the tree with duplicates and missing data using UPGMASolver
2. Verifies the dissimilarity between specific nodes
3. Compares the resulting tree structure with an expected structure
## Dependencies
* unittest
* typing
* itertools
* networkx
* numpy
* pandas
* cassiopeia.data.CassiopeiaTree
* cassiopeia.solver.UPGMASolver
* cassiopeia.solver.dissimilarity\_functions
## Helper Functions
### find\_triplet\_structure
#### Description
Determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :-------------------------- |
| triplet | tuple | A tuple of three node names |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("ab", "ac", "bc", or "-") |
#### Internal Logic
1. Finds the ancestors of each node in the triplet
2. Compares the number of common ancestors between pairs
3. Returns the structure based on which pair has the most common ancestors
This file provides a comprehensive test suite for the UPGMASolver, covering various scenarios and edge cases to ensure the correctness of the UPGMA algorithm implementation in Cassiopeia.
# vanillagreedy_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/solver_tests/vanillagreedy_test.py
## High-level description
This file contains unit tests for the VanillaGreedySolver class, which is part of the Cassiopeia package for phylogenetic tree reconstruction. The tests cover various aspects of the solver, including frequency dictionary computation, missing data handling, ambiguous state handling, and tree reconstruction with different input scenarios.
## Code Structure
The main class `VanillaGreedySolverTest` inherits from `unittest.TestCase` and contains multiple test methods. Each test method focuses on a specific aspect of the VanillaGreedySolver's functionality, such as frequency dictionary computation, missing data handling, and tree reconstruction. The tests use sample character matrices and expected tree structures to verify the solver's behavior.
## Symbols
### VanillaGreedySolverTest
#### Description
A test class that contains unit tests for the VanillaGreedySolver class.
#### Internal Logic
The class contains multiple test methods, each focusing on a specific aspect of the VanillaGreedySolver's functionality:
1. Frequency dictionary computation
2. Missing data handling
3. Ambiguous state handling
4. Tree reconstruction with various input scenarios
### find\_triplet\_structure
#### Description
A helper function that determines the structure of a triplet in a given tree.
#### Inputs
| Name | Type | Description |
| :------ | :--------------- | :---------------------------------- |
| triplet | tuple | A tuple containing three node names |
| T | networkx.DiGraph | The tree to analyze |
#### Outputs
| Name | Type | Description |
| :-------- | :--- | :------------------------------------------------------ |
| structure | str | The structure of the triplet ("-", "ab", "ac", or "bc") |
#### Internal Logic
1. Find ancestors for each node in the triplet
2. Calculate the number of common ancestors for each pair of nodes
3. Determine the structure based on the number of common ancestors
### test\_basic\_freq\_dict
#### Description
Tests the computation of mutation frequencies for a basic character matrix.
#### Internal Logic
1. Create a sample character matrix
2. Initialize a VanillaGreedySolver
3. Compute mutation frequencies
4. Assert the correctness of the frequency dictionary
### test\_duplicate\_freq\_dict
#### Description
Tests the computation of mutation frequencies for a character matrix with duplicate rows.
#### Internal Logic
Similar to test\_basic\_freq\_dict, but with a character matrix containing duplicate rows.
### test\_ambiguous\_freq\_dict
#### Description
Tests the computation of mutation frequencies for a character matrix with ambiguous states.
#### Internal Logic
Similar to test\_basic\_freq\_dict, but with a character matrix containing ambiguous states (tuples).
### test\_ambiguous\_duplicate\_freq\_dict
#### Description
Tests the computation of mutation frequencies for a character matrix with ambiguous states and duplicate rows.
#### Internal Logic
Similar to test\_ambiguous\_freq\_dict, but with duplicate rows.
### test\_average\_missing\_data
#### Description
Tests the average missing data imputation method.
#### Internal Logic
1. Create a sample character matrix with missing data
2. Apply the average missing data imputation method
3. Assert the correctness of the resulting partitions
### test\_average\_missing\_data\_priors
#### Description
Tests the average missing data imputation method with priors.
#### Internal Logic
Similar to test\_average\_missing\_data, but includes priors for weighting.
### test\_all\_duplicates\_base\_case
#### Description
Tests the solver's behavior when all samples are identical.
#### Internal Logic
1. Create a character matrix with identical rows
2. Solve the tree
3. Verify the resulting tree structure
### test\_case\_1, test\_case\_2
#### Description
Tests the solver's behavior with more complex character matrices.
#### Internal Logic
1. Create a complex character matrix
2. Solve the tree
3. Verify the resulting tree structure using triplet comparisons
### test\_weighted\_case\_trivial
#### Description
Tests the solver's behavior with a weighted case (using priors).
#### Internal Logic
Similar to test\_case\_2, but includes priors for weighting.
### test\_priors\_case
#### Description
Tests the solver's behavior with more complex priors.
#### Internal Logic
Similar to test\_weighted\_case\_trivial, but with more complex priors.
### test\_ambiguous\_no\_missing, test\_ambiguous\_with\_missing, test\_ambiguous\_with\_missing\_and\_duplicates
#### Description
Tests the solver's behavior with ambiguous states, missing data, and duplicates.
#### Internal Logic
1. Create character matrices with ambiguous states, missing data, and/or duplicates
2. Solve the tree
3. Verify the resulting tree structure using triplet comparisons
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------------- |
| unittest | Provides the testing framework |
| itertools | Used for generating combinations |
| networkx | Used for tree representation and analysis |
| pandas | Used for character matrix representation |
| cassiopeia | The main package being tested |
## Error Handling
The test cases use assertions to verify the correctness of the solver's output. If any assertion fails, the test case will raise an AssertionError.
Your response should not exceed 3000 words or 4000 tokens. Focus on providing clear, concise information that can be directly inferred from the code. Include optional sections only when they provide significant value for understanding the code.
# autocorrelation_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/autocorrelation_test.py
## High-level description
This file contains unit tests for the `compute_morans_i` function, which calculates Moran's I statistic for spatial autocorrelation in a phylogenetic tree. The tests cover various scenarios, including single and multiple variables, custom weight matrices, and error handling.
## References
This file references the following symbols:
* `cassiopeia.tl.compute_morans_i`: The function being tested, which calculates Moran's I statistic.
* `cassiopeia.data.CassiopeiaTree`: A class representing a phylogenetic tree.
* `cassiopeia.mixins.AutocorrelationError`: An exception class for errors related to autocorrelation calculations.
## Symbols
### `TestAutocorrelation`
#### Description
This class contains unit tests for the `compute_morans_i` function. It sets up a basic tree structure and example data for testing.
#### Inputs
This class does not take any inputs.
#### Outputs
This class does not return any outputs. It uses the `unittest` framework to run assertions and report test results.
#### Internal Logic
The class defines several test methods, each testing a specific aspect of the `compute_morans_i` function:
* `test_simple_moran_single_variable`: Tests the function with a single variable and compares the result to a known value.
* `test_moran_bivariate`: Tests the function with multiple variables and compares the resulting correlation matrix to expected values.
* `test_moran_custom_weights`: Tests the function with a user-provided weight matrix.
* `test_moran_exceptions`: Tests various error conditions, such as invalid input data types and mismatched tree leaves.
### `setUp`
#### Description
This method sets up the test environment by creating a basic tree structure and example data.
#### Inputs
This method does not take any inputs.
#### Outputs
This method does not explicitly return any outputs. It sets the `self.basic_tree` and `self.X` attributes, which are used by the test methods.
## Error Handling
The `test_moran_exceptions` method specifically tests for various error conditions that can occur in the `compute_morans_i` function, such as:
* Passing non-numerical data in the `X` argument.
* Providing a weight matrix (`W`) that does not have the same leaves as the tree.
* Not providing any data for calculating autocorrelations.
* Providing a weight matrix with incorrect leaves.
These error conditions are handled by raising `AutocorrelationError` exceptions with informative messages.
# iid_exponential_bayesian_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/branch_length_estimator_tests/iid_exponential_bayesian_test.py
## High-level description
This file contains unit tests for the `IIDExponentialBayesian` class, which is a branch length estimator in the Cassiopeia library. The tests primarily focus on comparing the estimator's output with numerically computed values for log likelihood, log joints, and posterior time distributions.
## Code Structure
The code defines several helper functions for numerical calculations and error handling, followed by a `TestIIDExponentialBayesian` class containing multiple test methods. Each test method focuses on a specific aspect of the `IIDExponentialBayesian` class, such as comparing its output to closed-form solutions or numerical approximations.
## References
This code references the `IIDExponentialBayesian` class from `cassiopeia.tools` and the `CassiopeiaTree` class from `cassiopeia.data`.
## Symbols
### `TestIIDExponentialBayesian`
#### Description
A class containing unit tests for the `IIDExponentialBayesian` branch length estimator.
#### Inputs
This class does not take any inputs.
#### Outputs
This class does not return any outputs.
#### Internal Logic
The class contains several test methods, each designed to validate different aspects of the `IIDExponentialBayesian` class:
* `test_against_closed_form_solution_small`: This test compares the estimator's output to closed-form solutions for a small tree with one internal node. It tests various hyperparameter settings and validates the log likelihood, log joints, posterior time distribution, and posterior mean.
* `test_against_closed_form_solution_medium`: Similar to the previous test, but uses a slightly larger tree with two internal nodes. This test is marked as slow and only runs with one set of hyperparameters.
* `test_small_discretization_level_raises_error`: This test ensures that the estimator raises a `ValueError` when the discretization level is too small.
* `test_invalid_tree_topology_raises_error`: This test checks that the estimator raises a `ValueError` for invalid tree topologies, such as trees that are not rooted or have internal nodes with degrees other than 3.
* `test_invalid_sampling_probability_raises_error`: This test ensures that the estimator raises a `ValueError` for invalid sampling probabilities (outside the range (0, 1]).
#### Side Effects
This class does not have any side effects.
***
### `calc_exact_log_full_joint`
#### Description
Calculates the exact log full joint probability density of the observed tree topology, state vectors, and branch lengths.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------- | :--------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the tree topology and character states. |
| mutation\_rate | float | The mutation rate of the model. |
| birth\_rate | float | The birth rate of the model. |
| sampling\_probability | float | The sampling probability of the model. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :-------------------------------------- |
| ll | float | The log full joint probability density. |
#### Internal Logic
The function iterates through the edges of the tree and calculates the likelihood contribution from both the birth process and the mutation process. It considers the sampling probability and uses binomial coefficients to account for the number of mutations along each edge.
***
### `calc_numerical_log_likelihood`
#### Description
Calculates the numerical log likelihood of the observed data (character states and tree topology) using numerical integration.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------- | :--------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the tree topology and character states. |
| mutation\_rate | float | The mutation rate of the model. |
| birth\_rate | float | The birth rate of the model. |
| sampling\_probability | float | The sampling probability of the model. |
| epsrel | float | The relative error tolerance for numerical integration. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :---------------------------- |
| res | float | The numerical log likelihood. |
#### Internal Logic
The function defines an inner function `f` that calculates the full joint probability for a given set of internal node times. It then uses `scipy.integrate.nquad` to numerically integrate `f` over all possible internal node times, effectively marginalizing out the times and obtaining the likelihood of the observed data.
***
### `calc_numerical_log_joints`
#### Description
Calculates the numerical log joint probability density of the observed data and all possible times for a specific node in the tree.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------- | :--------------------------------------------------------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the tree topology and character states. |
| node | str | The name of the internal node for which to calculate the log joints. |
| mutation\_rate | float | The mutation rate of the model. |
| birth\_rate | float | The birth rate of the model. |
| sampling\_probability | float | The sampling probability of the model. |
| discretization\_level | int | The number of timesteps used to discretize time. |
| epsrel | float | The relative error tolerance for numerical integration. |
#### Outputs
| Name | Type | Description |
| :--- | :------- | :-------------------------------------------------------------------------------- |
| res | np.array | An array of log joint probabilities for each possible time of the specified node. |
#### Internal Logic
The function iterates through all possible discretized times for the specified node. For each time, it defines an inner function `f` that calculates the full joint probability for a given set of times for the remaining internal nodes. It then uses `scipy.integrate.nquad` to numerically integrate `f` over all possible times for the other nodes, effectively marginalizing out their times and obtaining the joint probability of the observed data and the specified node's time.
***
### `numerical_posterior_time`
#### Description
Calculates the numerical posterior time distribution for a specific node in the tree.
#### Inputs
| Name | Type | Description |
| :-------------------- | :------------- | :------------------------------------------------------------------------------------ |
| tree | CassiopeiaTree | The CassiopeiaTree object containing the tree topology and character states. |
| node | str | The name of the internal node for which to calculate the posterior time distribution. |
| mutation\_rate | float | The mutation rate of the model. |
| birth\_rate | float | The birth rate of the model. |
| sampling\_probability | float | The sampling probability of the model. |
| discretization\_level | int | The number of timesteps used to discretize time. |
| epsrel | float | The relative error tolerance for numerical integration. |
#### Outputs
| Name | Type | Description |
| :------------------- | :------- | :--------------------------------------------------------------------------- |
| numerical\_posterior | np.array | An array representing the posterior time distribution of the specified node. |
#### Internal Logic
The function first calculates the numerical log joints using `calc_numerical_log_joints`. It then normalizes the log joints to obtain the posterior time distribution.
***
### `relative_error`
#### Description
Calculates the relative error between two positive floating-point numbers.
#### Inputs
| Name | Type | Description |
| :--- | :---- | :----------------- |
| x | float | The first number. |
| y | float | The second number. |
#### Outputs
| Name | Type | Description |
| :--- | :---- | :---------------------------------- |
| | float | The relative error between x and y. |
#### Internal Logic
The function calculates the maximum of the absolute differences between the ratios of the two numbers and their inverses.
***
### `_non_root_internal_nodes`
#### Description
Returns a list of internal nodes in the tree, excluding the root node.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object. |
#### Outputs
| Name | Type | Description |
| :--- | :--------- | :------------------------------------------------- |
| | List\[str] | A list of internal node names, excluding the root. |
#### Internal Logic
The function uses set operations to find the difference between the set of internal nodes and the set containing only the root node.
***
### `_get_number_of_mutated_characters_in_node`
#### Description
Counts the number of mutated characters (excluding missing characters) in a given node.
#### Inputs
| Name | Type | Description |
| :--- | :------------- | :------------------------- |
| tree | CassiopeiaTree | The CassiopeiaTree object. |
| v | str | The name of the node. |
#### Outputs
| Name | Type | Description |
| :--- | :--- | :-------------------------------------------- |
| | int | The number of mutated characters in the node. |
#### Internal Logic
The function retrieves the character states of the node and counts the number of states that are neither 0 nor the missing state indicator.
# iid_exponential_mle_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/branch_length_estimator_tests/iid_exponential_mle_test.py
## High-level description
This file contains unit tests for the `IIDExponentialMLE` class, which is a branch length estimator in the Cassiopeia package. The tests cover various scenarios, including degenerate cases (no mutations or saturation), hand-solvable problems, regression tests, and simulations.
## Code Structure
The code consists of a single class `TestIIDExponentialMLE` which inherits from `unittest.TestCase`. This class contains multiple test methods, each testing a specific aspect of the `IIDExponentialMLE` class. The test methods are decorated with `parameterized.expand` to run the same test with different solvers ("ECOS" and "SCS").
## References
The tests reference the following classes:
* `cassiopeia.data.CassiopeiaTree`: Represents a phylogenetic tree with character states.
* `cassiopeia.simulator.Cas9LineageTracingDataSimulator`: Simulates Cas9-based lineage tracing data.
* `cassiopeia.tools.IIDExponentialMLE`: The branch length estimator being tested.
## Symbols
### `TestIIDExponentialMLE`
#### Description
This class contains unit tests for the `IIDExponentialMLE` class.
#### Inputs
None
#### Outputs
None
#### Internal Logic
The class defines several test methods, each testing a specific aspect of the `IIDExponentialMLE` class. The test methods use assertions to verify the expected behavior of the estimator.
#### Side Effects
The tests modify the `CassiopeiaTree` objects passed to the `IIDExponentialMLE` estimator.
#### Performance Considerations
None
***
### `test_no_mutations`
#### Description
Tests the behavior of the estimator when there are no mutations in the character matrix.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a simple tree with two nodes and no mutations.
* Initializes an `IIDExponentialMLE` object.
* Asserts that a `ValueError` is raised when calling `estimate_branch_lengths`.
***
### `test_saturation`
#### Description
Tests the behavior of the estimator when the character matrix is saturated (all characters are mutated).
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a simple tree with two nodes and a single mutated character.
* Initializes an `IIDExponentialMLE` object.
* Asserts that a `ValueError` is raised when calling `estimate_branch_lengths`.
***
### `test_hand_solvable_problem_1`, `test_hand_solvable_problem_2`, `test_hand_solvable_problem_3`
#### Description
These tests verify the estimator's results against hand-solvable problems with simple tree topologies and a small number of characters.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a simple tree with two nodes and a specified character matrix.
* Initializes an `IIDExponentialMLE` object.
* Calls `estimate_branch_lengths` and asserts that the estimated branch lengths, mutation rate, and log-likelihood match the expected values calculated by hand.
***
### `test_small_tree_with_one_mutation`
#### Description
Tests the estimator on a small perfect binary tree with a single mutation.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a perfect binary tree with one mutation at a specific node.
* Initializes an `IIDExponentialMLE` object.
* Calls `estimate_branch_lengths` and asserts that the estimated branch lengths, mutation rate, and log-likelihood match the expected values calculated by hand.
***
### `test_small_tree_regression`
#### Description
This is a regression test on a small perfect binary tree with a "normal" amount of mutations.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a perfect binary tree with a predefined character matrix.
* Initializes an `IIDExponentialMLE` object.
* Calls `estimate_branch_lengths` and asserts that the estimated branch lengths, mutation rate, and log-likelihood match the expected values from a previous run.
***
### `test_on_simulated_data`
#### Description
Tests the estimator on data simulated using the `Cas9LineageTracingDataSimulator`.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a perfect binary tree and sets predefined node times.
* Simulates Cas9-based lineage tracing data using the `Cas9LineageTracingDataSimulator`.
* Initializes an `IIDExponentialMLE` object.
* Calls `estimate_branch_lengths` and asserts that the estimated node times and mutation rate are close to the ground truth values used for simulation.
***
### `test_subtree_collapses_when_no_mutations`
#### Description
Tests the behavior of the estimator when a subtree has no mutations, ensuring it collapses to 0 length.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a tree with a subtree containing no mutations.
* Initializes an `IIDExponentialMLE` object.
* Calls `estimate_branch_lengths` and asserts that the branch lengths of the mutationless subtree are 0 and the remaining branch lengths, mutation rate, and log-likelihood match the expected values.
***
### `test_minimum_branch_length`
#### Description
Tests the functionality of the `minimum_branch_length` parameter.
#### Inputs
| Name | Type | Description |
| :----- | :--- | :------------------------------ |
| name | str | Name of the test case |
| solver | str | Solver to use ("ECOS" or "SCS") |
#### Outputs
None
#### Internal Logic
* Creates a perfect binary tree with a single mutation.
* Initializes an `IIDExponentialMLE` object with a specified `minimum_branch_length`.
* Calls `estimate_branch_lengths` and asserts that the estimated branch lengths, mutation rate, and log-likelihood match the expected values, taking into account the minimum branch length constraint.
***
### `test_hand_solvable_problem_with_site_rates`, `test_larger_hand_solvable_problem_with_site_rates`
#### Description
These tests verify the estimator's results against hand-solvable problems with specified site-specific mutation rates.
#### Inputs
| Name | Type | Description |
| :------------ | :----------- | :------------------------------------------------------ |
| name | str | Name of the test case |
| solver\_rates | List\[float] | Site-specific mutation rates used by the solver |
| math\_rates | List\[float] | Site-specific mutation rates used for hand calculations |
#### Outputs
None
#### Internal Logic
* Creates a tree with a specified character matrix.
* Initializes an `IIDExponentialMLE` object with the `relative_mutation_rates` parameter set to `solver_rates`.
* Calls `estimate_branch_lengths` and asserts that the estimated branch lengths, node times, and mutation rates match the expected values calculated by hand using `math_rates`.
***
### `test_invalid_site_rates`
#### Description
Tests the behavior of the estimator when invalid site-specific mutation rates are provided.
#### Inputs
| Name | Type | Description |
| :---- | :----------- | :----------------------------------- |
| name | str | Name of the test case |
| rates | List\[float] | Invalid site-specific mutation rates |
#### Outputs
None
#### Internal Logic
* Creates a tree with a specified character matrix.
* Initializes an `IIDExponentialMLE` object with the `relative_mutation_rates` parameter set to `rates`.
* Asserts that a `ValueError` is raised when calling `estimate_branch_lengths`.
## Dependencies
* `networkx`: For working with graph structures.
* `numpy`: For numerical operations.
* `parameterized`: For parameterized testing.
* `cvxpy`: For convex optimization.
## Error Handling
The tests check for `ValueError` and `IIDExponentialMLEError` exceptions in various scenarios.
## API/Interface Reference
None
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/branch_length_estimator_tests/overview
## High-level description
This directory contains unit tests for branch length estimators in the Cassiopeia library, specifically focusing on two classes: `IIDExponentialBayesian` and `IIDExponentialMLE`. These tests are designed to validate the functionality and accuracy of these estimators under various scenarios, including edge cases, hand-solvable problems, and simulated data.
## What does it do?
The tests in this directory serve several purposes:
1. Validate the correctness of the `IIDExponentialBayesian` estimator by comparing its output to closed-form solutions and numerical approximations for log likelihood, log joints, and posterior time distributions.
2. Verify the behavior of the `IIDExponentialMLE` estimator in various scenarios, including:
* Degenerate cases (no mutations or saturation)
* Hand-solvable problems with simple tree topologies
* Regression tests on small trees
* Performance on simulated data
* Handling of subtree collapses when no mutations are present
* Enforcement of minimum branch length constraints
* Incorporation of site-specific mutation rates
3. Ensure proper error handling for invalid inputs or edge cases in both estimators.
These tests help maintain the reliability and accuracy of the branch length estimation algorithms in the Cassiopeia library, which is crucial for phylogenetic analysis and lineage tracing applications.
## Key Files
1. `iid_exponential_bayesian_test.py`:
This file contains tests for the `IIDExponentialBayesian` class. It includes:
* Comparisons against closed-form solutions for small and medium-sized trees
* Validation of error handling for invalid inputs
* Numerical calculations of log likelihood, log joints, and posterior time distributions
2. `iid_exponential_mle_test.py`:
This file contains tests for the `IIDExponentialMLE` class. It includes:
* Tests for degenerate cases (no mutations and saturation)
* Verification of results for hand-solvable problems
* Regression tests on small trees
* Tests using simulated data
* Validation of subtree collapse behavior
* Tests for minimum branch length enforcement
* Verification of site-specific mutation rate handling
Both files use parameterized testing to run the same tests with different solvers (ECOS and SCS) where applicable.
## Dependencies
The tests rely on several external libraries and frameworks:
1. `unittest`: The standard Python unit testing framework.
2. `parameterized`: Used for creating parameterized tests, allowing the same test to be run with different inputs.
3. `numpy`: Used for numerical operations and array manipulations.
4. `scipy`: Specifically `scipy.integrate` is used for numerical integration in the Bayesian tests.
5. `networkx`: Used for working with graph structures, particularly in creating and manipulating tree topologies.
6. `cvxpy`: A Python-embedded modeling language for convex optimization problems, used in the MLE estimator.
From the Cassiopeia library:
* `cassiopeia.data.CassiopeiaTree`: Represents the phylogenetic tree structure.
* `cassiopeia.simulator.Cas9LineageTracingDataSimulator`: Used for generating simulated data.
* `cassiopeia.tools.IIDExponentialBayesian` and `cassiopeia.tools.IIDExponentialMLE`: The classes being tested.
## Configuration
The tests do not rely on external configuration files or environment variables. However, they do use various parameters to configure the estimators and test scenarios:
1. For `IIDExponentialBayesian`:
* `mutation_rate`: The mutation rate of the model.
* `birth_rate`: The birth rate of the model.
* `sampling_probability`: The sampling probability of the model.
* `discretization_level`: The number of timesteps used to discretize time.
2. For `IIDExponentialMLE`:
* `solver`: The optimization solver to use (ECOS or SCS).
* `minimum_branch_length`: A constraint on the minimum allowed branch length.
* `relative_mutation_rates`: Site-specific mutation rates.
These parameters are set within the test methods to create various test scenarios and validate the estimators' behavior under different conditions.
In summary, this directory contains comprehensive unit tests for two branch length estimators in the Cassiopeia library, ensuring their accuracy and reliability across a wide range of scenarios and input conditions.
# coupling_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/coupling_test.py
## High-level description
This file contains unit tests for the coupling estimators implemented in `cassiopeia/tools/coupling.py`, specifically focusing on the `compute_evolutionary_coupling` function. These tests ensure the correctness of the evolutionary coupling calculations under various scenarios, including different input parameters and data configurations.
## Code Structure
The code consists of a single test class, `TestDataUtilities`, which houses multiple test methods. Each test method focuses on a specific aspect of the `compute_evolutionary_coupling` function, such as its basic functionality, handling of custom dissimilarity maps, minimum proportion thresholds, and potential error cases.
## Symbols
### `TestDataUtilities`
#### Description
This class contains unit tests for data utility functions, specifically focusing on the `compute_evolutionary_coupling` function.
#### Inputs
* **None**
#### Outputs
* **None**
#### Internal Logic
This class utilizes the `unittest` framework to define and execute test methods. Each test method sets up a specific scenario using a predefined tree structure and cell metadata, then calls the `compute_evolutionary_coupling` function with specific parameters. The expected output is then compared to the actual output to verify the correctness of the function.
##### `test_evolutionary_coupling_basic`
* **Description:** Tests the basic functionality of the `compute_evolutionary_coupling` function with default parameters.
* **Inputs:** None
* **Outputs:** None
* **Internal Logic:**
1. Sets up a `CassiopeiaTree` with predefined tree topology and cell metadata.
2. Calls `compute_evolutionary_coupling` with the tree, "CellType" as the meta variable, a fixed random state, minimum proportion of 0, and 10 shuffles.
3. Computes the expected inter-cluster distances and evolutionary coupling matrix based on the predefined data and parameters.
4. Asserts that the calculated evolutionary coupling matrix matches the expected matrix.
5. Asserts that a `CassiopeiaError` is raised when attempting to compute evolutionary coupling with a numerical meta variable ("nUMI").
##### `test_evolutionary_coupling_custom_dissimilarity_map`
* **Description:** Tests the functionality of `compute_evolutionary_coupling` with a custom dissimilarity map.
* **Inputs:** None
* **Outputs:** None
* **Internal Logic:**
1. Sets up a `CassiopeiaTree` and a custom weight matrix representing the dissimilarity map.
2. Calls `compute_evolutionary_coupling` with the tree, "CellType" as the meta variable, a fixed random state, minimum proportion of 0, 10 shuffles, and the custom dissimilarity map.
3. Computes the expected inter-cluster distances and evolutionary coupling matrix based on the predefined data, parameters, and custom dissimilarity map.
4. Asserts that the calculated evolutionary coupling matrix matches the expected matrix.
##### `test_evolutionary_coupling_minimum_proportion`
* **Description:** Tests the functionality of `compute_evolutionary_coupling` with a minimum proportion threshold for categories.
* **Inputs:** None
* **Outputs:** None
* **Internal Logic:**
1. Sets up a `CassiopeiaTree` and modifies the cell metadata to include a category ("TypeD") that falls below the minimum proportion threshold.
2. Calls `compute_evolutionary_coupling` with the tree, "CellType" as the meta variable, a fixed random state, a minimum proportion of 1/6 (which excludes "TypeC" and "TypeD"), and 10 shuffles.
3. Asserts that the resulting evolutionary coupling matrix does not include the excluded categories ("TypeC" and "TypeD").
4. Computes the expected inter-cluster distances and evolutionary coupling matrix based on the remaining categories and asserts that the calculated matrix matches the expected matrix.
# lbi_jungle_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/fitness_estimator_tests/lbi_jungle_test.py
## High-level description
This file contains unit tests for the LBIJungle fitness estimator in the cassiopeia.tools module. It tests the functionality of the LBIJungle class on a small tree to ensure that fitness estimates are calculated correctly and make sense based on the tree structure.
## Code Structure
The file defines a single test class `TestLBIJungle` which inherits from `unittest.TestCase`. This class contains two test methods: `test_small_tree` and `test_raises_error_if_leaf_name_startswith_underscore`.
## Symbols
### TestLBIJungle
#### Description
A test class that contains unit tests for the LBIJungle fitness estimator.
#### Internal Logic
The class sets up test scenarios and asserts expected outcomes for the LBIJungle fitness estimator.
### test\_small\_tree
#### Description
Tests the LBIJungle fitness estimator on a small, manually constructed tree to verify that the fitness estimates are sensible.
#### Internal Logic
1. Constructs a small tree using networkx.
2. Converts the networkx tree to a CassiopeiaTree.
3. Sets node times for the tree.
4. Creates an LBIJungle fitness estimator and applies it to the tree.
5. Retrieves fitness estimates for all nodes except the root.
6. Performs several assertions to check if the fitness estimates make sense based on the tree structure:
* Compares fitness of internal nodes.
* Checks equality of fitness for leaves with the same parent.
* Compares fitness between different groups of leaves.
* Verifies that parent nodes have higher fitness than their children.
### test\_raises\_error\_if\_leaf\_name\_startswith\_underscore
#### Description
Tests that the LBIJungle fitness estimator raises a FitnessEstimatorError when a leaf name starts with an underscore.
#### Internal Logic
1. Constructs a small tree with a leaf name starting with an underscore.
2. Converts the networkx tree to a CassiopeiaTree.
3. Creates an LBIJungle fitness estimator.
4. Asserts that applying the estimator to the tree raises a FitnessEstimatorError.
## Dependencies
* unittest: Python's built-in unit testing framework
* networkx: For creating and manipulating graph structures
* cassiopeia.data.CassiopeiaTree: The tree data structure used in Cassiopeia
* cassiopeia.tools.FitnessEstimatorError: Custom error for fitness estimation issues
* cassiopeia.tools.LBIJungle: The fitness estimator being tested
## Error Handling
The tests check for proper error handling when invalid input is provided (e.g., leaf names starting with underscores).
## Performance Considerations
These are unit tests, so performance is not a primary concern. However, the tests use a small tree to ensure quick execution.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/fitness_estimator_tests/overview
## High-level description
This file contains unit tests for the LBIJungle fitness estimator in the cassiopeia.tools module. It tests the functionality of the LBIJungle class on a small tree to ensure that fitness estimates are calculated correctly and make sense based on the tree structure.
## Code Structure
The file defines a single test class `TestLBIJungle` which inherits from `unittest.TestCase`. This class contains two test methods: `test_small_tree` and `test_raises_error_if_leaf_name_startswith_underscore`.
## Symbols
### TestLBIJungle
#### Description
A test class that contains unit tests for the LBIJungle fitness estimator.
#### Internal Logic
The class sets up test scenarios and asserts expected outcomes for the LBIJungle fitness estimator.
### test\_small\_tree
#### Description
Tests the LBIJungle fitness estimator on a small, manually constructed tree to verify that the fitness estimates are sensible.
#### Internal Logic
1. Constructs a small tree using networkx.
2. Converts the networkx tree to a CassiopeiaTree.
3. Sets node times for the tree.
4. Creates an LBIJungle fitness estimator and applies it to the tree.
5. Retrieves fitness estimates for all nodes except the root.
6. Performs several assertions to check if the fitness estimates make sense based on the tree structure:
* Compares fitness of internal nodes.
* Checks equality of fitness for leaves with the same parent.
* Compares fitness between different groups of leaves.
* Verifies that parent nodes have higher fitness than their children.
### test\_raises\_error\_if\_leaf\_name\_startswith\_underscore
#### Description
Tests that the LBIJungle fitness estimator raises a FitnessEstimatorError when a leaf name starts with an underscore.
#### Internal Logic
1. Constructs a small tree with a leaf name starting with an underscore.
2. Converts the networkx tree to a CassiopeiaTree.
3. Creates an LBIJungle fitness estimator.
4. Asserts that applying the estimator to the tree raises a FitnessEstimatorError.
## Dependencies
* unittest: Python's built-in unit testing framework
* networkx: For creating and manipulating graph structures
* cassiopeia.data.CassiopeiaTree: The tree data structure used in Cassiopeia
* cassiopeia.tools.FitnessEstimatorError: Custom error for fitness estimation issues
* cassiopeia.tools.LBIJungle: The fitness estimator being tested
## Error Handling
The tests check for proper error handling when invalid input is provided (e.g., leaf names starting with underscores).
## Performance Considerations
These are unit tests, so performance is not a primary concern. However, the tests use a small tree to ensure quick execution.
# Overview
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/overview
## High-level description
This directory contains unit tests for various tools and algorithms implemented in the Cassiopeia library, focusing on phylogenetic analysis and lineage tracing. The tests cover a wide range of functionalities, including autocorrelation, branch length estimation, coupling, fitness estimation, parameter estimation, small parsimony, topology analysis, and tree metrics.
## What does it do?
The tests in this directory serve to validate and ensure the correctness of the following key functionalities:
1. Autocorrelation: Tests for spatial autocorrelation calculations using Moran's I statistic.
2. Branch length estimation: Validates Bayesian and Maximum Likelihood Estimation (MLE) methods for estimating branch lengths in phylogenetic trees.
3. Coupling: Verifies the calculation of evolutionary coupling between different cell types or states.
4. Fitness estimation: Tests the Local Branch Index (LBI) method for estimating fitness in phylogenetic trees.
5. Parameter estimation: Validates functions that estimate mutation rates and missing data probabilities in lineage tracing data.
6. Small parsimony: Tests algorithms for inferring ancestral states and calculating parsimony scores on phylogenetic trees.
7. Topology analysis: Verifies functions related to tree topology, including cophenetic correlation calculations.
8. Tree metrics: Tests various tree-based metrics, including parsimony scores, likelihood calculations, and parameter estimations for lineage tracing models.
These tests ensure that the core algorithms and functions in Cassiopeia perform correctly under various scenarios, including edge cases and simulated data.
## Entry points
The main entry points for the tests are the individual test files, each focusing on a specific aspect of the Cassiopeia toolkit:
1. `autocorrelation_test.py`: Tests for spatial autocorrelation calculations.
2. `branch_length_estimator_tests/`: Directory containing tests for branch length estimation methods.
3. `coupling_test.py`: Tests for evolutionary coupling calculations.
4. `fitness_estimator_tests/`: Directory containing tests for fitness estimation methods.
5. `parameter_estimators_test.py`: Tests for parameter estimation functions.
6. `small_parsimony_test.py`: Tests for small parsimony algorithms.
7. `topology_test.py`: Tests for topology-related functions.
8. `tree_metrics_test.py`: Tests for various tree-based metrics and calculations.
Each of these files contains a set of unit tests that can be run independently or as part of a larger test suite.
## Key Files
1. `autocorrelation_test.py`: Tests the `compute_morans_i` function for calculating Moran's I statistic.
2. `branch_length_estimator_tests/`:
* `iid_exponential_bayesian_test.py`: Tests the `IIDExponentialBayesian` estimator.
* `iid_exponential_mle_test.py`: Tests the `IIDExponentialMLE` estimator.
3. `coupling_test.py`: Tests the `compute_evolutionary_coupling` function.
4. `fitness_estimator_tests/lbi_jungle_test.py`: Tests the Local Branch Index (LBI) fitness estimator.
5. `parameter_estimators_test.py`: Tests functions for estimating mutation rates and missing data probabilities.
6. `small_parsimony_test.py`: Tests Fitch-Hartigan algorithm and related parsimony functions.
7. `topology_test.py`: Tests topology-related functions, including cophenetic correlation calculations.
8. `tree_metrics_test.py`: Tests various tree-based metrics and likelihood calculations.
## Dependencies
The tests rely on several external libraries and frameworks:
1. `unittest`: The standard Python unit testing framework.
2. `parameterized`: Used for creating parameterized tests.
3. `numpy`: Used for numerical operations and array manipulations.
4. `scipy`: Used for various scientific computing tasks, including numerical integration.
5. `networkx`: Used for working with graph structures, particularly in creating and manipulating tree topologies.
6. `pandas`: Used for data manipulation and analysis.
7. `cvxpy`: Used in optimization problems, particularly in the MLE branch length estimator.
From the Cassiopeia library:
* Various modules and classes, including `CassiopeiaTree`, `IIDExponentialBayesian`, `IIDExponentialMLE`, `LBIJungle`, and others.
## Configuration
The tests do not rely on external configuration files or environment variables. Test parameters and configurations are set within the test methods themselves to create various scenarios and validate the behavior of the functions under different conditions.
In summary, this directory contains a comprehensive set of unit tests that ensure the correctness and reliability of the core algorithms and functions in the Cassiopeia library for phylogenetic analysis and lineage tracing.
# parameter_estimators_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/parameter_estimators_test.py
## High-level description
This file contains unit tests for the `parameter_estimators.py` module in Cassiopeia. The tests cover functions that estimate parameters related to mutation rates and missing data in lineage tracing data.
## Code Structure
The `TestCassiopeiaTree` class contains multiple test methods, each testing different aspects of the parameter estimation functions. The tests are structured around two example trees: `self.discrete_tree` and `self.continuous_tree`, representing discrete and continuous character data respectively.
## Symbols
### `TestCassiopeiaTree`
#### Description
A class containing unit tests for parameter estimation functions.
#### Inputs
None
#### Outputs
None
#### Internal Logic
The class uses the `unittest` framework to define and run tests. It sets up two example trees (`self.discrete_tree` and `self.continuous_tree`) in the `setUp` method, which are then used by the individual test methods.
***
### `test_proportions`
#### Description
Tests the `get_proportion_of_mutation` and `get_proportion_of_missing_data` functions.
#### Inputs
Uses `self.discrete_tree` and `self.continuous_tree`
#### Outputs
None
#### Internal Logic
* Calculates the proportion of mutations and missing data for both trees using the functions being tested.
* Asserts that the calculated proportions match the expected values.
***
### `test_estimate_mutation_rate`
#### Description
Tests the `estimate_mutation_rate` function.
#### Inputs
Uses `self.discrete_tree` and `self.continuous_tree`
#### Outputs
None
#### Internal Logic
* Estimates the mutation rate for both trees using the function being tested, with different combinations of the `continuous` and `assume_root_implicit_branch` parameters.
* Asserts that the estimated mutation rates are close to the expected values.
***
### `test_estimate_missing_data_bad_cases`
#### Description
Tests the error handling of the `estimate_missing_data_rates` function.
#### Inputs
Uses `self.discrete_tree` and `self.continuous_tree`
#### Outputs
None
#### Internal Logic
* Defines several scenarios where the `estimate_missing_data_rates` function should raise either a `ParameterEstimateError` or a `ParameterEstimateWarning`.
* Uses `assertRaises` to verify that the expected exceptions are raised in each scenario.
***
### `test_estimate_stochastic_missing_data_probability`
#### Description
Tests the `estimate_missing_data_rates` function for estimating the stochastic missing data probability.
#### Inputs
Uses `self.discrete_tree` and `self.continuous_tree`
#### Outputs
None
#### Internal Logic
* Estimates the stochastic missing data probability for both trees using the function being tested, with different combinations of the `continuous`, `assume_root_implicit_branch`, and `heritable_missing_rate` parameters.
* Asserts that the estimated probabilities are close to the expected values.
***
### `test_estimate_heritable_missing_data_rate`
#### Description
Tests the `estimate_missing_data_rates` function for estimating the heritable missing data rate.
#### Inputs
Uses `self.discrete_tree` and `self.continuous_tree`
#### Outputs
None
#### Internal Logic
* Estimates the heritable missing data rate for both trees using the function being tested, with different combinations of the `continuous`, `assume_root_implicit_branch`, and `stochastic_missing_probability` parameters.
* Asserts that the estimated rates are close to the expected values.
## Dependencies
* unittest
* networkx
* numpy
* pandas
* cassiopeia
* cassiopeia.tools.parameter\_estimators
* cassiopeia.mixins
## Error Handling
The tests in this file primarily focus on verifying the correct error handling of the parameter estimation functions. They check for scenarios where `ParameterEstimateError` and `ParameterEstimateWarning` exceptions should be raised due to invalid input parameters or estimated values.
# small_parsimony_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/small_parsimony_test.py
## High-level description
This file contains unit tests for the `small_parsimony.py` module in Cassiopeia. The tests cover functions related to small parsimony inference on phylogenetic trees, including Fitch-Hartigan reconstruction, parsimony scoring, and the FitchCount algorithm.
## Code Structure
The code defines a single test class `TestSmallParsimony` that inherits from `unittest.TestCase`. This class contains multiple test methods, each testing a specific function or aspect of the `small_parsimony` module. The test methods use various assertions provided by the `unittest` framework to check for expected outcomes.
## References
This test file references the following symbols from the `cassiopeia.tools.small_parsimony` module:
* `fitch_hartigan_bottom_up`
* `fitch_hartigan_top_down`
* `fitch_hartigan`
* `score_small_parsimony`
* `fitch_count`
## Symbols
### `TestSmallParsimony`
#### Description
This class contains unit tests for functions in the `cassiopeia.tools.small_parsimony` module. It sets up two test trees, a binary tree and a general tree, with associated character data, to be used in the test cases.
#### Inputs
None
#### Outputs
None
#### Internal Logic
The class defines several test methods, each testing a specific function or aspect of the `small_parsimony` module. Each test method sets up the necessary input data and calls the function being tested. Assertions are then used to verify that the function behaves as expected.
### `test_fitch_hartigan_bottom_up`
#### Description
Tests the `fitch_hartigan_bottom_up` function, which performs the bottom-up phase of the Fitch-Hartigan algorithm.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Checks for expected errors when invalid input is provided.
* Runs the `fitch_hartigan_bottom_up` function on the binary tree.
* Verifies that the correct ancestral state sets are assigned to each node.
* Tests the function with different `add_key` parameters.
### `test_fitch_hartigan_top_down`
#### Description
Tests the `fitch_hartigan_top_down` function, which performs the top-down refinement step of the Fitch-Hartigan algorithm.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_hartigan_bottom_up` function to initialize ancestral state sets.
* Runs the `fitch_hartigan_top_down` function on the binary tree.
* Verifies that the correct character states are assigned to each node.
* Tests the function with different `label_key` parameters.
### `test_fitch_hartigan`
#### Description
Tests the `fitch_hartigan` function, which combines the bottom-up and top-down steps of the Fitch-Hartigan algorithm.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_hartigan` function on the binary tree.
* Verifies that the correct character states are assigned to each node.
### `test_score_parsimony`
#### Description
Tests the `score_small_parsimony` function, which calculates the parsimony score of a tree.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Checks for expected errors when invalid input is provided.
* Runs the `score_small_parsimony` function with and without ancestral state inference.
* Verifies that the calculated parsimony score is correct in both cases.
### `test_general_tree_fitch_bottom_up`
#### Description
Tests the `fitch_hartigan_bottom_up` function on a general (non-binary) tree.
#### Inputs
Uses the `self.general_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_hartigan_bottom_up` function on the general tree.
* Verifies that the correct ancestral state sets are assigned to each node.
### `test_general_tree_fitch_hartigan`
#### Description
Tests the `fitch_hartigan` function on a general (non-binary) tree.
#### Inputs
Uses the `self.general_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_hartigan` function on the general tree.
* Verifies that the correct character states are assigned to each node.
### `test_general_tree_parsimony`
#### Description
Tests the `score_small_parsimony` function on a general (non-binary) tree.
#### Inputs
Uses the `self.general_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `score_small_parsimony` function on the general tree.
* Verifies that the calculated parsimony score is correct.
### `test_fitch_count_basic_binary`
#### Description
Tests the `fitch_count` function on a basic binary tree.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_count` function on the binary tree.
* Verifies that the resulting count matrix has the correct dimensions and values.
* Tests the function with and without pre-computed ancestral states.
### `test_fitch_count_basic_binary_custom_state_space`
#### Description
Tests the `fitch_count` function with a custom state space.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_count` function with a specified state space.
* Verifies that the resulting count matrix has the correct dimensions and values.
* Checks for expected errors when an invalid state space is provided.
### `test_fitch_count_basic_binary_internal_node`
#### Description
Tests the `fitch_count` function with a specified root node.
#### Inputs
Uses the `self.binary_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_count` function with a specified root node.
* Verifies that the resulting count matrix has the correct dimensions and values.
### `test_fitch_count_general_tree`
#### Description
Tests the `fitch_count` function on a general (non-binary) tree.
#### Inputs
Uses the `self.general_tree` object as input.
#### Outputs
None
#### Internal Logic
* Runs the `fitch_count` function on the general tree.
* Verifies that the resulting count matrix has the correct dimensions and values.
## Dependencies
* `unittest`
* `networkx`
* `numpy`
* `pandas`
* `cassiopeia`
## Error Handling
The test methods use assertions provided by the `unittest` framework to check for expected errors. For example, the `assertRaises` method is used to verify that a specific exception is raised when invalid input is provided to a function.
# topology_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/topology_test.py
## High-level description
This file contains a test suite for the topology functions in the `cassiopeia/tools/topology.py` module. It tests various functions related to tree topology, including expansion probabilities, cophenetic correlation, and simple coalescent probability calculations.
## Code Structure
The main class `TestTopology` inherits from `unittest.TestCase` and contains several test methods. The `setUp` method creates a sample tree and character matrix for use in the tests. Each test method focuses on a specific functionality of the topology module.
## Symbols
### TestTopology
#### Description
A test class that inherits from `unittest.TestCase` and contains multiple test methods for the topology functions.
#### Internal Logic
1. Sets up a sample tree and character matrix in the `setUp` method.
2. Implements various test methods to check different aspects of the topology functions.
### test\_simple\_choose\_function
#### Description
Tests the `nCk` function from the topology module.
#### Internal Logic
1. Checks if `nCk(10, 2)` returns the expected value of 45.
2. Verifies that `nCk(5, 7)` raises a `CassiopeiaError`.
### test\_simple\_coalescent\_probability
#### Description
Tests the `simple_coalescent_probability` function from the topology module.
#### Internal Logic
1. Calculates the coalescent probability for given parameters and compares it to an expected value.
2. Verifies that invalid input raises a `CassiopeiaError`.
### test\_expansion\_probability
#### Description
Tests the `compute_expansion_pvalues` function from the topology module.
#### Internal Logic
1. Calls `compute_expansion_pvalues` with different parameters.
2. Compares the resulting expansion probabilities with expected values for each node in the tree.
### test\_expansion\_probability\_variable\_depths
#### Description
Tests the `compute_expansion_pvalues` function with variable depths.
#### Internal Logic
1. Calls `compute_expansion_pvalues` with a minimum depth parameter.
2. Verifies the resulting expansion probabilities against expected values.
### test\_expansion\_probability\_copy\_tree
#### Description
Tests the `compute_expansion_pvalues` function with the copy option.
#### Internal Logic
1. Calls `compute_expansion_pvalues` with the copy option set to True.
2. Verifies that the original tree is not modified and the new tree has the expected probabilities.
### test\_cophenetic\_correlation\_perfect
#### Description
Tests the `compute_cophenetic_correlation` function with a perfect correlation scenario.
#### Internal Logic
1. Creates a custom dissimilarity map.
2. Calls `compute_cophenetic_correlation` and verifies that it returns a perfect correlation (1.0).
3. Tests the function with a custom weight matrix.
### test\_cophenetic\_correlation\_default
#### Description
Tests the `compute_cophenetic_correlation` function with default parameters.
#### Internal Logic
1. Calls `compute_cophenetic_correlation` without custom parameters.
2. Verifies that the resulting correlation is close to an expected value.
## Dependencies
| Dependency | Purpose |
| :--------- | :---------------------------------------- |
| unittest | Provides the testing framework |
| networkx | Used for creating and manipulating graphs |
| numpy | Used for numerical operations |
| pandas | Used for data manipulation and analysis |
| cassiopeia | The main package being tested |
## Error Handling
The test suite uses assertions to check for expected values and behaviors. It also tests for specific error cases, such as raising `CassiopeiaError` for invalid inputs.
# tree_metrics_test.py
Source: https://demo.agenticlabs.com/detailed-code-docs/test/tools_tests/tree_metrics_test.py
## High-level description
This file contains unit tests for the `tree_metrics.py` module in the Cassiopeia package. These tests cover functions that calculate various tree metrics, including parsimony, log transition probability, log likelihood (both discrete and continuous), and parameter estimation for lineage tracing models.
## Code Structure
This file defines a single test class, `TestCassiopeiaTree`, which houses multiple test methods. Each test method focuses on a specific function or aspect of the `tree_metrics.py` module. The tests utilize small, predefined tree structures and character matrices to validate the correctness of the implemented algorithms.
## Symbols
### `TestCassiopeiaTree`
#### Description
This class contains unit tests for functions in the `tree_metrics.py` module. It sets up test data representing small trees and character matrices, then uses these to test the calculation of various tree metrics.
#### Inputs
This class does not take any explicit inputs.
#### Outputs
This class does not return any explicit outputs. It is used for running tests.
#### Internal Logic
The class defines several test methods, each testing a specific function or aspect of the `tree_metrics.py` module. These methods use `assertEqual`, `assertTrue`, `assertRaises`, and `assertAlmostEqual` to check for expected outcomes.
##### `test_parsimony_bad_cases`
Tests the `calculate_parsimony` function for expected error handling in cases where the tree is not properly initialized or ancestral character inference is required but not enabled.
##### `test_parsimony_reconstruct_internal_states`
Tests the `calculate_parsimony` function with the option to infer ancestral character states, verifying the calculated parsimony score.
##### `test_parsimony_specify_internal_states`
Tests the `calculate_parsimony` function with pre-specified internal character states, verifying the calculated parsimony score.
##### `test_log_transition_probability`
Tests the `log_transition_probability` function, which calculates the log probability of transitioning between character states, considering factors like mutation rate, missing data rate, and time.
##### `test_log_likelihood_of_character`
Tests the `log_likelihood_of_character` function, which calculates the log likelihood of observing a specific character on the tree, considering mutation and missing data probabilities.
##### `test_bad_lineage_tracing_parameters`
Tests for expected error handling when invalid lineage tracing parameters (mutation rate, heritable missing rate, stochastic missing probability) are provided.
##### `test_get_lineage_tracing_parameters`
Tests the `get_lineage_tracing_parameters` function, which retrieves or estimates lineage tracing parameters from the tree, ensuring the returned values are as expected.
##### `test_likelihood_bad_cases`
Tests the likelihood calculation functions (`calculate_likelihood_discrete` and `calculate_likelihood_continuous`) for expected error handling in cases of improper tree initialization or missing data.
##### `test_likelihood_simple_mostly_missing`
Tests the `calculate_likelihood_discrete` function with a simple tree and character matrix containing mostly missing data, verifying the calculated likelihood.
##### `test_likelihood_more_complex_case`
Tests the `calculate_likelihood_discrete` function with a more complex tree and character matrix, verifying the calculated likelihood.
##### `test_likelihood_set_internal_states`
Tests the `calculate_likelihood_discrete` function with pre-specified internal character states, verifying the calculated likelihood.
##### `test_likelihood_time`
Tests the `calculate_likelihood_continuous` function, which incorporates branch lengths (time) into the likelihood calculation, verifying the results.
##### `test_likelihood_sum_to_one`
Tests that the calculated likelihoods for all possible character state combinations sum to one, as expected for a probability distribution.
## Dependencies
* unittest
* itertools
* networkx
* numpy
* pandas
* cassiopeia
```python theme={null}
import unittest
import itertools
import networkx as nx
import numpy as np
import pandas as pd
import cassiopeia as cas
```
# version.py
Source: https://demo.agenticlabs.com/detailed-code-docs/version.py
## High-level description
This Python file defines and stores the version number of a software project. It also includes a simple script to print the version when the file is run directly.
## Symbols
### `version`
#### Description
A string variable that holds the current version number of the software.
#### Inputs
None
#### Outputs
| Name | Type | Description |
| :------ | :--- | :--------------------------------- |
| version | str | The version number of the software |
## Main Script
The file includes a conditional block that executes when the script is run directly (not imported as a module).
### Description
When the script is run as the main program, it prints the `version` to the console.
### Internal Logic
1. Checks if the script is being run as the main program using the `__name__ == '__main__'` condition.
2. If true, it prints the `version` using the `print()` function.
## Usage
This file can be used in two ways:
1. Imported as a module in other Python scripts to access the `version` variable.
2. Run directly as a script to print the current version number.
## Performance Considerations
This is a simple script with minimal performance impact. It's primarily used for version management and doesn't involve any computationally intensive operations.
# Welcome to the Stanza Demo
Source: https://demo.agenticlabs.com/get-started/introduction
### What is Stanza?
Stanza by Agentic Labs is an AI-powered documentation tool that generates comprehensive, queryable documentation for your codebase in minutes.
Engineering teams use Stanza to make their code accessible to all members of the team, technical and non-technical.
### What's in this demo?
We documented Cassiopeia - a scientific python package for analyzing cell division from DNA, developed by Yosef Lab at UC Berkeley.
### Get started
1. Technical summary: Start with the [top-level overview](/detailed-code-docs/project_root) to understand Cassiopeia's structure and how it works.
2. AI-powered answers: Use the search bar to ask questions like:
* "How do I start a processing pipeline?"
* "What algorithms are used for phylogenetic tree reconstruction?"
* "Explain the simulation models in Cassiopeia"
To get AI completions, click `Can you tell me about...` button after typing your question
3. Key components: Dive into some of Cassiopeia's main modules:
* Tree Reconstruction: See [GreedySolver](/detailed-code-docs/cassiopeia/solver/GreedySolver.py) to understand how trees are inferred from a set of samples.
* Visualization: Look at [cassiopeia/plotting](/detailed-code-docs/cassiopeia/plotting/) for data visualization
### What we offer
* Lightning-fast doc generation
* Real-time syncing with your code
* Code hosting integrations (Github, Gitlab, Bitbucket)
* Documentation provider intgrations (Confluence, Gitbook, Mintlify, Github Pages)
### Get in touch
* Email: [rob@agenticlabs.com](mailto:rob@agenticlabs.com)
* Book a meeting: [View my calendar](https://calendly.com/rob-mck/15-minute-meeting)