Home

Awesome

toffy

<div align="center">
CI / CDCI Coverage Status
PackagePyPI - Version PyPI - Downloads PyPI - Python Version
Metacode style - Black types - Mypy PyPI - License
</div>

The toffy repo is designed to simplify the process of generating and processing data on the MIBIScope platform.

This repo is currently in beta testing. None of the code has been published yet, and we will be making breaking changes frequently. If you find bugs, please open an issue.

Table of Contents

Overview

The repo has four main parts, with associated code and jupyter notebooks for each. We have also recorded workshop talks which complement the repository. MIBI Workshop (Pre-Recorded Lectures) Playlist.

1. Using toffy for the first time

The first time you use toffy on one of the commercial instruments, you'll need to perform some basic tasks to ensure everything is working properly. The set up jupyter notebook will guide you through this process and the resulting directory structure is explained below (directory structure). For more information, see the setup toffy walkthrough.

2. Setting up a MIBI run

For large MIBI runs, it is often convenient to automatically generate the JSON file containing the individual FOVs. There are two notebooks for this task, one for large tiled regions, the second for TMAs. If you will be tiling multiple adjacent FOVs together into a single image, the tiling notebook can automate this process. You provide the location of the top corner of the tiled region, along with the number of fovs along the rows and columns, and it will automatically create the appropriate JSON file.

The second notebook is for TMAs. This notebook is run after you have selected the appropriate cores from the TMA. It will generate an overlay with the image of the TMA and the locations you picked to ensure you selected the correct cores. It will then check that they are named correctly and that there are no duplicates.

For more information, see the MIBI tiling walkthrough.

3. Evaluating a MIBI run

There are a number of different computational tasks to complete once a MIBI run has finished to ensure everything went smoothly.

4. Processing After a MIBI run

Once your run has finished, you can begin to process the data to make it ready for analysis. To remove background signal contamination, as well as compensate for channel crosstalk, you can use the compensation notebook. This will guide you through the Rosetta algorithm, which uses a flow-cytometry style compensation approach to remove spurious signal.

Following compensation, you will want to normalize your images to ensure consistent intensity across the run. You can use the normalization notebook to perform this step.

5. Formatting MIBI Data for Downstream Analysis

After the image processing and cleanup from toffy is complete, the final step is to format your data to faciliate easy downstream analysis. The reorganization notebook will walk you through the process of renaming FOVs, combining partial runs, and consolidating your images. For more information, see the reorganizing your data walkthrough.

Pipeline Flowchart

flow-chart

Installation

In order to get toffy working, you'll need to first install some dependencies and the repository itself. For more information, see the toffy setup walkthrough

Requirements for specific operating systems

The process of setting up is largely the same for different operating systems. However, there are a few key differences.

Windows

macOS

Setting up the virtual environment

<p align="center"> <img height="400" src="templates/img/conda_powershell.png" width="500"/> </p>

If you do not already have git installed, run

conda install git

Navigate to the desired location (ex: Documents) and clone the repo.

cd .\Documents\
git clone https://github.com/angelolab/toffy.git

Move into directory and build the environment

cd toffy
conda env create -f release-environment.yml

This creates a Python 3.11 environment named toffy_env. You can view everything that gets installed by looking at the release-environment.yml file.

If you would like to use the latest version of toffy, you can instead run:

cd toffy
conda env create -f prerelease-environment.yml

Which will create a Python 3.11 environment named toffy_prerelease_env. This will install the latest version of toffy from the main branch.

Using the repo

Once you're ready to use the repo, enter the following commands.

First, activate the environment:

conda activate toffy_env

Once activated, notebooks can be used via this command for Windows:

start_jupyter.sh

or this command for macOS:

./start_jupyter.sh

You can leave the jupyter notebook running once you're done. If it ever gets closed or you need to reopen it, just follow the steps above.

Updating the repo

The toffy repo is constantly being updated. In order to get those changes to your version, you'll need to tell git to update with the following command:

git pull

After performing the above command, you will sometimes need to update your environment:

conda remove --name toffy_env --all
conda env create -f release-environment.yml

or

conda remove --name toffy_prerelease_env --all
conda env create -f prerelease-environment.yml

To update the notebooks, run this command for Windows:

start_jupyter.sh -u

or this command for macOS:

./start_jupyter.sh -u

Directory structure

Data from each run on the MIBI will be stored in the default base directory D:\\Data, in a subdirectory labeled with the run name. The set up jupyter notebook creates the following folders that will be used throughout toffy. <br> <br> Four new folders are created on the D drive:

<figcaption align = "center"><b>Directories in D drive</b></figcaption>

D directories

Within C:\\Users\\Customer.ION\\Documents are directories that store necessary files used to set up and monitor a MIBI run.

<figcaption align = "center"><b>Directories in C drive</b></figcaption>

C directories

<br><br> You can see below how to pin a folder to Quick Access, which can then be easily located in the section of the same name on the left side of File Explorer.

We suggest pinning the following folders: tiled_image_jsons, autolabeled_jsons, run_metrics.

Quick Access

Panel format

Many of the scripts in toffy require a panel file, identifying which targets have been put on which masses. You can download your panel online from the Ionpath MibiTracker under the resources tab. In the panels section, open your panel and click Download csv.

panel download

You should then copy the file to the C:\\Users\\Customer.ION\\Documents\panel_files directory, and rename the file to be descriptive of your run. The toffy notebooks expect the panel files to be formatted slightly differently than the Ionpath default. The first time your panel is read in to one of the notebooks, it will be automatically modified by our scripts to contain the necessary information for toffy processing. This includes adding additional channels which are used for compensation, a full list of which can be found in the example panel file.

Median Pulse Height

The median pulse height (MPH) provides a way to assess the sensitivity of the detector, independent of the specific sample being acquired. It uses characteristics of the output from the detector itself to determine what fraction of maximum sensitivity the instrument is currently running at. We use this fraction of the maximum sensitivity to determine 1) when the detector needs to be swept again and 2) how much to normalize our images by after the fact the correct for this change in sensitivity. The minimum MPH required to still have acceptable signal will depend in part on the markers in your panel, the settings of the instrument, and other factors. However, we often find that the miniumum is somewhere between 5,000 and 6,000 MPH.

Development Notes

Requirements

Setup

  1. Clone the repo: git clone https://github.com/angelolab/toffy.git
  2. cd into toffy.
  3. Install the pre-commit hooks with pre-commit install --install-hooks
  4. Set up Poetry for toffy
    1. Run poetry install to install toffy into your virtual environment (this will create a local virtual environment).
      1. Poetry utilizes Python's Virtual Environments)
    2. Run poetry install --with test: Installs all the dependencies needed for tests (labeled under tool.poetry.group.test.dependencies)
    3. Run poetry install --with dev: Installs all the dependencies needed for development (labeled under tool.poetry.group.dev.dependencies)
    4. You may combine these as well with poetry install --with dev,test. Installing the base dependencies and the two optional groups.
  5. In order to test to see if Poetry is working properly, run poetry show --tree. This will output the dependency tree for the base dependencies (labeled under tool.poetry.dependencies).
  6. All tests can be run with poetry run pytest
  7. For every commit, pre-commit will format your changes to be complient with pycodestyle.

Questions?

If you have a general question or are having trouble with part of the repo, head to the discussions tab to get help. If you've found a bug with the codebase, first make sure there's not already an open issue, and if not, you can then open an issue describing the bug.

Before opening, please double check and see that someone else hasn't opened an issue for your question already.

How to cite

Please directly cite the toffy repo (https://github.com/angelolab/toffy) if it was a part of your analysis.