This is an analysis pipeline designed for freely moving behaviour experiment, which places tracking data into a unified framework with meta data.
Input: time series data from other softwares such as deeplabcut, sleap, bonsai (customised codes), blobmaster.
Output: time series data (single or multiple animals) + meta data in HDF file format
The instruction is based on the instruction in Armin Bahl's group and modified to run analysis on visual studio code
Install Windows 11 (optional)
Create a Github account
Ask Chi-Yu to add you on the contributor of this repository
Install Git
Use the default setting to install git and create a folder called "GitHub" (at maybe Documents, this will be handy later)
General anaconda environment for all setups and data analyses
Install the latest anaconda version for your operating system (https://www.anaconda.com/products/individual).
Open the anaconda prompt and create a virtual environment via conda
conda create --name tracking_analysis --channel conda-forge python=3.11
conda activate tracking_analysis
conda update -n base -c defaults conda
conda config --add channels conda-forge
conda config --set channel_priority strict
conda install --yes -c conda-forge -v h5py opencv ipython jupyter matplotlib pandas matplotlib scipy numpy openpyxl
pip install roipoly lsq-ellipse opencv-contrib-python==4.5.5.64
feel free to install other packages but the abovementioned packages should be enough to do most of the analysis in the lab
For example, I am trying to use multiprocessing on jupyter notebook so I installed,
pip install ipyparallel
But bare in mind that Armin once said
We try to have everything in one conda channel and avoid pip as much as possible. This should help building a healthy dependency tree
Install Hdf5view
Important software to check the structure of behavior data and imaging data files. https://www.hdfgroup.org/downloads/hdfview/
Make some acount and download. As I used window, the file I download is somehow like HDFView-3.1.3-win10_64-vs16.zip
Install Visual Studio Code
Install extension on Visual Studio Code: press ctrl + shift + x or click on Extensions when you are in the VS code window, install Python, Code Runner, Code Spell Checker.
The following extension are optional depends on your need
C#: useful to create customise bonsai node or workflow
Github copilot: sounds useful but never used it
Remote SSH: if you want to remote in the PC in the lab from your own PC somewhere else (doable, but it needs MPIAB VPN and some additional SSH configuration)
If you want to use VCC's GPU cluster, you need to install WSL.
Note that WSL does not share all of the extensions from Windows so you need to check if abovementioned extensions are installed
And then inside the WSL environment, you install Kubernetes
And any kind of formatters VS code suggest you to install should not be harmful for your coding environment
Clone this repository on Visual Studio Code
On this page, click Code above the repository and then At the Clone section, copy the https path.
On the VS code window, go to source control and then click clone repository and paste the https path there and press enter
Cloning the main branch but
do all the changes on your own branch and make committ and push only on that branch
under your own branch. This can be done by going to bottom left of VS code window and click main and then select your branch or
git checkout YOUR_BRANCH
And then pull the update from the main or from other brance before solving conflict and merging the two together and finally pushing the updated branch to github
git pull origin main #or git pull origin YOUR_BRANCH if you want to get updates from your own branch
git merge main
git push origin YOUR_BRANCH
When you want to merge your branch with the main branch, setting up a meeting with cllaobrators of this repository to discuss what functions/lines to keep and discard
Start coding
The simplist way to configure the python environment is to open VS code on your virtual environment.
This can be done by opening the anaconda prompt
conda activate tracking_analysis
code .
If this command fails, check if the directory of VS code, eg. YOURCOMPUTER\AppData\Local\Programs\Microsoft VS Code\bin, is added in the System Properties > Environment Variables > User Variables > Path
database format: each animal has its own folder under a parent folder. For example, I named a folder as SN23101, representing Solitarious Nymph in the year 2023 no. 101.
Inside the folder, there is a subfolder for each behavioural assay and inside that subfolder, you will find raw data
and curated data. The name of these folders do not really matter as long as the python dictionary, named as
analysis_methods can locate them. We used analysis_methods_dictionary.json to manage analysis_methods. json file is good as it can be opened via various languages.
First sets of code to use:
create_analysis_methods.py to create a analysis_methods_dictionary.json file. It overwrite the existing json file each time.
freely_moving_analysis.py to analyse raw data from Bonsai workflow meta_workflow and save them as a curated_dataset. We tried to save the output of analysis in either csv or h5 file so that other computer language can access them.
ROI_analysis.py input: curated_datase and then output: preference index, travel distance, and proportion of time around corner
Codes still need improvement:
freely_moving_analysis.py need to make sure the Trex tracking can handle more than 4 animals
ROI_analysis.py I still want to add some classifier to tell if animal is walking or not across time (also in the future if they are grooming or not)
if you are not familiar wit the coding, use tutorial.ipynb, which includes the essence of freely_moving_analysis.py, ROI_analysis.py and ploting some group data
lastly, how to make notes on github https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax