Novel Transformer based methods for Automatic Classification and Quantification of Stromule Dynamics from Microscopy Images

Submission Number: 166
Submission ID: 3784
Submission UUID: 22681e46-4673-45bc-8535-adb4b5d77add
Submission URI: /form/project

Created: Mon, 06/19/2023 - 18:34
Completed: Mon, 06/19/2023 - 18:46
Changed: Mon, 03/18/2024 - 15:58

Remote IP address: 98.43.37.78
Submitted by: Anita Schwartz
Language: English

Is draft: No
Webform: Project
Novel Transformer based methods for Automatic Classification and Quantification of Stromule Dynamics from Microscopy Images
CAREERS
NTACQStromuleDynamicsMI.PNG
bioinformatics (277)
Complete

Project Leader

Chandra Kambhamettu
{Empty}
{Empty}

Project Personnel

Jeffrey Caplan
Huining Liang
{Empty}

Project Information

Our work involves developing an image processing pipeline for automatically classifying and quantifying the dynamics of chloroplasts, stromules, and the plant’s cytoskeleton to better understand the function, relationships, and movement characteristics of these intracellular structures. Our current pipeline consists of fast, automatic segmentation of microscopy images, active contour-based tracking, and unsupervised movement classification based on a U-Net, a convolutional neural network (CNN) for segmentation, and Computer vision methods for tracking.

For the CAREERS project, we propose to undertake and complete a transformer based pipeline, more specifically, TransUNet to leverage the power of CNNs and transformers for the image segmentation task. The proposed TransUNet involves CNN-Transformer Hybrid Encoder, Patch Embedding and Cascaded Upsampler. For tracking, we propose to use the tracking-by-attention paradigm which not only applies attention for data association but jointly performs tracking and detection, using TrackFormer, an end-to-end trainable MOT (multi-object tracking) approach based on an encoder-decoder Transformer architecture.

Project Information Subsection

(1) Transformer based segmentation system
(2) Transformer based tracking system

{Empty}
- Grad or undergrad: Grad
- Interested in structural biology research
- Experienced Linux or Unix user
- Comfortable working in a remote Linux environment (HPC cluster): Yes
- Some experience with Python programming
- Structural modeling experience (understanding general concepts) will be helpful
- Familiarity with machine learning concepts will be helpful
{Empty}
Can work with any level
{Empty}
University of Delaware
Newark, Delaware. 19716
CR-University of Delaware
06/01/2023
Yes
Already behind3Start date is flexible
6 months
09/13/2023
01/19/2024
  • Milestone Title: Beginning
    Milestone Description: Learn the basics of Transformers and set up TransUNet to work on Microscopy images, previously trained on UNet. TransUNet will also be run on other generic datasets to ensure the version works correctly. Give a Launch Presentation.
    Completion Date Goal: 2023-07-01
    Actual Completion Date: 2023-09-13
  • Milestone Title: Middle
    Milestone Description: This will entail the completion of the segmentation task using TransUNet, and comparison to the previous UNet version results in terms of computational cost, memory, and accuracy. The algorithm design for the tracking task using TrackFormer will be started using transformers.
    Completion Date Goal: 2023-09-01
    Actual Completion Date: 2023-12-01
  • Milestone Title: End
    Milestone Description: TrackFormer, an end-to-end trainable multi-object tracker will be trained for the tracking task. It will be first tested on generic data before applying to the microscopy imagery. Results will be compared in terms of efficiency and accuracy to the previous Computer vision based approach that we have developed. Give a Wrap-up presentation.
    Completion Date Goal: 2023-11-02
    Actual Completion Date: 2024-01-19
{Empty}
{Empty}
Possible CVPR Workshop paper
Transformers technology in Deep learning
{Empty}
Effort involved in recruiting and training
{Empty}
{Empty}

Final Report

Huining Liang has worked on a new deep-learning Transformer-based segmentation technique. Since no one has applied it to stromule data before, this helps plant science researchers achieve a higher order, high throughput multiplexing. Huining has also applied Transformers for tracking. The overall impact includes the improvement over the previous pipeline, which involves UNet and traditional Computer vision-based methods. The current pipeline achieved higher accuracy than the earlier work.
Dr. Kambhamettu directs the Video/Image Modeling and Synthesis (VIMS) Lab, which has Ten PhD students working on the deep learning approaches. The approaches developed in this project open a way to incorporate some of the concepts in other projects.
Dr. Caplan directs a Bio-Imaging Center that is used by 19 different departments at the University of Delaware, spanning a wide array of disciplines. The approaches developed in this project can be translated to other projects in the network.
Nothing to report.
Yes, Huining Liang will use this training in her Ph.D. work and to assist others in research computing, therefore, adding to our human resource infrastructure.
Nothing to report.
In VIMS Lab, this project helps update the repository of techniques impacted by Huining’s work under CAREERS. It now includes transformer-based techniques as a contribution to this repository.
Nothing to report from PI and Mentor, but as far as Student Facilitator see notes taken during exit interview.

During Huining's exit interview she relayed the following that is very impactful about CAREERS and her experience:
(1) Because this was her second project, she felt she was given more responsibility to take the lead versus her first project which required more guidance and decision making for the overall project. This was confirmed by the PI and Mentor during their exit interviews as well that Huining learned so much after the first project in CAREERS, they felt she was capable of taking ownership and the lead as they felt comfortable she would ask them for help when needed.
(2) She appreciated the format change during the monthly meetings to require students to make an elevator pitch about their research/project in 2-3 mins before giving an update. She found this skill (elevator pitch) to be invaluable to think about explaining to your parents or anyone not in your field what you are doing, but it also allowed her to easily understand what other students were doing and to be able to find connections or possible networking opportunities in different domains using similar methodology/technology.
The project that Huining Liang worked on developed the use of a latest deep learning technique and compared against the previous pipeline, and achieved improved results. It will assist her future innovation in deep learning, and also will assist a Plant Genome Research Program looking at maize development for crop management. Thus, this project may potentially benefit crop production and food security which is a major societal impact.
* Explored Transformer based models for segmentation, detection and tracking on Microscopy Images.
* Improve and evaluate a working pipeline in research.
* Facilitate research with machine learning methods and HPC resources
* Trained the TransUNet model for segmentation of microscopy images.
* Compared with the previous UNet version in terms of computational cost, memory, and accuracy.
* Trained the TrackFormer for Stromule detection and tracking, and compared the tracking result with previous computer vision based approach.