Scalable and Automated AI Image Analysis for Volume Electron Microscopy

ZEISS arivis Cloud Deep Neural Network Training

The arivis Cloud platform uses a U-net convolutional neural network for deep learning training using a representative sub-set of the images and hand-drawn segments (‘ground-truth’) that have been provided by the user. The number of segment types sets the number of model classes. To generate a ground truth class, the user paints instances of a class within the image. Here, mitochondria and the nucleus were painted within the cell as individual classes to be used for the training (Figure 1B-C).

Our (sparse) training sets comprised just 73 planes from the top of the volume to the bottom, spaced every 10 planes. We also trained on a minimal number of ground truth objects for each model. The nucleus model training used 53 objects (mostly full nucleus and some partial annotations) and the mitochondria model training used 1,133 objects (the majority were full mitochondria). Occasionally all the objects were painted in a plane, but most of the time only sub-portions of planes were painted.

Note, we also attempted to segment nuclear pores and proxy objects directly via deep learning. The former fails because representing the many orientations of pores via manual 2D painting is not feasible. For the latter, we trained a network by using 688 objects. The result was highly biased in producing objects with more completeness in the XY orientation, which makes for unreliable 3D processing and measurements (due to errors in XZ and YZ).

Neural network models for the mitochondria and the nucleus were trained individually, resulting in two separate deep learning models (.CZANN). These models were then run on the entire image set and are applicable to any comparable image sets to predict all instances of each feature class.

Segmentation and measurements of organelles

The first step in our cell-profiling workflow was to use the neural network models from our arivis Cloud training to automate measurement of organelle volume. The objects produced by our arivis Cloud models for the mitochondria and nucleus required several improvements by both ZEISS arivis Pro Analysis Pipeline operations and manual proof-editing. This end products of the segmentation workflow are objects representing the volumes of mitochondria and the nucleus (Figure 2A).

ZEISS arivis Pro computes the volume for all 3D Objects, which made it easy to calculate the percentage of total cell volume occupied by each organelle (Figure 2C). Our profiling was consistent with previous measurements, which have shown that mitochondrial volume is on average ~10% of the cytoplasm volume within HeLa cells (Posakony et al. 1977)².

We then set up analysis pipelines in ZEISS arivis Pro to compute the distances of mitochondria to these cellular structures. Taking the distances of each mitochondrion’s center of geometry to the nuclear membrane (Figure 3C) or the plasma membrane (Figure 3D) did not result in significant correlations. However, when combining these two membranes to measure the minimum distance of each mitochondrial center of geometry to either membrane showed a significant correlation (Figure 3E).

These measurements highlight the ability of this approach to profile multiple distances between cell organelles and identify significant correlations. This method can be used with any cell structures that have been segmented and can measure distances between object surfaces or centers of geometry. Moreover, this approach can be scaled using the arivis VisionHub, so that multiple cell image sets can be analyzed in parallel and produce automated, high-quality profiles.

3D segmentation and distribution analysis of nuclear pore complex regions

The accuracy of our whole nucleus and nuclear membrane masks enabled us to use the volumetric rendering and clipping tools in ZEISS arivis Pro to explore the nucleus and nucleus-associated structures in 3D. Struck with how well we could see the nuclear pores in this volume, we wondered whether we could use arivis Cloud and ZEISS arivis Pro to segment and measure them.

However, within individual 2D planes the nuclear pores are difficult to recognize. The resolution of the image provides only a 100-150 voxels per pore (for reference, the total pixels in the image is more than 1 billion) and the 3D structure of each pore is uniquely oriented to the curvature of the nuclear membrane. Thus, a direct, traditional 2D deep learning approach would require extremely tedious annotating of NPCs in all possible orientations and have to cover all variability in sample preparation and image acquisition. We decided to take advantage of relatively large pockets under the pores, which we discovered were in a 1:1 stratified relationship with pores throughout the nucleus and can be segmented in 3D. These pockets are regions with presumably less chromatin density, more transcripts, mobile protein complexes, etc.

We used these objects as proxies to view and quantify the 3D distribution of NPCs throughout the nuclear membrane. To accomplish this, we used the ZEISS arivis Pro python application program interface (API) to make a custom python operator integrated with the arivis Pro software. Specifically, this python operator takes the 3D centroids of all the objects and uses the kernel density function within the scikit-learn python library to calculate their 3D densities. This kernel density function calculates a score based on the number of other objects in proximity, with a Gaussian smoothing of scores over a given radius. This custom python operator then exported the resulting density scores as a numeric feature of these objects within arivis Pro. Color-coding these objects according to their density score permitted visual assessment of the distribution of these scores (Figure 5A).

Taking the density scores of these two sections highlights that NPC density is higher within the smaller section of the nucleus with higher curvature. In contrast, the larger section with a lower curvature degree has more low-density regions for nuclear pores. Overall, this indicates that there is variation in our measured nuclear pore density over different portions of the nucleus.

Making a new image mask from these objects highlights the nuclear pore complexes visualized on Figure 6F. While not as precise a result as we would get from our original concept, this worked remarkably well and with manual curation we will be able to create ground truth annotations for many thousands of pores for the subsequent deep learning training.