Artifact Suppression using Deep Neural Networks
Sinogram Inpainting Deep Neural Networks for artifact suppression in tomogram reconstructions
During the collection of tomography data (e.g. Electron Micrographs, X-ray CTs, etc.) multiple projections of an imaged sample are obtained for angles covering a 180 or 360 degree span. Once the projections are collected, it is possible to attain a construct a 3D image using an inverse Radon Transformation (see Figure 1). However, in specific cases either the tomogram’s projections cannot be collected for specific ranges of angles (therefore creating Missing-Wedges) or too few projections are collected to adequately sample the 180/360 degree span (Projection Undersampling). These missing projections can have adverse effects on the tomogram’s reconstruction, reducing its signal-to-noise ratio and also introducing undesirable visual artifacts. To correct for these adverse effects, we propose the use of Sinogram Inpainting Deep Neural Networks (DNNs) to inpaint (in other words, to predict and reintroduce) the uncollected projections in the tomogram’s sinograms (marked as red in Figure 2).
To train the Sinogram Inpainting DNNs, sinograms that are already affected by the Missing-Wedge or the Projection Undersampling problem are being used. The idea behind it, is to train tomogram-exclusive DNNs, that will learn how to inpaint a specific tomogram’s sinograms using the same Missing-Wedge or Projection Undersampling affected sinograms. This approach is chosen in an effort to prevent the introduction of hallucination artifacts (see Figure 3), that may occur by inpainting with a DNN trained on a different tomogram’s sinograms.
To train while using sinograms affected by the Missing-Wedge or Projection Undersampling problems, the sinograms are first split to multiple sinogram-patches. This splitting increases the number of training instances. Within these sinogram-patches, the areas which correspond to collected projections are split into two groups; the first group is given as input to the DNN, whilst the second group are artificially obscured (see Figure 4). The DNN’s goal during training is to offer inpainting predictions for the artificially obscured projections, and since for them the ground truth is available, a prediction error can be measured and the DNN can be trained. Once trained, it can be used to inpaint the uncollected projections of the sinograms.
Our motivation for our proposed Sinogram Inpainting DNNs is for the tomogram reconstructions made by the inpainted sinograms to not replace the reconstructions made by the original sinograms. Instead, the resulting tomogram reconstructions will act as auxiliary modalities.
Namely, the resulting tomogram reconstructions from the inpainted sinograms can be used to aid human annotators and/or particle picking software to locate a high number of the particles present in the tomograms and to accurately estimate their locations and orientations. Then using the tomogram reconstructions made by the original sinograms, subtomogram averaging may offer more accurate results by utilising the high number of particles located and the accurate knowledge of their locations and orientations.
Alternatively, in tomograms where subtomogram averaging is not applicable, the tomogram reconstructions made by the inpainted sinograms may help the analysis of these tomograms by offering a denoised and de-artifacted version of their reconstructions, devoid of externally introduced hallucination artifacts.
Below in Figures 5 and 6 some early sinogram inpainting results are being presented.
Dr Dimitrios Bellos – Research Software Engineer in the Artificial Intelligence and Informatics Theme
Dr Mark Basham – Head of the Artificial Intelligence and Informatics Theme
Professor Angus Kirkland – Science Director of the Correlated Imaging Theme
Dr Judy Kim – Deputy Theme Director of Correlated Imaging Theme
Dr Chen Huang – Electron Microscopy Development Scientist in the Correlated Imaging Theme
Dr James Parkhurst – Postdoctoral Research Associate in the Structural Biology Theme
Dr Maud Dumoux – Technology Lead for Cryo Imaging in the Structural Biology Theme
Dr Neville Yee – Research Software Engineer in the Artificial Intelligence and Informatics Theme
This project is being funded by the Rosalind Franklin Institute and Department of Materials, University of Oxford.