2DKD: a toolkit for content-based local image search

Background Direct comparison of 2D images is computationally inefficient due to the need for translation, rotation, and scaling of the images to evaluate their similarity. In many biological applications, such as digital pathology and cryo-EM, often identifying specific local regions of images is of particular interest. Therefore, finding invariant descriptors that can efficiently retrieve local image patches or subimages becomes necessary. Results We present a software package called Two-Dimensional Krawtchouk Descriptors that allows to perform local subimage search in 2D images. The new toolkit uses only a small number of invariant descriptors per image for efficient local image retrieval. This enables querying an image and comparing similar patterns locally across a potentially large database. We show that these descriptors appear to be useful for searching local patterns or small particles in images and demonstrate some test cases that can be helpful for both assembly software developers and their users. Conclusions Local image comparison and subimage search can prove cumbersome in both computational complexity and runtime, due to factors such as the rotation, scaling, and translation of the object in question. By using the 2DKD toolkit, relatively few descriptors are developed to describe a given image, and this can be achieved with minimal memory usage.


Background
Moment-based approaches are very useful for representing biological and medical images as they are pixelized [1] or voxelized data [2][3][4]. In medical imaging, such as computerized tomography (CT) scan and magnetic resonance imaging (MRI), objects are observed at different viewpoints and local images need to be extracted and examined. In digital pathology, for instance, pathologists are interested in information about specific structures rather than the whole image [5]. Thus, it is necessary to construct moment invariants that do not change by translation, rotation, and scaling and can efficiently retrieve local image patches or subimages.
Here we present the software package 2DKD, twodimensional Krawtchouk descriptors, for local compari-*Correspondence: atilla.sit@eku.edu 1 Department of Mathematics and Statistics, Eastern Kentucky University, 521 Lancaster Ave., 40475 Richmond, KY, USA Full list of author information is available at the end of the article son of 2D images. The mathematical formulation of 2DKD was already established in [1], which brings in the following advantages: 1) Krawtchouk polynomials are defined on a discrete space, so the moments derived from them do not carry any error due to discretization. 2) These polynomials are orthogonal; each moment extracts a new feature of the image, where minimum redundancy is critical in their discriminative performance. 3) They are complete with a finite number of functions (equal to the image size), while many other polynomial spaces have infinitely many members. 4) They have the ability to retrieve local image patches by only changing the resolution of reconstruction and using low order moments. 5) The location of the patch can also be controlled by changing two parameters and hence shifting the region-of-interest along each dimension [6]. 6) These moments can be transformed into local descriptors, which are invariant under translation, rotation, and scaling [1].
2DKD also has the potential to be used in cryoelectron microscopy imaging (cryo-EM), in particular, single-particle cryo-EM. This method generates a 3D reconstruction of the structure by combining data from many 2D projection images, in which identical copies of a protein complex are found in different orientations [7]. From the images of fields containing a large number of molecular complexes, individual particles need to be selected manually or by automated algorithms for further image processing. In addition to the acquisition of high-quality projection images of the particles, fast and accurate particle selection is also critical to ensure a highresolution 3D reconstruction of structures [8]. We test the software 2DKD by applying it to particle selection in a 2D projection image of GroEL complexes obtained using cryo-EM.
The recognition accuracy of 2DKD was tested in [1] and compared to the traditional Hu invariants on two different datasets, a dataset of binary images and another one with gray-scale clip art images. The comparisons were made based on the top-ranked hits, where the Euclidean distance was used as the similarity measure between two descriptor vectors. Overall, 2DKD showed better prediction accuracies than Hu invariants. The descriptors in [1] were only tested up to 4% noise. Here, we introduce a more stable version of 2DKD, which shows tolerance up to 30% noise in the image data.

Workflow
The workflow of 2DKD software is shown in Fig. 1. For a given query image and the pixel location (x p , y p ) of a point-of-interest on the image data, 2DKD performs the following six functions.
its geometric momentsM 00 ,M 10 , andM 01 , the center of mass (x,ỹ), and the central momentsμ 20 , μ 02 , andμ 11 off (x, y). It then finds the unique anglẽ θ between the principal axis of the auxiliary imagẽ f (x, y) and the x-axis of the 2D plane. This angle is critical for building the rotation invariant descriptors. The exact computation ofθ is provided in [9]. Using M 00 ,x,ỹ, andθ , this script calculates the geometric invariantsλ ij for i, j = 0, 1, 2, 3 using the formula provided in [1]. We finally compute the 2DKD using for n, m = 0, 1, 2, 3 and p x = p y = 0.5. The descriptorsQ 00 ,Q 01 ,Q 10 , andQ 11 are removed because they take a constant value irrespective of the region-of-interest we are working with. In this work, we use the 2DKD of order up to 3, that is, Usage example: It scans each image in the database by computing the 2DKD for each point-of-interest location and saves the descriptors with the image number and the location of the subimage in that image. The result is stored in a potentially large matrix with rows of the form < Image number, x p , y p , V > for easy access later when a subimage is queried. Note that unless there is a change in the database, this only needs to be run once offline to save computational time. 6. dbSearch : dbSearch is another high-level script and is used to search the output of dbIndex for descriptors similar to the ones corresponding the query. A query image is supplied as input, then compDesc is run on the query, producing descriptors for it, and then the matrix from dbIndex is sorted by Euclidean distance of descriptors to the new ones obtained, giving a

Results
In this section, we present some experimental results and evaluate the discriminative power of 2DKD. For each point of interest (x p , y p ) corresponding to a subimage, we compute and use the feature vector V given in (3). To compare the descriptors for a query with those for subimages in a database, we use the squared Euclidean distance as a similarity measure, namely ( 4 )

Experiment I
To construct the first database, we use nine clip art icons that are downloaded from Microsoft Office Online. These images are shown in Fig. 2. They are transformed to 60 × 60 gray-scale images and placed in the center of a 150 × 150 frame to be used as queries. The same set of gray-scale images are also used to generate a database. These images are rotated by the angles and scaled by the factors s = 0.8, 0.9, 1.0, to obtain a set of 9×12×3 = 324 subimages. These subimages are randomly placed in 81 positions to form an image of size 600 × 600. In this experiment, four such images are generated, one of which is shown in Fig. 3.
We run dbIndex to produce descriptors for all subimages in the database so a query can be compared with them. After computing the norms, coefficients, and the central weight for S = 150, dbIndex scans each image in the database by trimming a 150 × 150 region from the image containing each point-of-interest, computes 2DKD for each corresponding subimage, and saves them with the image number and the location of the subimage in that image.
From Table 1, it is clear that 2DKD correctly matches the query subimage with the subimages in the dataset successfully with 100% accuracy if we consider the top-ranked hit and 93.3% accuracy when we look at the top 5 hits in the dataset.
We also tested 2DKD for searching for subimages in the salt-and-pepper noise degraded version of the dataset, with noise densities 10%, 20%, and 30%. The results are summarized in Table 2. Considering only top-ranked hits, our descriptors show tolerance up to 30% noise with only one miss. Among top 5 results, it shows 91.1% accuracy with 10% noise, whereas it decreases to 77.8% with 20% noise, and to 71.1% with 30% noise. Three example queries and corresponding top 5 retrievals from the dataset with 30% noise are shown in Fig. 4.

Experiment II
Next, we test the local search performance of 2DKD on a more realistic problem, particle selection in 2D projection images of cryo-EM. In single-particle cryo-EM, these projection images contain identical copies of a protein complex in different orientations. One such example is GroEL, a molecular chaperonin found in a large number of bacteria [10]. An example projection image of GroEL protein complexes is shown in Fig. 5a. From these images, individual particles need to be selected by hand or by automated algorithms. Once selected, they are sorted based on variations on their structural features. Similar images are then averaged to obtain representative projection views of the complex at much higher signal-to-noise ratios than in the original images (see Figs. 5b and c.) Finally, the 3D Fourier transform is built up from a collection of 2D images spanning a complete range of orientations and used to recover the 3D structure of the complex via inverse Fourier transform (see Fig. 5d) [7]. Thus, selection accuracy and speed in particle selection are highly important to increase the resolution of reconstructed 3D structures.
We run the script, dbIndex, as in Experiment I to produce descriptors for all subimages in a 1024 × 1024 projection image (a section of which is shown in Fig. 5a) so a query can be compared with them. The image is very noisy, and there are many flat regions or regions inbetween-particles that should not be considered. We compute the local variance of pixel densities of each 40 × 40 subregion and compare it against the global pixel density variance. The subregion centers corresponding to lower value of local variances are not indexed. This way, we ensure that we keep the regions with high visibility and discard those with undesired particles. Then    Fig. 6. As justified by the figure, most of the retrievals from global image visually match the query except only three of them: the eleventh, thirteenth, and fourteenth. In this experiment, we only search within one image, but the code can be easily adapted to handle a database with multiple projection images. Table 3 shows the average times taken for computing 2DKD and using them for database indexing and searching. The programs were run 100 times for each task, and the average times were recorded. For each experiment, the programs were tested on a Windows computer with Intel Core i7-8650U processor of 1.90 GHz and 16 GB memory using GNU Octave, version 5.1.0. The table shows that the average time for computing 2DKD of a typical subimage is in the order of 10 −3 , which allows the database indexing to finish in a reasonable amount of time (within a second to under a minute). Assuming that the descriptors were precomputed and stored, the search can be performed in real-time, which makes the software promising for larger datasets.

Conclusions
Searching biological images for local patterns or specific structures can be computationally challenging due to very low signal-to-noise ratio of these images and the limited number of efficient local invariant descriptors available to perform such searches. We developed 2DKD to address these issues and be used for potentially large biological image databases. 2DKD is developed in Octave (open source) and is publicly available at GitHub website. The source codes can be readily applied to image databases in other fields as well.