Extracting low-level features in content based image retrieval systems using open CV environment

EXTRACTING LOW LEVEL FEATURES IN CONTENT BASED IMAGE RETRIEVAL SYSTEMS USING OPEN CV ENVIRONMENT

Abstract—Feature extraction is a significant and also the initial phase of Content Based Image Retrieval (CBIR) where the features of an image are extracted using a particular technique and thereafter stored in the form of a feature vector. This work basically shows the feature extraction of images using OPEN CV platform. OPEN CV is a library of programming functions mainly aimed at real-time computer vision. We have used its inbuilt functions and libraries and extracted the features that include key points, contours and texture of images. The output to the feature extraction can be taken as the input at the next level of CBIR.

Keywords—Feature extraction,SIFT,SURF,OPEN CV.

I. INTRODUCTION

The Content based image retrieval is a process of fetching the image from a database of image based on the query produced by the user. It is composed of various phases that include feature extraction, similarity matching and indexing [4].Feature extraction is one of the important phases of CBIR without which the whole process can be considered to be impossible. At this phase features that comprises of the data or content of the image are extracted and stored for taking as an input to the next phase. Figure 1 shows the various phases of CBIR where being the emphasis of our work , the feature extraction phase is highlighted.

There are several techniques followed under feature extraction where SIFT and SURF is one of them as discussed in this work. For the implementation part OPEN CV platform is put into use. It is an environment provided for image processing implementation using C++ interface. Open CV (Open Source Computer Vision Library) is a library of programming functions mainly aimed at real-time computer vision, developed by Intel, and now supported by Willow Garage and Itseez [1]. The main goal behind the launch of OPEN CV was

1. PHASES OF CBIR

to get a portable and an optimised code with programming library functions available for free. OPEN CV was developed by Intel Microprocessor Research Lab which was distributed under a BSD style license and allows free commercial or research use. It is supported under Windows and Linux, but it is also portable at many other operating systems. It is a open source computer vision library in C or C++ and optimized for real time applications. The paper is organized into various sections: Section II explains the feature extraction and further the Sub Section A types of features as broadly classified into level 1, level 2 and level 3 features. In the section III the two major techniques under feature extraction process namely SIFT and SURF are covered. The next section shows the implementation phase of our work.

II. FEATURE EXTRACTION

This is an initial phase of CBIR where the features stored in an image are extracted and further stored in a feature vector. Features are basically the data or content stored in an image that describes the image. This process transforms the content of the images into the form of content features. The features thus extracted are utilized further in selection and classification phase [2].We can consider the image to be a two dimensional array taken as{F(x,y):x=1,2,…..,X, y=1,2,….,Y}.

For colored images (x,y) is taken as color value at pixel and that for gray scale it is taken as grayscale intensity value of that pixel[2].

Feature extraction phase performs the grid creation used for the identification of the features on an array and then computes the intensity value for each feature [3].In CBIR systems the images are stored at their corresponding indices [4].After the feature extraction process the features are stored using the feature vector as a data structure and therefore retrieval of images is done by performing the similarity matching of these feature vectors [4].The distance ‘D’ calculated between the feature vectors of query image ‘Q’ and image database ‘T’ is defined as,

D (Feature (Q), Feature (T)) ≤ t(1)

A. Types of Features

Features are basically the data or the content stored by the image that gives a proper description of the image. They are functions defined for one or more measurements taken for the certain characteristics of the object [4].The features are represented using the feature descriptors. Features are of following types:

1) Level 1 features: These are basically based on the whole image or on the part of an image as well.

• Color: The color features are extracted and studied by using the color histograms. Here the matching is done on the basis of the color histograms which resemble to the color histogram of the query image most closely [6].Other representations for color feature are color moments, color correlograms, coherence vectors [8].
• Texture: It is a powerful regional descriptor that supports the image retrieval process. It helps in classification, recognition and finding the similarities between the images from multimedia databases [2].Similarity done on the basis of texture helps in distinguishing between areas of images with similar color [6].Texture can be represented using structural that represent texture as texels (texture element, or texture pixel ) that are arranged regularly on surface based on particular arrangement [8] and statistical methods that includes Fourier power spectra, co-occurrence matrices, shift-invariant principal component analysis (SPCA), Tamura features, Wold decomposition, Markov random field, fractal model, and multi-resolution filtering techniques such as Gabor and wavelet transform, etc[2].
• Shape: Shape features comprises of aspect ratio, circularity, Fourier descriptors, moment invariants, consecutive boundary segments, etc. Shape descriptors are used for the representation of shape features [7].
• Spatial location: It is used in region classification. This particular feature may be used to differentiate when color and texture features are same. It is defined as upper, bottom, top according to the location of the region in an image [7].

2) Level 2 features: These are also known as high level features. They give a semantic or a rational meaning to the features of an image. There is a semantic gap between low level and high level features [9] which is defined as a space of disappointment between the high level features of CBIR and low level features that are used for query purpose. The low level features can be translated into their semantic interpretation by using various techniques. The semantic information basically extracts the logic of the low level features like color, texture etc as given by the user and processes it further and finally sets a relevance feedback for the user and concludes whether the fetched output is as desired or not. The relevance feedback is obtained in several iterations. It basically reduces the semantic gap in CBIR systems [10].Following are some major techniques under representation of level 2 features:

• SFL-Semantic Feature Layer: This technique is basically a design semantically related classes containing low level features and also includes additional knowledge like modeling information, Domain knowledge etc. This approach is implemented for high level features describing human world properties and for evaluation of 300 queries [9].
• Fuzzy production rules: This technique takes the extracts low level features like contrast of hue, light-dark contract, shape features and texture features and convert them into high level semantic features using fuzzy production rules those are derived from an image mining technique. The list of structures containing the content for the high level semantic features is created on the basis of Dempster-Shafer theory [11].
• Semantic grouping algorithm: It is a statistical algorithm for keyword annotations based on user feedback. This is an intelligent system that can gradually learn the user’s searching habits in terms of semantic relations among concepts and further uses this information for improvement of the retrieval performances [12].

3) Level 3 features: These features are used for automatic image retrieval [6]. These are used to study the semantic relations of user’s search. Here semantic grouping of keywords is done using a particular algorithm. Keywords are generated from textual and manual annotations of images [12].

III. TECHNIQUES FOR FEATURE EXTRACTION

There are two famous techniques used under feature extraction named SIFT and SURF.

A. SURF

SURF stands for Speeded up Robust Features. It is an algorithm following multiple stages and thus speeds up the feature extraction process in-case of images. The SURF descriptor is based upon the Hessian based scale-space pyramid for searching of image feature points [13].It is a method that is perform ant scale and rotation-invariant interest point detector and descriptor. It is much better than other feature extraction techniques in terms of repeatability, distinctiveness, speed as well as robustness. It is better since it depends upon integral images for image convolutions [14].It is the speeded up version of SIFT feature extraction algorithm. Surf is three times faster than SIFT and same can be viewed in terms of performance also [15].It works well for the images with blurring and rotation but does not succeed well in handling [16].

SURF also helps in automatic image annotation where it selects the correct number of features and the features itself for annotation [16].Medical image annotation can also be done using SURF descriptor and the SVM classifier. There is a Fast-Hessian detector used to perform feature extraction and feature matching is performed using SVM with a quadratic kernel and on comparison with SVM classification it resulted into the improved classification of lung images with 96% accuracy. It is counted as a strong tool for medical image annotation [17].

B. SIFT

SIFT stands for Scale Invariant Feature Transform. It was proposed by David Lowe in 1999.Its name conveys its meaning as the content of the image is converted to scale invariant coordinates relative to local features [18].It is one of the techniques in image processing and computer vision that is used to capture the scene semantics [19].The major steps covered under SIFT algorithm are as follows [20]:

1. Find scale space extrema.
2. Key point localization and filtering.

• Improve key points and throw out bad ones.

3. Orientation assignment.

• Remove effects of rotation and scale.

4. Create descriptor.

• Using histograms of orientations.

First two steps are the parts of Detector and last two are part of Descriptor [21].The algorithm has points or features that are local minima and maxima in a scale space and further composed of differences of Gaussians also known as DOG. Normalized histograms of the gradient directions around those points are used for describing them [21].The visual contents of a query image and database images can be extracted and further described using 128-dimensional SIFT feature vectors. Here indexing and matching of these SIFT features can be done using KD-tree and Approximate Nearest Neighbor (ANN) Search algorithms [22].

2. SYSTEM ARCHITECTURE

IV. IMPLEMENTATION

This section shows the implementation part of our work where we have extracted some common features from two test images using OPEN CV platform. It is an environment that provides c++ interface for image processing algorithms. The OPEN CV library is used by a large number of companies like Intel, IBM, Microsoft, SONY, Siemens, Google, and research centres Stanford, MIT, CMU, Cambridge, INRIA etc [23]. It is a open source computer vision library in C or C++ and optimized for real time applications. Figure 2 shows the

system architecture of the work that has been done. The libraries and files can be downloaded and installed from [25].

1. We load two different images taken in jpeg format.
2. Extract features using SURF in OPENCV.
3. Display the different images where three different features are highlighted separately.

There are three basic features extracted:

1) Key points: Key points are basically those points in an object of a particular image that give the description of features in that object of that image. Such a description can be used to locate that object in an image containing many other objects [24] . The key points are detected using SURF detector. The key points are detected and its descriptors are calculated and finally those extracted key points are stored in a vector array. After this the key points are drawn and then displayed in the form of a new image.There are various other algorithms available in OPEN CV for feature detection like SIFT, ORB, FAST.

2) Contours: The contours are the outlines of an image. It includes edges and boundaries. They are extracted by declaring and defining a function in OPEN CV that detects the edges of the image using canny edge detector. Then the contours are found and drawn from an initial point up to a final point thus creating a hierarchy of points. Hierarchy is a vector that contains the information about the image topology. [25].Finally, the contours are shown in the form of a new image.

3) Texture: The texture is extracted by using al algorithm provided in OPEN CV.As a result the textured data will be printed on the new image. Therefore, as an output we get texture of the original image in the form of a new image.

V. EXPERIMENT RESULTS

We first laod the images and display them using the imshow function of OPEN CV.The images taken are gray scale. Then we have extracted the features of both the images simultaneosly one after the other and these feeatures thus extracted are stored in a feature vectors.Each feature is stored in its corresponding feature vectyor.Figure 3 shows the two test images taken whose features are to be taken and below them there corresponding extracted features are shown in series where at first keypoints are shown,next are the contours and at the end we have extracted the texture.The images store a lot of content and are complex to store since there are a number of matrices that are formedSo, the hardware configurations do also matter while implementing image processing algorithms.

The output of the feature extraction phase can be taken as the input to the next phases of CBIR that may include indexing and similarity matching.

VI. CONCLUSION

The paper shows the most significant phase of CBIR that is feature extraction and how we have extracted out the features using OPEN CV platform. The features include key points, contours and texture. The extracted features are stored in a feature vector and further can be taken as an input to the next phases of CBIR. This is a sequential implementation and to obtain better speed ups and effective execution time we would perform it in parallel using Graphics Processing Unit and CUDA.

3. RESULTS

ACKNOWLEDGMENT

The authors would like to thank Dr.Satvir Singh and Mr. Sarabjit Singh from Shaheed Bhagat Singh,State Technical Campus,,Ferozepur, India for their valuable assistance in formulating this work.

REFERENCES

1. http://en.wikipedia.org/wiki/OpenCV
2. Image Feature Extraction Techniques and Their Applications for CBIR and Biometrics Systems, Ryszard S. Chora´s, INTERNATIONAL JOURNAL OF BIOLOGY AND BIOMEDICAL ENGINEERING,2007.
3. Feature Extraction and Normalization Algorithms for high density Oligonucleotide Gene Expression Array Data,Erich E Schadt,Cheng Li,Journal of Cellular Biochemistry Supplement,2001.
4. Content based Image Retrieval with Graphical Processing Unit,Bhavneet Kaur,Sonika Jindal,ITC 2014,2014.
5. Feature Extraction with Graphics Processing Unit, Bhavneet Kaur, Sonika Jindal,ICAET 2014,Under the process of publacation,2014.
6. Content based image retrieval,John Eakins,Margret Graham,1999.
7. A suvey of content-based image retrieval with high level semantics,Ying Liu,Dengsheng Zhang,Elsevier,2006.
8. Content-based Image Retrieval: Feature Extraction Techniques and Applications, Amandeep Khokher, Rajneesh Talwar, iRAFIT2012.
9. Semantic Feature Layers in Content Based Image Retrieval:Implementation of Human World Features,Horst Eidenberger.
10. Relevance Feedback: A Power Tool for Interactive Content-Based Image Retrieval, Rui, Y., Huang, T., Ortega, M. and Mehrotra, IEEE Transactions on Circuits and Systems for Video Technology,1998.
11. Content Based Image Retrieval Using High Level Semantic,Prof. Sharvari Tmane,2008
12. CBIR: From Low-Level Features to High-Level Semantics,Xiang Sean Zhou*, Thomas S. Huang.
13. Fast Scale Invariant Feature Detection and Matching on Programmable Graphics Hardware, Nico Cornelis, Luc Van Gool.
14. http://www.vision.ee.ethz.ch/~surf/
15. www.opencv.org/Surf/
16. Automatic image annotation using SURF descriptors, Muhammed, A.V. ,Kumar, G.S. ,Sreeraj, M. ,IEEE,2012.
17. ANNOTATION OF MEDICAL IMAGES USING THE SURF DESCRIPTOR, Anna Wojnar, Ant´onio M. G. Pinheiro
18. SIFT applied to CBIR, Jurandy Almeida, Ricardo da S. Torres and Siome Goldenstein,2009
19. Qi Zhang, Yurong Chen, Yimin Zhang, Yinlong Xu,SIFT Implementation and Optimization for Multi-Core Systems,IEEE,2008.
20. Distinctive image features from scale-invariant keypoints. David G. Lowe, International Journal of Computer Vision, 60, 2 (2004), pp. 91-110
21. CBIR in Cultural Databases for Identification of Images: A Local-Descriptors Approach, Eduardo Valle, Matthieu Cord, Sylvie Philipp-Foliguet
22. Application of Image SIFT Features to the Context of CBIR, Xu Wangming,2008.
23. Pisarevsky, V.: 2007, Intel Corporation, Software and Solution
24. homepages.inf.ed.ac.uk
25. www.opencv.org.