ACCUrACy AssessMeNT ANd Cl AssifiCATioN effiCieNC y of objeCT-bAsed iMAGe ANAlysis of AeriAl iMAGery

Following text is focused on finding a proper balance between classification accuracy and classification efficiency of object based image analysis of aerial imagery. For image classification were used tools of Feature Analyst, an ArcGIS extension, which are based on image filtering and searching for homogeneous segments corresponding to the threshold values defined by the training areas and corrected with tools of object recognition. The classification accuracy was assessed using the error matrices and calculating user’s and producer’s accuracy. To evaluate the time and work consumption, index of classification efficiency was introduced, which is a weighted value of accuracy and number of objects. To evaluate the tools of object recognition, the principle of fuzzy logic was used, which calculates accuracy with inclusion of alternative category. For the study was used a cut of aerial imagery of protected area of Křivoklátsko as a typical example of the character of available image data in the territory of Czech Republic – large area of fields, forests and meadows with a very heterogeneous character, water body with sun reflection, net of narrow roads and small villages. From the final values of accuracy is obvious, that Feature analyst provides comparable results with other authors. From the point of view of classification efficiency, application of tools of object recognition and its effect on the time and work consumption, it is a question of the extent of study area and the heterogeneity of the features of the physical world on the image data.


Introduction
In the territory of the Czech republic is aerial imagery currently the most accessible image data that can be used for mapping the land cover structure and its time-changes.The imagery covers the whole state territory in several time horizons (the oldest ones date back to the mid-20th century) in a detailed spatial resolution.Until recently, the evaluation process was based above all on a visual interpretation and manual vectorization.The development of image analysis methods has enabled particular automation and application of procedures.
The basic unit for an object based image analysis (oBIa) is a group of spatially related pixels with similar characteristics, which are joined by the process of segmentation until the classification criteria (shape, size, homogeneity) are met (dobrovolný 1998).Through these methods, the image is divided into segments (objects) that relate to features in the physical world and correspond with features in the input image (Železný 2005).
The basic principle of object-based image analysis is the search for homogeneous areas within the image data.In this study were used the tools of Feature analyst, an arcgIS extension, to classify the aerial imagery.The tools are based on image filtering, during which the filter window searches through the image for homogeneous segments corresponding to the threshold values defined by the training areas.Classification runs together with the segmentation, and during one set of iterations all image areas which correspond to the given conditions (one class of land cover) are found.
Since the spatial delimitation of naturally homogeneous areas in the landscape, and therefore also in the image data, often do not match the individual areas of land cover, the classified vector features also do not correspond with features in the physical world.
accuracy assessment is currently perceived as a fundamental component of the thematic classification of the image data, although a standard evaluation method has not yet been accepted (Foody 2002).The general potential of the accuracy evaluation was discussed, for instance, by Foody (2002 and2006), and an evaluation of the result accuracy by means of segmentation and object-based image analyses by Bruzzone (2008) and Chmiel (2010).
accuracy assessment is also often specified by the principle of fuzzy logic.Benz (2004) defined fuzzy logic as multi-valued logic quantifying uncertain statements.The basic idea consists in replacement of the two Boolean logical statements "true" and "false" with a continuous range of values in the interval <0,1>, where "0" means "false" and "1" means "true".all values between "0" and "1" represent a transition between the statements "false" and "true" (Benz et al. 2004).
This principle is used when interpreting analysis results, where a numerical value of affiliation degree to various categories of the analysis is assigned to the individual features.However, the principle is only used by analysis methods which first segment the image into homogeneous features, and in the second step classify all categories on the basis of training areas represented by segmented homogeneous features.The affiliation values are calculated by means of affiliation equations.
The classification accuracy depends mostly on the character of the imagery, on the definition of the training areas and on the number of the classification iterations.The specific value of accuracy (both producers and users) is calculated using the error matrices, where the area of correctly-and wrongly-classified objects is compared.In general -the better specified training areas and the higher number of iterations, the higher is the final accuracy but also the higher number of objects that correspond to one feature in the physical world and bigger time consumption.Because of that a proper balance between classification accuracy and classification efficiency must be found.

study area
For the needs of this project, emphasis was put during the selection of the study area on the data representativeness with respect to the character of commonly available data sources.The selection of classification categories was aimed to the needs of a hydromorphological evaluation of the watercourses, which needs to evaluate, in addition to the basic categories of land cover, the presence of single trees and green belts on the banks and along the rivers.
as the study area was used a mosaic of two orthorectified aerial photographs taken above a protected landscape area in the Křivoklát region (near the village of Kalinova ves) in the Berounka river basin.It represents a settled, rural landscape with a road network, a significant proportion of forest and green belts, and a watercourse with an alluvial plain.
The photographs were scanned in the spring 2003, with a spatial resolution of 1 m; the size of the processed area was 10 km 2 (2.5 × 4 km).There are only few overshadowed features and the image data is representative in term of the homogeneity of individual features.The aim was to determine the classification accuracy and efficiency of commonly available data, which is usually complicated by cast shadows, reflecting water bodies and heterogeneity within vegetation objects of the physical world and its change during the year seasons.The features are heterogeneous within one category of land cover as well as within the individual objects.Furthermore, the various categories are represented by a similar interval of the dn pixel values.Therefore, these categories coincided during the analysis.
on figure 1 is shows a cut-out of an aerial photograph, where the internal heterogeneity of the individual objects (fields are partly bare and partly covered with crops) and their mutual diversity (some fields have differently-grown crops, others are ploughed) is obvious.The similarity between various land cover categories is also obvious (e.g.roads x parts of ploughed field, or meadow x fields with crops, etc.).

Image Analysis
The Feature analyst extension works on the basis of defining the training sets delimited in the image.The image is classified and segmented in the same time while the filter window searches through the image for homogeneous segments corresponding to the threshold values defined by the training areas.during one set of iterations all image areas which correspond to the given conditions (one class of land cover) are found.It is possible to use a predefined settings for single land cover classes which takes into account the size, shape and homogeneity of searched features (e.g.long and narrow objects for the class "roads").a user settings is also possible.
The objects classified according the training sets (spectral class) are however not identical with a class of the land cover (information classes).For example the spectral class "permanent herbage" can be further divided, e.g.into the information classes "meadows", "gardens", "parks", etc. or, on the contrary, the classes "red roofs" and "brown roofs" can be merged into the "built-up area" category.
In the second step objects can be automatically removed on the basis of the dn pixel values or by means of the characteristics of the object's shape and size.according to (opitz 2008), the combination of methods that works with both spectral (radiometric) pixel values and spatial parameters is called "object recognition".
The individual categories were classified in several steps.In the first step, all image features which met the given conditions were selected on the basis of the definition of the training areas.In the following steps, the wrongly-detected features were removed, and the missing ones were added.
except the "water" category, the analysis began with large and very heterogeneous objects and continued to tiny homogeneous objects.First was detected the Berounka watercourse and two small water bodies.The training areas for this class were defined so widely, that there were as few as possible final vector objects for the watercourse (the individual features displaying free water body, overshadowed water surfaces, rapids sections under the weir and reflecting surfaces were merged into one whole).It was also essential that the definition of water body features remained in contrast to the bank zone.The classification resulted in a layer of 936 objects, in which field and forest features were also delimited, in addition to the successful classification of the water bodies.Since only six features corresponded to the water bodies, the water bodies were exported into a single layer and the layer was removed from the vector mask for the next analysis.So it does not represent a real result of the classification, and therefore the user's accuracy was expected to be 100%.
next, the forest features were analysed.In the addition to the forests were also delimited the features of green belts along the watercourses and the fields covered with crops.These wrongly-classified areas were removed by means of the shape and size characteristics of the object recognition (large homogenous forest features vs. small non-compact wrong features).The minimum area of the features was set as large as the built-up area could not be includeded into the layer (the dn pixel value of some roofs corresponds to the dn pixel value of forests).
as third was analysed the built-up area and the roads.The clutter was removed using characteristics of the size of the object recognition.By removing the roads layer from the vector mask were definitively separated the areas, which could otherwise be joined over a narrow road in one object (e.g.field -road -field, field -roadmeadow, etc).
Then, the "field" land cover class was analysed.The fields in the study area had a very heterogeneous character at the moment of aerial scanning (Figure 1) and six different training sets had to be defined to detect all objects in the study area.Collision with the "meadow" land cover class occurred very often.Clutter was removed using characteristics of compactness and size by means of object recognition.
The "meadow" class was then analysed.Since other "green" areas had already been removed from the vector mask, the user's accuracy should have been 100%.The minimum classification area was set as large as the gardens were not included in this class.
The "green belts" class was analysed in order to detect the vegetation zones in the alluvium along the watercourses.This class was distinguished from the "single trees" on the basis of the size criterion.
Finally, the "garden" class was analysed.In this layer were included all remaining unclassified image areas.a total of 2,338 features were classified in this category, but the clutter could not be simply removed from the layer.Therefore alow value of user's accuracy can be expected in this category.

Accuracy assessment
The accuracy was assessed by comparing the areas of correctly-and wrongly-classified features from both points of view -the producer's and the user's.The producer's accuracy gives the probability with which the physical world feature (captured in the image data) is correctly classified.The user's accuracy gives the probability with which the vector feature corresponds to the physical world feature.
The classification accuracy was determined on the basis of two different approaches -one using the error matrix, and another error matrix based on a fuzzy evaluation according to Sarmento (Sarmento 2008).The fuzzy evaluation was based on the creation of a reference file of features, where to each feature was assigned two possible categories of land cover -the primary reference class and the alternative reference class.
The feature reference file was selected from the features set created by means of an unsupervised segmentation.The unsupervised segmentation works with the image data without setting the training areas, i.e.only according to the definition of the number of final classes, number of the iterations, and size of the minimum area.The unsupervised segmentation thus results in homogeneous image objects, whose degree of homogeneity is not defined by the user but results from the natural process of segmentation.on figure 2

classification efficiency
Internally heterogeneous features of physical world are mostly divided into several large correctly-classified objects and a high number of wrongly-classified, small-area objects.The final number of vector objects within one heterogeneous physical world feature depends on the precision of definition of the training sets.The more precise definition, the higher is the accuracy, but also higher number of the vector objects and bigger time consumption.To evaluate the time and work consumption, a classification efficiency index was defined (formula 1).

Fig. 2 Examples of processing of the aerial imagery
The index normalizes the frequency of the features by their area and is based on the principle of weighted values, which assigns different importance to the individual elements of the file.The assignment of importance can be viewed from two different points.The application of the index according to theof object frequency suppresses the importance of a large number of small-area objects and emphasises the importance of a small number of large objects.according to the object area, the small suma of areas of small objects is balanced by their high frequency, and the importance of large objects is decreased by their low frequency.
The formula (1) for the index calculation is as follows: where Obj(a) is weight index of features in interval a, n(a) is number of features in interval a, p(a) are areas of features in interval a, N is number of features in analysed file, p are areas of features in analysed file.
The indices were calculated for single intervals (Table 1), inserted into error matrices (the frequency in the "a" interval was multiplied by the index of the "a" interval), and the producer's efficiency and user's efficiency were determined.For the calculation was used a data file created by removing the reference file features from the final file to provide the independence of the reference file The final file contains 6,135 features.It must be stressed that this is not the classification accuracy.The final value includes the internal heterogeneity of features in individual categories (represented by the feature frequency) in addition to the accuracy, and therefore it also includes the efficiency and time consumption.The "efficiency and time consumption" term means multiple definitions of the training areas and the whole classification procedure, until the physical world features are represented in the given category by the vector features as much as possible.The index is not exactly numerically expressed, but if the training areas and the analysis are correctly set, it can generally be said that the lower the user's accuracy, the higher the heterogeneity of the physical world features and the lower the classifi-cation efficiency, as well as the need for a higher number of training sets.

Accuracy assessment using the principle of fuzzy logic
In the case of processing with the Feature analyst, where classification runs together with segmentation, the fuzzy logic principle has not been used for classification but rather for determination of the classification accuracy.The reason for using fuzzy logic consisted in the fact that the areas (which actually represent various categories of land cover) are represented in the image data by pixels in the same dn value interval, and in terms of processing and the search for image homogeneous areas, they are assigned to the same category.a typical example is the grown green parts of fields versus herbage areas in meadows and gardens.These areas can be distinguished by feature recognition on the basis of shape and size characteristics, but only to a limited extent.
These objects were thus detected correctly according to the definition of the training set; however, they did not belong to the classified category and it was not possible to remove them from this category using the tools of object recognition.For that reason was defined the alternative category, as the second possible classification category into which the object could be assigned, according to the training set.The use of the alternative category should help to evaluate the potential of the Feature analyst for detecting the image homogeneous areas more precisely.
When determining the classification accuracy, the following values was assigned to the reference object: • 1 if the feature was classified correctly, • 0.5 if the alternative category corresponded to the character of the physical feature, • 0 if the feature was classified wrongly.
By means of the contingency tables, error matrices were formed, and the producer's and user's accuracy were determined for all classification categories.The accuracy was calculated according to the frequency and the area of individual features.If the classification results did not correspond to the reference category but complied with the alternative category, only the half-value (both of frequency and area) was taken into account.

Accuracy assessed by error matrices
Classification of aerial images of the study area resulted in a file of 6,459 vector objects, classified into 9 categories of land cover (hereinafter the "final file").
In the first part of classification, the image was searched for large features (forest, field, meadow, water bodies, green belts) and uniquely detectable features (roads, built-up area).The final layers for these catego-ries contain (in comparison with other categories) a small number of objects whose area is relatively large or close to the physical world features.For these categories, we can expect both the producer's and user's accuracy to be high.
In the second part of classification, the image was searched for small-area features (gardens, single trees), which are not typical in terms of their spectral reflectance, but are different due to their size and shape.Besides the searched features, so-called remaining areas (smaller areas in the middle of large heterogeneous features) were also assigned to these categories during the classification.The vector layers of these categories contain a large number of small-area features.The high number of "remaining areas" caused a low value of the producer's accuracy, while the user's accuracy was relatively high.This means that a high percentage of, e.g.gardens in the study area were identified correctly in the image, but a low percentage of vector objects classified as gardens, really are gardens.
The reference file intended for a comparison of results has traditionally been created by manual vectorization.In the case of this study area, the individual physical-world objects are internally heterogeneous, and therefore one physical object is represented by several vector objects.For the reasons of a better accuracy determination as well as time savings, the reference file was selected from the final file.
The selection of the reference file features was first limited by the object area (according to frequency distribution in both the final file and categories of land cover), and then were objects chosen randomly.The frequency distribution in the final file is displayed in the following semi-logarithmic diagram (Figure 3).
The frequency distribution was studied in five intervals with the limits of 30 m 2 , 50 m 2 , 100 m 2 , 1,000 m 2 , and over 1,000 m 2 , which covered the whole interval of obtained values.The interval limits are marked (vertical lines) in figure 3, and were selected so that the number of features in eachinterval was equal, and corresponds to the specificity of the final categories.a percentage representation of the number of category features for the individual size intervals is provided in the following table 2. a limit of minimum 5% was set for the selection of features from the final file for the reference file.a total of 324 features were selected for the reference file and the percentage representation varied within the individual categories (5-50%).For example, 50% of the features from the water body category were selected; however, this represented only three features.on the contrary, 108 features from the garden category were selected, representing, however, only 5% of all features classified as a garden.
The selection of the reference objects itself was run in the database table outside the arcgIS workspace to avoid a preferential selection of wrongly-or correctly-classified features and the reference file was independent.Then was each object assigned to a correct classification category (according to aerial imagery), and by means of contingency tables were created error matrices (using the area Tab. 2 A percentage representation (%) of the analysis categories, by size intervals, for the final file (Ff) and the reference file (Rf), with the number of features (No.)  Fig. 3 Object frequency distribution in the final file, semi-logarithmic diagram with delimitation of limits of the size categories, the logarithmic scale was used for the category axis x of objects) and calculated producer's (hereinafter Pa) and user's accuracy (hereinafter Ua).
In the categories classified in the first part (forest, field, meadow, water bodies, green belts, roads and built-up area) were reached values of more than 90%, while the categories that included a high number of remaining areas reached relatively lower values.all values are written in table 3.
Classification efficiency was calculated also using contingency tables and error matrices, but the area of the objects was weighted by the quantity of objects according to formula 1. values of efficiency from the producer's and user's point of view are written in table 3.

Accuracy assessment using the principle of fuzzy logic
For accuracy assessment was used the result of unsupervised segmentation.This divides the image into homogeneous features on the basis of definition of the minimum area size, number of classification categories, size and shape of the input representation, and number of iterations.The result of the unsupervised analysis is represented by objects that should be internally homogeneous and should represent (or should be a part of) only one physical world feature.a total of 60,156 objects were created, from which was selected a reference file consisting of 263 objects with identical frequency distribution in the size intervals (analogously to the selection of the reference file described above).To all objects in the reference file were then assigned the results of classification by Feature analyst, the reference category according to image data, and the alternative category.accuracy values are given in the following table 4.

Discussion
The number of reference features is always limited within the study, and is mostly a compromise between the most accurate evaluation possible and time efficiency (grenier 2008).The particular numbers are from within tens and hundreds of reference samples.For example, Huang (2008) used 257 reference features, and Xiaoxia (2005) determined the accuracy on the basis of 65 checkpoints.The potential for the exact determination and calculation of the minimum number of reference points was discussed in detail by grenier (2008).
The accuracy of object-based analyses of aerial imagery by the Feature analyst extension mainly depends on the character of the analysed data.The homogeneity of individual areas is affected by vegetation season, overshadowed of the objects, and the sun reflection in the water bodies.The second important factor is the selection of the classification categories and correct definition of the training areas.
The above mentioned conditions influenced especially the number of final objects within one feature of physical world and within a category, and time and work consumption.Therefore the number of objects was included in the calculation of classification efficiency as a weighted index.
as for the accuracy assessment, in the main analysis categories were achieved values of 90% for both producer's and user's accuracy.as to the "single trees" and "gardens" categories, the producer's accuracies were 78% and 76%, and the user's accuracies 44% and 52%, respectively.Both categories were classified at the end of the analysis.Since they represent small-area features of various shapes and radiometric characteristics, it was not possible to easily remove the wrongly-classified objects from these categories using tools of object recognition.The values and number of reference objects are comparable with other authors (Table 5).
The highest differences of accuracy and efficiency values were noticed in the land cover categories with very heterogeneous character in the image data -forests, fields and meadows (Figure 2).These categories are in the image represented with a few large objects.From the point of view of time and work consumption it would be easier in a small-extent study area, to vectorise these land cover categories manually and run the classification for the categories with small heterogeneous objects in a masked-out extent.

Magdalena Králová Charles University in Prague, Faculty of Science Department of Physical Geography and Geoecology Albertov 6 128 43 Prague 2
Czech Republic E-mail: madla.bicanova@gmail.com

Fig. 1
Fig. 1 A cut-out of the aerial photograph of the study area are shown two examples of classification results and unsupervised segmentation of aerial imagery.