вход по аккаунту



код для вставкиСкачать
Measurement Science and Technology
Related content
Fused man-machine classification schemes to
enhance diagnosis of breast microcalcifications
To cite this article: Ioannis Andreadis et al 2017 Meas. Sci. Technol. 28 114003
View the article online for updates and enhancements.
- A comparative study of image features for
classification of breast microcalcifications
I I Andreadis, G M Spyrou and K S Nikita
- Comparison of computer-aided detection
systems on FFDM and SFM
Jun Ge, Lubomir M Hadjiiski, Berkman
Sahiner et al.
- Multiresolution local binary pattern texture
analysis combined with variable selection
for application to false-positive reduction in
computer-aided detection of breast
masses on mammograms
Jae Young Choi and Yong Man Ro
This content was downloaded from IP address on 28/10/2017 at 19:33
Measurement Science and Technology
Meas. Sci. Technol. 28 (2017) 114003 (12pp)
Fused man-machine classification
schemes to enhance diagnosis of breast
Ioannis Andreadis1,3,Chatzistergos Sevastianos1, Spyrou George2
and Nikita Konstantina1
Department of Electrical and Computer Engineering, National Technical University of Athens, Athens,
The Cyprus Institute of Neurology and Genetics, Nicosia, Cyprus
Received 16 March 2017, revised 8 August 2017
Accepted for publication 24 August 2017
Published 17 October 2017
Computer aided diagnosis (CADx) approaches are developed towards the effective
discrimination between benign and malignant clusters of microcalcifications. Different sources
of information are exploited, such as features extracted from the image analysis of the region
of interest, features related to the location of the cluster inside the breast, age of the patient
and descriptors provided by the radiologists while performing their diagnostic task. A series
of different CADx schemes are implemented, each of which uses a different category of
features and adopts a variety of machine learning algorithms and alternative image processing
techniques. A novel framework is introduced where these independent diagnostic components
are properly combined according to features critical to a radiologist in an attempt to identify
the most appropriate CADx schemes for the case under consideration. An open access database
(Digital Database of Screening Mammography (DDSM)) has been elaborated to construct a
large dataset with cases of varying subtlety, in order to ensure the development of schemes
with high generalization ability, as well as extensive evaluation of their performance. The
obtained results indicate that the proposed framework succeeds in improving the diagnostic
procedure, as the achieved overall classification performance outperforms all the independent
single diagnostic components, as well as the radiologists that assessed the same cases, in
terms of accuracy, sensitivity, specificity and area under the curve following receiver operating
characteristic analysis.
Keywords: computer aided diagnosis, microcalcifications, interaction with radiologist,
BIRADS descriptors
(Some figures may appear in colour only in the online journal)
1. Introduction
either to oversights or incorrect recommendations of follow-up
procedures [2]. One of the most important mammographic findings associated with the existence of breast cancer are clustered
microcalcifications (MCs), that constitute tiny white spots—
deposits of calcium salts—that may be found anywhere inside
the breast tissue [3]. Both the inherent limitations of the mammography and the subtle nature of MCs make reading mammograms and analyzing breast MCs a very demanding job for
radiologists, since judgments depend on training, experience
and subjective criteria [4].
According to the American Cancer Society, breast cancer is the
second leading cause of cancer death in women and it is expected
that in 2017, 252 710 new cases of invasive breast cancer will
have been diagnosed in the United States [1]. Mammography
remains the main imaging modality for the diagnosis of breast
cancer, however radiological errors still occur, mainly due
Author to whom any correspondence should be addressed.
© 2017 IOP Publishing Ltd Printed in the UK
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
processes are not trivial problems on their own and may add
high computational complexity in the system. On the contrary,
in the case where such features are directly available from the
radiologist we can (i) reduce the complexity of the system and
obtain in a relatively convenient way critical information, as
such tasks are part of a radiologist’s daily diagnostic practice,
and (ii) promote the interaction between the radiologist and
the system by increasing the level of trust.
Since the CADx system is oriented to perform as a second
reader, proper interaction and collaboration with the radiologist, through the mutual exchange of critical information, is
needed in order to fulfill its role. Motivated by these assumptions, we proceed, initially, on the implementation of several
independent diagnostic components and, then, on the introduction of a novel framework where features provided by
the radiologists determine the proper selection of the developed CADx schemes, in an attempt to optimize the diagnostic
The rest of this work is organized as follows: section 2
provides brief information regarding the dataset used, collected from a public database of mammograms. The methodologies applied for developing CADx schemes are presented
in section 3, followed by detailed information regarding the
construction of the proposed framework in section 4. In sections 5 and 6, the classification performance of the proposed
approach is evaluated and discussed, respectively. The most
important conclusions and the future steps that have to be followed for the refinement of the diagnostic task are finally discussed in section 7.
Towards assisting the radiologists in their diagnostic task,
computer aided diagnosis (CADx) systems have been introduced, whose role is to support the expert in the discrimination of benign and malignant findings [5]. It is a well reported
fact that radiologists, in an attempt to minimize the number of
missed cancers, refer a great number of cases for biopsy test,
but a small percentage is finally proved to present malignant
pathology [4–7]. However, this significant number of unnecessary biopsies causes both physical discomfort and associated
redundant costs to the patient. As a result, an effective CADx
system should focus on minimizing the number of unnecessary biopsies, by retaining the number of missed malignant
cases at low levels.
While there are many studies reported in the literature
concerning CADx approaches for the classification of breast
lesions [5, 8], there are still many challenges that have to be
faced. Firstly, the general practice followed in CADx systems
concerning the pursuit of constructing a representative dataset
and training with it the adopted classifier, in order to ensure
high generalization ability, introduces a specific limitation.
Specifically, the attempt to achieve a proper training of the
system usually excludes the hypothesis that the variations of
the inherent properties of the mammograms included in the
dataset may need alternative approaches to refine the diag­
nostic procedure. The second limitation is related to the low
extent of the radiologist’s interference in the CADx process,
which is mainly based on automated image processing methodologies. However, the role of a CADx system is not to act
independently of the radiologists, but instead to assist them
in the diagnostic task by providing a reliable second opinion.
To the best of authors’ knowledge, there are few studies that
focus on the interaction between the radiologist and the CADx
process, which should ideally constitute a diagnostic team [9].
Moreover, in earlier works, we have shown that features critical to radiologists, such as the density of the breast, descriptors for the cluster of MCs and the location of the cluster, could
be exploited for the optimization of the diagnostic procedure
[10–12]. Specifically, we have demonstrated that the breast’s
density and descriptors according to the American College of
Radiology Breast Imaging Reporting and Data System (ACRBIRADS) standard [13] may provide high discrimination
ability between benign and malignant clusters [10] and play
a crucial role in the proper selection of image enhancement
techniques [11]. Furthermore, using the location of the cluster
of MCs as provided by the radiologist, we have proposed a
framework leading to the generation of probabilistic maps
which provide quantified topological information and a priori
risk estimation of the malignancy of the cluster that appears to
have a positive impact on classification performance [12].
The objective of the current study is to optimize the diag­
nostic process by reinforcing the interaction between the CADx
system and the radiologist. Based on the abovementioned
findings of our previous studies [10–12], we conclude that the
exploitation of specific factors, such as a breast’s density and
the location of the cluster, the final phase of a CADx system
may lead to the enhancement of its classification performance.
Although it is possible to automatically estimate breast composition [14] and detect suspicious clusters of MCs [15], such
2. Database of mammograms
The largest publicly available database of mammograms,
the Digital Database of Screening Mammography (DDSM),
has been used in the current study [16]. The DDSM database
constitutes the most popular choice for the development of
CAD methodologies. It serves therefore as a representative
testbed that enables both proper evaluation of the proposed
approaches and fair comparisons with the results reported
in previous studies in the specific scientific field. Apart from
the large number of cases included, one of the major advantages of this database is related to the wealth of information
provided for each case included. Except for the available
digitized mammograms, the accompanying files include information on the assessment of the region of interest (ROI) performed by radiologists who participated in the preparation of
the DDSM database. Specifically, we were able to extract for
each case data regarding the exact location of the ROI inside
the breast, its pathology status after the biopsy test (benign/
malignant) and critical radiologist descriptors according to
the ACR-BIRADS standard that constitutes the basis of the
diagnostic task [13]. This lexicon is a quality assurance tool
originally designed for use with mammography, in order to
standardize reporting and encourage uniform use of terms
among radiologists. The lexicon has become central to the
practice of mammography, since the utilization of common
language enables clearer communication and more objective
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
or malignancy, comprises a series of well-known and efficient machine learning algorithms that ensure proper exploitation of the features and reliability in discrimination.
Following the practice used in our previous study [12] and
in an attempt to further improve classification performance,
all classifiers are combined using a voting scheme in order to
(i) have as output a continuous risk and not binary decisions
and (ii) investigate the possibility of developing a powerful
ensemble that may ensure satisfactory diversity in the classification rules and exploit the individual strengths of the
base classifiers. Preliminary results have revealed the positive contribution of the ensemble of classifiers in relation to
base classifiers [12]. To this end, we implement and properly combine ten different binary classifiers: four support
vector machine (SVM) classifiers using different kernels,
one k-nearest neighbors (k-NN), one classification tree (CT),
one Random Forests classifier, one multi-perceptron neural
network (MLP-NN) and two probabilistic neural networks
(PNN) using different functions. The final estimate of the
ensemble classifier for malignancy is computed by dividing
the votes for malignancy to the total number (10) of classifiers. As a result, the final output of the CADx scheme is a
risk percentage ranging from 0–100.
In the case of ROI analysis, we exploit the outcomes and
the approaches proposed in our previous studies to analyze
the cluster of MCs in the ROI. Initially, image enhancement
preprocessing of the ROI is applied to improve the contrast
features [11]. This step is not a prerequisite, since, although it
helps with the visual inspection of MCs, it adds computational
cost. Segmentation of the MCs is followed to isolate each
finding from the surrounding breast tissue [18]. Analyzing the
shape of each individual MC, the morphology of the whole
cluster, and the texture of the ROI, we end in a set of 188
image features [10]. To reduce the number of features and
locate a satisfactory significant subset, a feature selection step
is performed, using the sequential forward selection (SFS)
method. The subset of features extracted is used for the final
classification phase.
analysis of mammographic features [17]. The standard uses
descriptors that include the radiologist’s determination of the
type of the MCs and the distribution of the cluster (e.g. cluster
with amorphous MCs and linear distribution), the rate of the
density of the breast (e.g. BIRADS category d, the breast is
extremely dense) and his final assessment for the case (e.g.
BIRADS category 5, highly suggestive of malignancy, tissue
diagnosis is suggested). Additionally, a subjective metric,
namely subtlety, is provided that ranges between 1 and 5 and
represents the grade of difficulty, according to the radiologist,
for the analysis of the finding. Finally, the age of the patient is
also available in the DDSM files, and is the sole information
extracted from the patient’s medical file.
We exploited almost all cases including an annotated cluster
of MCs, ending in a dataset consisting of 1715 different ROIs.
Following the separation of the dataset performed in our previous study [12], we formed three different subsets. 65% of
cases (1114 cases including 557 benign and 557 malignant)
have been included in the balanced training set and the rest,
35% (601 cases), have been used for evaluation performance.
Specifically, the 20% among them (120 cases: 60 benign/60
malignant) has been used to create a balanced validation set
and the rest, 481 cases (265 benign/216 malignant), has been
exploited to form an independent test set. The selection was
carried out randomly, but special care was taken to ensure that
all three subsets contained cases of both varying subtlety and
breast density.
Each subset has a specific role in the approach proposed
in the current study. The training set will serve as a pool of
cases of different properties that will be exploited to train and
develop alternative CADx schemes, each of which will be oriented for analyzing specific cases, based on breast density and
the BIRADS assessment. The validation set will serve as the
basis to determine rules and associations that will lead to a
novel framework that simulates the interaction between the
radiologists and the developed CADx schemes. Finally, the
large test set enables the proper evaluation of the proposed
approaches, using cases that have not been involved during
the training nor the validation phase.
3.2. List of CADx schemes
3. CADx schemes
3.2.1. CADx schemes based on image analysis. The CADx
schemes reported in this section are developed following the
general structure presented in figure 1. The difference between
them is mainly between the cases selected from the dataset
pool (training set) and the application or not of the preprocessing phase. In each case, proper parameterization of the
involved classifiers is conducted, before integrating them into
the ensemble classifier. We should note that this parameterization is necessary to pursue the optimization of the classification performance, despite the high complexity introduced
due to the large parameter space of the involved classifiers.
However, as soon as the optimal values of the parameters are
identified and selected, the evaluation of a new case (ROI) is
a simple process that adds no further time delay. Therefore,
the application of the proposed CADx schemes in the testing
of new cases and their adoption in daily clinical practice is
In this section, we present the CADx schemes developed
in this study. We present, initially, the general pipeline followed and, then, a full list accompanied by the details and the
methods used in each different scheme. All CADx schemes are
developed using the available 1114 cases of the training set.
3.1. Structure of CADx scheme
The general structure of each CADx scheme is presented in
figure 1. For each case in the DDSM database, both the ROI
and accompanying information (patient’s age and features
critical to radiologist) are available. Either extracting features from image analysis algorithms or using directly the
data (descriptors) available in the files of the database, the
final classification step, towards the prediction of benignity
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
Figure 1. Flowchart for a CADx scheme. In the case of analyzing ROIs, the steps depicted in blue are followed.
•CAD2, trained with ‘obvious’ cases rated as BIRADS 2
or BIRADS 5.
•CAD3, trained with ‘probably benign’ cases rated as
•CAD4, trained with ‘probably malignant’ cases assessed
as BIRADS 4.
The first proposed CADx scheme constitutes a ‘general’
scheme, that means a scheme trained with all the available
1114 ROIs of the training set, including cases of varying
subtlety and density. This scheme, CAD1, has been obtained
without preprocessing of the ROI.
However, as already discussed, we avoid focusing on this
single ‘general’ CAD1 scheme. Instead, our main objective
in this work is to develop more ‘specified’ schemes, oriented
towards the analysis of cases with specific properties in an
attempt to further refine the diagnostic procedure.
The first available feature is the BIRADS assessment that
the radiologists performed [13]. According to the provided
guidelines, a mammogram is classified using a rate from 0–5.
In general, obvious benign and malignant cases are rated
as BIRADS 2 and 5 respectively. On the contrary, probably
benign cases needing, however, short follow-up are rated
as BIRADS 3, while cases with high probability of malignancy needing biopsy are rated as BIRADS 4. Cases rated
as BIRADS 0 need extra medical examinations and, finally,
cases rated as BIRADS 1 are out of the scope of this study,
since they concern negative cases with no findings observed.
In a previous study [19], we indicated that CADx methodologies perform satisfactorily in subsets of obscure cases
(such as BIRADS 3), where the radiologists recommend a
short follow-up. This observation implies that such ‘oriented’
schemes, following radiologist’s BIRADS assessment, may
be adopted during the diagnostic process to assist expert’s
diagnosis. Therefore, we properly separated the training set
of 1114 ROIs in subsets, following the BIRADS rating provided by the radiologists, in order to develop more ‘specified’
schemes, namely:
The CADx schemes discussed so far (CAD1–CAD4) have
not included the initial step of ROI’s enhancement (preprocessing). Thus, the second radiologist’s feature considered is
related to the density of the breast that influences directly
the need for preprocessing of the ROI [14]. According to the
BIRADS standard [13], the breast composition is assigned
into four categories, category a (breast entirely fat), category
b (scattered areas of fibroglandular density), category c (heterogeneously dense breast) and category d (extremely dense
breast). Although there may exist inter- and intra-variations
among radiologists regarding mammographic breast density
assessment, it has been shown that this variability decreases
when grouping the density rating into two broader categories,
considering categories a and b as fatty tissues and categories
as c and d as dense tissues [20, 21]. This practice has been followed in the current study, so as to reduce the effect of these
variations on the proposed framework.
In a previous work we studied the contribution of the image
enhancement phase to the correct categorization of clusters
of MCs, and the most important observation extracted concerned the fact that there was not a superior algorithm outperforming the rest [11]. Instead, we showed that the density of
the breast is the main factor that determines the appropriate
selection, since the analysis of fatty tissues (categories a and
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
b) was favored by the local range modification (LRM) algorithm, while a wavelet-based (WB) algorithm proved to have
the potential to improve the classification when cases of dense
tissue (categories c and d) are under consideration. Thus, the
density of the breast, as indicated by the radiologist, is a crucial
factor that determines the algorithm that has to be followed for
the cluster’s analysis. Exploiting the specific conclusion, we
proceed with proper division of the training set based on the
breast density, in order to apply different enhancement algorithms and obtain more ‘specified’ classifiers. In particular,
the following schemes are obtained:
his diagnosis. We should note here that the input requested
by the expert for the CADR scheme is part of the common
diagnostic workflow following the ACR-BIRADS standard
and, therefore, the radiologist’s task is not burdened with
additional information and the application of extra standards.
4. Proposed framework
The CADx schemes obtained in the previous step using the
training subset have to be validated using a new set of cases,
not involved during the training phase. Furthermore, the main
aim of this study is to exploit the available critical features
provided by radiologists to choose the best fitted computational pipelines, based on the particular properties of each
case. As a result, we propose a novel framework (a set of
rules) where alternative CADx schemes are combined in an
appropriate way to further enhance the diagnostic process. To
this end, the validation set is used, that enables both the validation of the CADx schemes and the dynamic creation of the
requested framework, through a process of proper modifications and tests pursuing the optimization of the classification
In case we used the general practice followed in the CADx
domain, we would only examine the performance of the CAD1
scheme presented previously. This specific scheme is based
exclusively on the image analysis of the ROIs and is trained
using all the available cases of the training set. Applying the
specific scheme in each of the 120 ROIs included in the validation set, we extract for each case a percentage indicating
its risk of malignancy. This risk, RISKIM, reflects the output
that is achieved using a scheme, trained with a great variety
of cases, which is exclusively based on the image analysis of
the ROI.
In order to investigate the contribution of radiologists’
descriptors, we may correspondingly evaluate the performance of the CADR scheme. Applying the specific scheme
on a case of the validation set, we extract an independent
risk, RISKRAD, that represents the risk estimation extracted
using data from the patient’s folder (age) and from the
radiologist’s descriptors (according to BIRADS standard),
without using automated methodologies for the image
analysis of the ROI.
However, as already mentioned, our aim is not to independently evaluate the two components, but investigate instead
whether they may contribute to each other towards improving
the overall performance. Therefore, we proceed on the configuration of a framework that combines the CADx schemes
along with the available features provided by the radiologists,
in an attempt to optimize the diagnostic process. The most
important features that are difficult to extract automatically
and have been proved to influence the proper selection of
CADx schemes and include (i) the BIRADS assessment of the
cases, (ii) the density of the breast and (iii) the exact location
of the cluster of MCs in the breast. Each of these factors is
available in the DDSM files. We proceed on the analysis of
these factors and their contribution in the selection of the most
appropriate CADx schemes.
•CAD5, trained with cases of fatty tissue (density category
a or density category b), without preprocessing.
•CAD6, trained with cases of dense tissue (density category c or density category d), without preprocessing.
•CAD7, trained with cases of fatty tissue, using the LRM
enhancement algorithm.
•CAD8, trained with cases of dense tissue using the WB
enhancement algorithm.
3.2.2. CADx scheme based on radiologists’ descriptors. The
CADx schemes considered so far have been based on the automated processing of the ROIs, through the extended analysis
of image properties. However, apart from the automated methodologies for the image analysis, an independent diagnostic
component is needed that exploits the patient’s data and the
radiologist’s descriptors used during the diagnostic process.
The contribution of such descriptors has been investigated in
recent years [10, 22–25]. To the best of our knowledge, no
study has combined image-processing algorithms with radiologist’s descriptors in order to design a framework where
features provided by the radiologists determine the proper
selection of different CADx pipelines.
Herein, we exploit the information provided in the accompanying files in the DDSM database. For each case, eight features are available, namely (i) the age of the patient, (ii) the
breast density, (iii) the type of MCs, (iv) the distribution of
MCs, (v) the type of mass, (vi) the type of the boundaries of
the mass, (vii) the subtlety of the ROI and (viii) the BIRADS
assessment. Apart from feature (i), all other features are provided by the radiologist who assessed the case. Features (ii)–
(vi) and feature (viii) are based on the ACR-BIRADS standard.
Features (v) and (vi) are applicable only in the case that the
ROI contains both MCs and a mass. Feature (vii) (subtlety of
the ROI) consists of a subjective rating provided by the radiologist concerning the grade of difficulty for cluster’s analysis.
The age of the patient is a quite important indicator that is
co-estimated in various models for the risk analysis of a patient
to suffer from breast cancer [26]. The remaining five features
(ii)–(vi) are not related to the automated image analysis, but
constitute the diagnosis performed by the radiologist in terms
of the ACR-BIRADS standard. These six features are used as
input to the diagnostic component we plan to develop. As a
result, using the classifiers presented in figure 1, we develop a
CADx model, namely CADR, that has the potential to predict
the pathology status of a cluster of MCs, using exclusively the
age of the patient and descriptors that the radiologist uses for
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
Figure 2. Proposed framework. Selection of appropriate guided schemes by radiologist’s descriptors.
framework in figure 2, the effect of breast density. In the case
of fatty tissues (category density a, entirely fatty, or category
density b, scattered areas of fibroglandular density), ROIs are
evaluated by the CAD4, CAD5 and CAD7 schemes and the
final risk (RISK1) assigned is equal to the mean value of risks
extracted by the three alternative CADx schemes. In other
words, we exploit all the available CADx schemes that have
already been trained with cases of similar properties, either
regarding the BIRADS assessment (CAD4) or the density of
the tissue (CAD5 and CAD7, for fatty tissues with/without
image enhancement, respectively). The same practice is followed in the case of dense tissues (density category c, heterogeneously dense, or category d, extremely dense), evaluating
each ROI by the corre­sponding schemes CAD4, CAD6 and
CAD8, where the final risk (RISK1) obtained is the mean value
of risks extracted by these three CADx schemes.
4.1. Effect of BIRADS assessment
The first factor studied is the BIRADS assessment provided
by the radiologist, without previously using any CADx
approaches. The specific information is exploited to use the
corresponding CADx scheme that is trained with cases of
the same BIRADS category. In other words, instead of using
the more ‘general’ CAD1 scheme, we choose among the more
‘specified’ schemes (CAD2, CAD3, CAD4). Based on this step,
if a single CAD scheme is applied (e.g. CAD2 or CAD3) the
value of the output RISK1 coincides with the risk percentage
of this single CAD. When a case is evaluated as BIRADS 4,
more ‘specified’ schemes are applied, as discussed in the next
subparagraph. The section of the framework incorporating
the BIRADS assessment (section A) is presented in figure 2.
Using as first criterion for the selection of CADx schemes the
radiologist’s BIRADS assessment, we pursue retaining the
primary role of the radiologist and help him verify his assessment by using reliable CADx schemes or manage difficult
cases belonging to ‘grey zones’, as those initially assessed as
BIRADS 3 [27].
4.3. Effect of cluster’s location
The last factor provided by the radiologist concerns the relative
location of the cluster of MCs inside the breast. Radiologists
take under consideration during the diagnostic task specific
information by splitting the mammogram into different quadrants [13]. The value of the specific factor has been studied
thoroughly in our previous study, where we introduced the
main principles for the use of location features, we proposed
the generation of breast probabilistic maps and investigated
their potential to provide quantified information to support
diagnosis [12].
Following the outcomes of this work, we propose the use
of an independent component that exploits the topological
4.2. Effect of breast density
Except for the BIRADS assessment, the density of the breast
may be also exploited towards selecting more ‘specific’
classifiers. We verified during the tests performed that the
majority of misclassifications were observed on cases that
were classified as BIRADS 4. Therefore, we further investigate whether more ‘specified’ schemes, based on breast density, may contribute to the CAD4 scheme and improve the
classification performance. We present in section B of the
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
information offered by breast probabilistic maps. Each ROI
of the validation set is projected on the corresponding location into the breast probabilistic map and a priori probability
for malignancy is extracted, namely Pm,loc . Obviously, the
corre­sponding probability for benignity, namely Pb,loc, is
equal to Pb,loc = 1 − Pm,loc. The risk provided by this comp­
onent, which is exclusively based on the location of the cluster
inside the breast, is defined as RISKloc = a ∗ (Pm,loc − Pb,loc ).
Its role is to serve as a shift to the risk obtained (RISK1) by
the previously applied CADx schemes that focus on the image
analysis of the ROI. The parameter a represents the grade that
the specific risk RISKloc may influence the final risk. Given
the fact that the final risk computed by the proposed framework is equal to RISK2 = RISK1 + RISKloc, it is obvious that
the risk RISKloc may contribute positively by increasing the
risk, in case the location of the cluster recommends malignancy ( Pm,loc > Pb,loc), or by reducing it otherwise. We add
consequently the use of an independent component that contributes in the risk estimation by properly modifying the risk
estimated by CADx schemes (sections A and B) that are based
on the image analysis of the ROI.
Figure 3. Esimated Az values applying LOO on the training set.
considered classifiers, while the leave one out (LOO) method
and receiver operating characteristic (ROC) analysis is followed for performance evaluation.
We present therefore in figure 3 the results achieved for
each different CADx scheme, in terms of the Az value (area
under curve), which is the best-suited metric for two-class
classification problems.
Observing the specific results, we may verify again some
conclusions extracted in previous works. Specifically, we may
see that the enhancement using the LRM algorithm for cases
of fatty tissue (CAD7) provides almost equal classification
results to those achieved when no enhancement algorithm is
applied (CAD5). In the case of dense tissues, the use of the WB
enhancement algorithm (CAD8) slightly improves the performance achieved when no preprocessing phase is considered
(CAD6). The poorest performance is achieved when using a
scheme trained exclusively with cases assessed by the radiologists as BIRADS 4. On the contrary, the other ‘­specified’
schemes (CAD2, CAD3) appear to have a high discrimination
ability. Therefore, comparing to the performance achieved
by the ‘general’ scheme CAD1, we see that there might exist
subsets of cases where we may achieve higher classification
performance through more ‘specified’ CADx schemes.
These results present, as already mentioned, an initial
assessment of the performance of the proposed schemes.
In order to proceed with the evaluation on new subsets, we
train properly all the CADx schemes. The schemes obtained
form the basis for the design and evaluation of the proposed
4.4. Combination with CADR diagnostic component
Our final contribution includes the combination of the output
of the framework with the CADR scheme, computing the final
risk estimation RISKFINAL as the mean value of these two
percentages (RISK2 and RISKRAD respectively). We achieve
consequently not only to decide the proper selection of computational schemes based on features provided by radiologists,
but also to highlight the contribution of their diagnosis in the
estimation of the final risk. We satisfy that way the primary
principle followed in computer-aided diagnosis systems that
requires computational approaches to have a supplementary
role into the diagnostic process.
5. Results
According to the separation of the initial dataset into the
training, validation and test set we present the results obtained
for each different subset. The training set has been used for the
training of the classification schemes, the validation set has
been used for the design and the evaluation of the proposed
framework and finally the test set serves the invest­igation of
the efficacy of the proposed methodologies, using cases which
are totally unknown during the training and the validation
5.2. Validation phase
The use of the validation set consists of two main axes: (i)
evaluation of the performance of the CADx schemes (CAD1
and CADR) developed during the training phase, and (ii)
design of a novel framework for the interaction between
CADs and radiologists that includes a set of rules that determine the proper selection of CADx schemes following critical
radiologist’s features.
Initially, we evaluate both the CAD1 and CADR schemes
on the 120 cases of the validation set and we extract the
ROC curve in order to estimate the Az value. The results are
presented in figure 4. The value achieved using the CAD1
scheme is equal to 0.789, while the corresponding value
5.1. Training phase
The training set consists of 1114 cases of varying subtlety.
For reasons of completeness, except for training the CADx
schemes, we investigate their potential using generalization
techniques, so as to obtain initial insights on their performance, before evaluating them on the independent validation and test sets. Therefore, we initially apply ten-fold cross
validation to adjust the regularization parameters for the
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
Figure 4. Intermediate and final Az values on the validation set.
achieved through the CADR scheme is equal to 0.791. The
former value represents the performance on unknown cases
of a ‘general’ CADx scheme, trained with a great variety of
images, focusing on the automated image analysis without the
interference of the expert physician. On the contrary, the latter
value represents the discrimination ability on the same cases
of a CADx scheme that is trained using the age of the patient
and some radiologist’s descriptors used during the diagnostic
task, according to the BIRADS standard. The fact that the two
values are almost identical motivated us to investigate whether
the use of alternative CADx schemes following features provided by expert’s may improve the classification performance,
ending consequently in the framework presented in figure 2.
We evaluated separately the performance of the different
sections of the proposed framework in order to highlight the
contribution of each of the three factors, i.e. the BIRADS
assessment (section A in figure 2), the density of the breast
(section B in figure 2) and the exact location of the cluster
(section C in figure 2). These intermediate results are presented in figure 4, along with the performance achieved by the
proposed framework at its final level.
We can observe that the achieved value using exclusively the
section A of the framework is equal to 0.858, that is increased
if compared to the corresponding value achieved by either the
CAD1 scheme (0.789) or CADR scheme (0.791). This fact
implies that the a priori knowledge of the BIRADS assessment
performed by the radiologist may enable the proper selection
of CADx schemes that are designed for cases of the specific
category. Similarly, the further adoption of alternative CADx
schemes based on the density of the breast (section B of the
framework) leads to a slight improvement of the Az value, since
it is increased from 0.858–0.883. It seems indeed that the breast
density, as provided by the radiologist, may permit a faceted
analysis of new cases by determining the optimal selection
of computational pipelines. Finally, focusing on the contrib­
ution of the location of the finding, the proposed approach in
section C of the framework succeeds to further improve the
performance, as the Az value is increased from 0.883–0.916.
This increase implies that the location of the cluster provides
quanti­fied information and enhances the diagnostic process. We
should note here that the parameter a used for the estimation of
RISKloc was selected to be equal to 0.5, as we observed that this
Figure 5. ROC curves of the proposed framework (blue line) and
the radiologists (black line) on the validation set.
value provided the best results on the validation set, after proper
param­eter selection into the space (0,2].
The last step of the proposed framework counts on the combination of the output of the framework so far (sections A, B
and C in figure 2) with the output of the diagnostic comp­onent
CADR. Through this last modification, the Az value is further
increased to 0.945. Overall, the proposed framework has a
positive impact in the classification results, since the performance on the validation set using either the image-analysis
(CAD1) scheme or the radiologist-based (CADR) scheme is
improved achieving an increase of Az equal to 0.156.
We further exploit the BIRADS assessment provided in the
DDSM files to extract the ROC curve by the radiologists, following the steps discussed in previous works [12, 28]. Figure 5
includes the ROC curve achieved by the radiologist, along
with the ROC curve provided by the proposed framework,
accompanied by the corresponding Az values. Comparing
straightforwardly the performance between the radiologists
and the proposed framework, we may observe that there is a
quite large difference of 0.165 between the two values.
In order to estimate additional metrics (accuracy, sensitivity, specificity) we proceed on the further analysis of the
ROC curve and the identification of the optimal point, which
is defined as the point presenting the lowest distance to the
upper left axes. Based on the specific approach, the optimal
threshold selected is equal to 31%. However, since medical
classification processes usually prefer higher sensitivity
to higher specificity rates, an alternative threshold may be
selected that permits pursuing stable behavior of the system
oriented towards effective identification of malignant cases.
Therefore, we seek a second threshold value that assures perfect sensitivity and the maximum possible specificity. This
demand is satisfied using a threshold of 15%. If the final risk
is equal to or greater than this value, the case is classified as
malignant, and if not the case is classified as benign.
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
Table 1. Proposed framework’s and radiologists’ performance on the validation set.
Proposed framework (threshold 31%)
Proposed framework (threshold 15%)
Using these two thresholds, we estimate the accuracy, sensitivity and specificity for the proposed framework. We proceed with corresponding measurements for the radiologists,
considering as malignant cases those assessed as BIRADS 0,
4 and 5 and benign cases those assessed as BIRADS 2 and 3.
The results obtained are presented in table 1.
The results indicate that indeed the maximum classification
accuracy is achieved through the proposed framework, using
the high threshold. In case we pursue high levels of sensitivity,
we may observe that the proposed framework outperforms the
radiologists, since both achieve perfect sensitivity, but the
former provides improved accuracy and specificity.
Figure 6. Intermediate and final Az values on the test set.
5.3. Testing phase
classification performance than that achieved by the radiologists who assessed the same cases. The difference between
the Az values is almost equal to 0.21. Considering the high
threshold (that was selected pursuing high levels of accuracy),
we achieve the highest classification accuracy on the test set
(0.746). Instead, through the use of the low threshold (that
was previously selected pursuing high levels of sensitivity),
we did indeed achieve almost perfect sensitivity (0.972) and
satisfying specificity (0.377). The importance of these results
is further highlighted by the straightforward comparison, since
all values are superior to the corresponding achieved by the
radiologists. We manage consequently to retain high levels of
sensitivity, which is extremely important in medical processes,
and improve at the same time both the accuracy and specificity.
The validation set in the previous phase was mainly used for
the design of the framework and the rules that determine the
appropriate selection of the CADx schemes. In order to verify
the efficacy of the proposed approach, we proceed on proper
evaluation on a test set that has not been involved at all during
either the training of the CADx schemes or the design of the
framework. We repeat the process followed during the validation phase, by applying the CAD1, CADR and the proposed
framework on each of the 481 cases (265 benign and 216
malignant) of the test set. The obtained classification results
in terms of Az value are presented in figure 6.
Analyzing the results, we could observe similar trends as
in the case of the validation set. It seems that exploiting the
BIRADS assessment (section A), the density of the breast
(section B) and the location of the cluster (section C) have a
positive impact on performance since the achieved Az values
present an ascending trend and they are greater than the corre­
sponding value achieved when only the CAD1 scheme is considered. The main difference between these results and those
previously reported in the case of the validation set (figure 4) is
related to the performance of the CADR scheme. The specific
Az value (0.829) is greater than the value of 0.746 provided
by the CAD1 scheme, in contrast to the corresponding values
on the validation set (figure 4), where the Az values achieved
by CAD1 and CADR were almost identical. However, the
combination of both the CADR and the proposed framework
seems to have the potential to further increase the overall performance, since the 0.862 Az value is the greatest among all.
As far as the comparison with the radiologists is concerned,
we estimate the corresponding values of accuracy, sensitivity
and specificity. The results are presented in table 2. The proposed framework has been validated using both thresholds
(31% and 15%) obtained through the process described on the
validation set.
The results are quite encouraging, indicating that the combined use of the developed CADx schemes may lead to better
6. Discussion
In this paper, a novel design and evaluation approach for the
refinement of the computer-aided diagnosis of clustered MCs
is proposed. Aligned with the demand that CAD systems
should assist the radiologist and cooperate with him in the
interpretation of mammograms, we proceed on the development of CADx schemes that are based on automatic processes
guided by features provided by the radiologist.
The framework proposed has been obtained after rigorous
evaluation, in an attempt to adopt the most important outcomes
and conclusions from our previous studies. We have already
demonstrated that inherent properties of mammograms dramatically affect the performance of CADx approaches and we
have indicated that features provided by radiologists, such as
BIRADS assessment without using CAD approaches or the
density of the breast, may require different computational
pipelines for achieving improved performance [11]. We have
also investigated the role of the location of the cluster of MCs
inside the breast and managed to extract quantified information that supports the diagnostic task [12].
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
Table 2. Proposed framework’s and radiologists’ performance on the test set.
Proposed framework (threshold 31%)
Proposed framework (threshold 15%)
The major contribution of this work concerns the integration of these conclusions into a unified framework, where
alternative CADx schemes are selected and employed, based
on features provided by the radiologist who performs the analysis of a suspicious finding. Instead of using a single CADx
scheme, alternative methodologies are developed in order
to exploit more ‘specified’ tools depending on the mammogram’s properties. Each CADx scheme includes the use of
several machine learning algorithms, combined into a voting
scheme, in an attempt to ensure satisfactory diversity of the
classification rules [12, 29]. Obscure cases, such as cases classified as probably malignant needing biopsy, are evaluated by
more schemes to ensure in-depth analysis. The information
concerning the relative location of the cluster inside the breast
is also exploited as an independent diagnostic component, that
shifts properly the risk computed by the image analysis of the
ROI. Finally, an independent diagnostic component is developed, based on the descriptors provided by the radiologists
and the age of the patient. This component plays a primary
role in the computation of the final risk provided by the proposed framework, so as to highlight the role and the contrib­
ution of the radiologist in the flow of the CAD processes.
A large dataset consisting of 1715 cases, provided by the
publicly available DDSM database, has been properly separated in the training, validation and test subsets to enable
the development of various CADx schemes, the design of
the novel framework and its proper evaluation, respectively.
The results reported on the validation set reveal the potential
of the proposed pipeline to enhance the diagnostic process.
These outcomes are clearly verified on the test set. Each of the
framework’s sections considered seems to provide a positive
impact on the classification performance. Based on the results
presented in figures 4 and 6, it appears that the proposed pipeline outperforms the use of a single CADx scheme (CAD1)
trained with all the available cases of the training set.
Regarding the straightforward comparison between the
proposed framework and the radiologists who assessed the
exact same mammograms, we extract promising conclusions
for the potential of our approach to be implemented in daily
clinical practice. As presented in tables 1 and 2, the results
achieved are superior to those of the radiologists on both the
validation and the test set. More specifically, in the case of
the test set, the sensitivity of the radiologists is quite high
(93.5%) indicating that almost all malignant cases are recommended for biopsy. However, both the accuracy (51.1%) and
the specificity (16.6%) are too low. These values demonstrate
that almost one out of two cases are misclassified, leading to a
great number of unnecessary biopsies for patients with benign
findings. Despite the fact that based on the radiologists’
descriptors we could develop a relatively powerful diagnostic
component (CADR), the final assessment performed according
to BIRADS categories induces poor results. Probably, the pursuit of radiologists to eliminate missed malignant cases (false
negative recommendations) leads to an increased number of
false positive recommendations and, thus, to lower classification performance. On the contrary, through the proposed
approach all metrics are improved. The sensitivity remains at
high levels and, actually, is slightly improved (97.2%), while
we manage to improve the achieved value of specificity more
than two times. This fact implies that the proposed system
seems to have the potential to reduce the number of unnecessary biopsies performed, by further minimizing the number of
false negative cases.
The training, validation and test sets considered were identical to those used in our previous work [12], so as to enable
fair comparison with the reported results, as well as with future
studies. In comparison to [12], where the contribution of the
breast probabilistic maps was exclusively studied, we observe
that the overall classification performance is further increased
in the current study. The positive impact of the topological
information provided by the projection of the ROI on breast
probabilistic maps (section C in figure 2) is verified also in the
current work, as indicated by the results in figures 4 and 6.
Nevertheless, the additional diagnostic components introduced in this study (CADR and alternative CADx schemes
in sections A and B of the framework presented in figure 2),
when combined to the breast probabilistic maps (section C
in figure 2), seem to have the potential to further improve the
overall discrimination ability. The Az value obtained in the
current study is 0.862, compared with 0.786 reported in [12].
The evaluation of new cases is not a computational-intensive process and it depends mainly on the size of the ROI. For
a typical ROI’s size the average time needed to estimate the
risk percentage based on the framework proposed is almost
equal to 25 seconds, making feasible its adoption in daily
clinical practice. Although the system performs towards the
right direction for the evaluation of MCs, we identify hereafter current limitations of the proposed system, that consist
also of future steps that we plan to follow towards improving
its usability. The efficacy of the proposed approach has been
investigated using the DDSM database which includes exclusively digitized screen film mammograms. The system should
be examined in the future with cases available from modern
full-field-digital mammography, preferably in a large openaccess database (including data based on the ACR-BIRADS
standard) which is not, however, currently available. Given the
encouraging results of this work, we believe that the improved
image quality associated with digital mammography could
improve further the achieved performance. Currently, the only
feature related to patient’s medical file is the age of the patient.
More features could be adopted and/or existing models (e.g.
the Gail model [26]) could be exploited in order to add to the
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
proposed framework (figure 2) extra independent diagnostic
components related to the analysis of patient’s record. The
proposed approaches should expand to other mammographic
findings (e.g. architectural distortions) and computational
algorithms could be introduced for the automated estimation
of critical features (e.g. breast density prediction). However,
the aforementioned problems are not trivial on their own and
require independent and rigorous research work. Additionally,
the need for such steps should be properly investigated, in an
attempt to maintain a balance between acceptable performance
and satisfactory use in clinical practice. In any event, a carefully
designed graphical user interface that implements the proposed
approaches is a prerequisite in order to adapt them to the radiologist’s daily clinical routine. An extended evaluation of the
system in clinical practice should then be properly designed, in
order to investigate several issues that are currently unknown
and may occur, such as how inter-radiologist variations on
mammographic analysis affect a system’s performance.
MCs in mammograms in daily clinical practice. Refining the
interaction between radiologists and CADx schemes, incorporating extra diagnostic components, taking into consideration
other mammographic findings, such as masses or architectural
distortions, and exploring the contribution of the proposed
framework’s recommendation to radiologist’s final assessment are indicative steps of our future work, towards further
improving the diagnostic process.
[1] American Cancer Society 2015–2016 Breast Cancer Facts and
Figures (Atlanta, GA: American Cancer Society)
[2] Pinto A and Brunese L 2010 Spectrum of diagnostic errors in
radiology World J. Radiol. 2 377–83
[3] Baldwin P 2013 Breast calcification imaging Radiol. Technol.
84 383–404
[4] Jalalian A, Mashohor S B, Mahmud H R, Saripan M I,
Ramli A R and Karasfi B 2013 Computer-aided detection/
diagnosis of breast cancer in mammography and ultrasound:
a review Clin. Imaging 35 420–26
[5] Wang J, Naqa I and Yang Y 2015 Classification of malignant
and benign tumors Machine Learning in Radiation
Oncology: Theory and Applications (Berlin: Springer) pp
[6] Elmore J, Nakano C, Koepsell T, Desnick L, D’Orsi C and
Ransohoff D 2003 International variation in screening
mammography interpretations in community-based
programs J. Natl Cancer Inst. 95 1384–93
[7] Barlow W, Chi C, Carney P, Taplin S, D’Orsi C, Cutter G,
Hendrick R and Elmore J 2004 Accuracy of screening
mammography interpretation by characteristics of
radiologists J. Natl Cancer Inst. 96 1840–50
[8] Elter M and Horsh A 2009 CADx of mammographic masses
and clustered microcalcifications: a review Med. Phys.
36 2052–68
[9] Jorritsma W, Cnossen F and van Ooijen P M 2015 Improving
the radiologist-CAD interaction: designing for appropriate
trust Clin. Radiol. 70 115–22
[10] Andreadis I, Spyrou G and Nikita K 2011 A comparative
study of image features for classification of breast
microcalcifications Meas. Sci. Technol. 22 114005–13
[11] Andreadis I, Spyrou G, Ligomenides P and Nikita K 2013
Variations on breast density and subtlety of the findings
require different computational intelligence pipelines for
the diagnosis of clustered microcalcifications 13th Int. IEEE
Conf. Bioinformatics and Bioengineering (Chania, Greece)
pp 1–4
[12] Andreadis I, Spyrou G and Nikita K 2015 A CADx scheme for
mammography empowered with topological information
from clustered microcalcifications’ atlases IEEE J. Biomed.
Health Inform. 19 166–73
[13] D’Orsi C J et al 2013 ACR BI-RADS® Atlas, Breast Imaging
Reporting and Data System (Reston, VA: American College
of Radiology)
[14] Gastounioti Α, Conant Ε and Kontos D 2016 Beyond breast
density: a review on the advancing role of parenchymal
texture analysis in breast cancer risk assessment Breast
Cancer Res. 18 91
[15] Dromain C, Boyer B, Ferré R, Canale S, Delaloge S and
Balleyguier C 2013 Computed-aided diagnosis (CAD) in
the detection of breast cancer Eur. J. Radiol. 82 417–23
[16] Heath M, Bowyer K, Kopand D, Moore R and Kegelmeyer W
2001 The digital database for screening mammography
Proc. 5th IWDM ed M Yaffe (Medical Physics Publishing)
pp 212–8
7. Conclusions
The main question investigated in this paper is the study of the
factors that influence the performance of a CADx system for
the diagnosis of clusters of MCs. Towards implementing our
main aim, which is the optimization of the diagnostic process,
and following the main principle that CAD systems should
assist in the radiologist’s task, we propose an approach where
features provided by the radiologist during the analysis of a
mammogram form the basis for the proper selection of computational pipelines for the CAD assessment of the case.
Instead of simply applying trained classification schemes,
new cases are processed by alternative pipelines (various classifiers, diverse enhancement techniques, different feature sets,
data from the patient’s medical file, breast topological information) the selection of which is properly determined by features critical to the radiologist. To this end (i) various CADx
schemes are developed, each of which adopts feature sets and
methodologies, so as to achieve more specialized analysis of
mammograms with specific inherent properties, (ii) a complete
framework is introduced, where radiologist’s features along
with the developed CADx schemes are properly combined to
refine the diagnostic process and (iii) extensive evaluation of
the approach is carried out and a straightforward comparison
to the performance of the radiologists is performed.
The evaluation, performed both on the validation and test
set, reveals that features provided by the radiologist, such as
the density of the breast, BIRADS assessment and the indication of the exact location of the cluster, affect the classification
performance. Overall, the results achieved are substantially
improved outperforming the performance achieved by the
radiologists. Given the high need for eliminating misclassification errors in medical processes, especially in the case of
malignant findings, the reported results are quite encouraging,
as we manage to retain high levels of sensitivity, by achieving
greater accuracy and specificity than the radiologists who
assessed the same mammograms. These outcomes highlight
the potential of the proposed approach to correctly diagnose
I Andreadis et al
Meas. Sci. Technol. 28 (2017) 114003
[24] Bent C, Bassett L, D’Orsi C and Sayre J 2010 The positive
predictive value of BI-RADS microcalcification descriptors
and final assessment categories Am. J. Roentgenol.
194 1378–83
[25] Kim S, Kim H, Kim E, Kim M, Moon H and Yoon J
2015 Evaluation of malignancy risk stratification of
microcalcifications detected on mammography: a study
based on the 5th edition of BI-RADS Ann. Surg. Oncol.
22 2285–901
[26] Matsuno R, Costantino J, Ziegler R, Anderson G, Li H,
Pee D and Gail M 2011 Projecting individualized
absolute invasive breast cancer risk in Asian and
Pacific Island American women J. Natl Cancer. Inst.
103 951–61
[27] Giannakopoulou G et al 2010 Downgrading BIRADS
3 to BIRADS 2 category using a computer-aided
microcalcification analysis and risk assessment system for
early breast cancer Comput. Biol. Med. 40 853–9
[28] Karahaliou A, Skiadopoulos S, Boniatis I, Sakellaropoulos P,
Likaki E, Panayiotakis P and Costaridou L 2007 Texture
analysis of tissue surrounding microcalcifications on
mammograms for breast cancer diagnosis Br. J. Radiol.
80 648–56
[29] Mougiakakou S, Valavanis I, Nikita K, Nikita A and Kelekis D
2003 Characterization of CT liver lesions based on texture
features and a multiple neural network classification scheme
Proc. Int. Conf. IEEE Engineering in Medicine and Biology
Society (EMBC) (Cancun, Mexico) pp 1287–90
[17] Lee A 2017 Inter-reader variability in the use of BI-RADS
descriptors for suspicious findings on diagnostic
mammography: a multi-institution study of 10 academic
radiologists Acad. Radiol. 24 60–6
[18] Spyrou G, Kapsimalakou S, Frigas A, Koufopoulos K,
Vassilaros S and Ligomenides P 2006 ‘Hippocrates-mst’:
a prototype for computer-aided microcalcification analysis
and risk assessment for breast cancer Med. Biol. Eng.
Comput. 44 1007–15
[19] Andreadis I, Spyrou G and Nikita K 2014 Investigating the
performance of a CADx scheme for mammography in
specific BIRADS categories Proc. Int. Conf. Imaging
Systems and Techniques (IST) (Santorini, Greece) pp 335–9
[20] Ekpo E, Ujong U, Mello-Thoms C and McEntee M 2016
Assessment of interradiologist agreement regarding
mammographic breast density classification using the
fifth edition of the BI-RADS atlas Am. J. Roentgenol.
206 1119–23
[21] Ciatto S et al 2005 Categorizing breast mammographic
density: intra- and interobserver reproducibility of
BI-RADS density categories Breast 14 269–75
[22] Baker J, Kornguth P, Lo J and Floyd C 1996 Artificial
neural network: improving the quality of breast biopsy
recommendations Radiology 198 131–5
[23] Lo J, Gavrielides M, Markey M and Jesneck J 2003 Computeraided classification of breast microcalcification clusters:
merging of features from image processing and radiologists
Proc. SPIE 5032 882–9
Без категории
Размер файла
1 006 Кб
1361, 6501, 2faa884e
Пожаловаться на содержимое документа