Inductive Fuzzy Classification in Marketing Analytics

Faculty of Science
Department of Informatics
University of Fribourg, Switzerland
Inductive Fuzzy Classification
in Marketing Analytics
Doctoral Thesis
presented to the Faculty of Science
of the University of Fribourg, Switzerland
for the award of the academic grade of
Doctor of Computer Science,
Doctor scientiarum informaticarum, Dr. sc. inf.
by
Michael Alexander Kaufmann
from Recherswil SO
Thesis No: 1751
Bern: Ruf
2012
2
For Alice.
Follow the white rabbit.
“The Caterpillar took the hookah out of its mouth and yawned once or
twice, and shook itself. Then it got down off the mushroom, and crawled
away into the grass, merely remarking as it went, “One side will make
you grow taller, and the other side will make you grow shorter.”
“One side of what? The other side of what? “ thought Alice to herself.
“Of the mushroom," said the Caterpillar, just as if she had asked it
aloud; and in another moment it was out of sight.“
Lewis Carrol (1916, p. 29)
3
4
Acknowledgements
A scientific thesis organically evolves from a primeval soup of
ideas originating from countless individuals past and present. I merely
tamed and organized these ideas and developed them further. In this
light, I thank all fellow human beings who made my thesis possible.
I thank my first advisor, Professor Andreas Meier, for teaching me
how to conduct scientific research. I will always remember those great
doctoral seminars we had all across Switzerland. Also, I thank my
second and third advisors, Professors Laurent Donzé and Kilian Stoffel,
for agreeing to evaluate my dissertation. I thank my fellow doctoral
students in the Information Systems Research Group, Daniel Fasel, Edy
Portmann, Marco Savini, Luis Teràn, Joël Vogt, and Darius Zumstein
for inspiring me with their research and for the all good times we had at
the educational and social events we attended together. Furthermore, I
thank my students Cédric Graf and Phillipe Mayer, who implemented
my research ideas with prototypes in their master’s theses. The former
achieved such a quality that we decided to publish his results in a book
chapter. Last, but not least, I thank David Wyder, professional data
miner, for allowing me to use PostFinance’s information system as a
research lab for the first experiments using my research approach,
which set the founding stone to my thesis proposal and to my first
publication.
Special thanks go to friends and family for supporting me through
years of hard work. I know it wasn’t always easy with me.
Rubigen, Switzerland, February 29th, 2012
Michael Alexander Kaufmann
5
6
Abstract
“Inductive fuzzy classification” (IFC) is the process of assigning
individuals to fuzzy sets for which membership functions are based on
inductive inferences. In this thesis, different methods for membership
function induction and multivariate aggregation are analyzed. For
univariate membership function induction, the current thesis proposes
the normalized comparisons (ratios and differences) of likelihoods. For
example, a normalized likelihood ratio can represent a membership
degree to an inductive fuzzy class. If the domain of the membership
function is numeric, continuous membership functions can be derived
using piecewise affine interpolation. If a target attribute is continuous,
it can be mapped into the “Zadehan” domain of numeric truth-values
between 0 and 1, and membership degrees can be computed by a
normalized ratio of likelihoods of fuzzy events.
A methodology for multivariate IFC for prediction has been
developed a part of this thesis: First, data is prepared into a matrix
format. Second, the relevant attributes are selected. Third, for each
relevant attribute, a membership function to the target class is induced.
Fourth, transforming the attributes into membership degrees in the
inductive fuzzy target class fuzzifies these attributes. Fifth, for every
data record, the membership degrees of the single attribute values are
aggregated into a membership degree in the target class. Finally, the
prediction accuracy of the inductive membership degrees in comparison
to the target variable is evaluated.
The proposed membership function induction method can be
applied to analytics for selection, visualization, and prediction. First,
transformation of attributes into inductive membership degrees in fuzzy
target classes provides a way to test the strength of target associations
of categorical and numerical variables using the same measure. Thus,
relevant attributes with a high target correlation can be selected.
Second, the resulting membership functions can be visualized as a
graph. This provides an intuitive depiction of target associations for
different values of relevant attributes and allows human decision
makers to recognize interesting parts of attribute domains regarding
the target. Third, transformation of attribute values into membership
degrees in inductive fuzzy classes can improve prediction of statistical
models because nonlinear associations can be represented in
membership functions.
In marketing analytics, these methods can be applied in several
domains. Customer analytics on existing data using attribute selection
and visualization of inductive membership functions provide insights
into different customer classes. Product analytics can be improved by
7
evaluating likelihood of product usage in the data for different customer
characteristics with inductive membership functions. Transforming
customer attributes into inductive membership degrees, which are
based on predictive models, can optimize target selection for individual
marketing and enhance the response rate of campaigns. This can be
embedded in integrated analytics for individual marketing.
A case study is presented, in which IFC was applied in online
marketing of a Swiss financial service provider. Fuzzy classification was
compared to crisp classification and to random selection. The case study
showed that, for individual marketing, a scoring approach can lead to
better response rates than a segmentation approach because of
compensation of threshold effects.
A prototype was implemented that supports all steps of the
prediction methodology. It is based on a script interpreter that
translates inductive fuzzy classification language (IFCL) statements
into corresponding SQL commands and executes them on a database
server. This software supports all steps of the proposed methodology,
including data preparation, membership function induction, attribute
selection, multivariate aggregation, data classification and prediction,
and evaluation of predictive models.
The software IFCL was applied in an experiment in order to
evaluate the properties of the proposed methods for membership
function induction. These algorithms were applied to 60 sets of real
data, of which 30 had a binary target variable and 30 had a gradual
target variable, and they were compared to existing methods. Different
parameters were tested in order to induce an optimal configuration of
IFC. Experiments showed a significant improvement in average
predictive performance of logistic regression for binary targets and
regression trees for gradual targets when, prior to model computation,
the attributes were inductively fuzzified using normalized likelihood
ratios or normalized likelihood differences respectively.
8
Zusammenfassung
Induktive unscharfe Klassifikation (inductive fuzzy classification,
IFC) ist der Prozess der Zuordnung von Individuen zu unscharfen
Mengen,
deren
Zugehörigkeitsfunktionen
auf
induktiven
Schlussfolgerungen basieren. In der vorliegenden These werden
verschiedene Methoden für die Induktion von Zugehörigkeitsfunktionen
und deren multivariaten Aggregation analysiert. Es wird
vorgeschlagen,
für
die
Induktion
von
univariaten
Zugehörigkeitsfunktionen Vergleiche (Verhältnisse und Differenzen)
von Wahrscheinlichkeiten (Likelihoods) zu normalisieren. Zum Beispiel
kann das normalisierte Wahrscheinlichkeitsverhältnis (normalized
Likelihood Ratio, NLR) einen Zugehörigkeitsgrad zu einer induktiven
unscharfen Klasse repräsentieren. Wenn der Wertebereich der
Zugehörigkeitsfunktion
nummerisch
ist,
können
stetige
Zugehörigkeitsfunktionen über eine stückweise affine Interpolation
hergeleitet werden. Wenn die Zielvariable stetig ist, kann sie in den
„Zadeh’schen“ Wertebereich der nummerischen Wahrheitswerte
zwischen 0 und 1 abgebildet werden; und die Zugehörigkeitsgrade
können als normalisierte Verhältnisse von empirischen bedingten
Wahrscheinlichkeiten unscharfer Ereignisse (Likelihoods of fuzzy
events) berechnet werden.
Eine Methode für multivariate induktive unscharfe Klassifikation
wurde in dieser These entwickelt. Als erstes werden die Daten in einem
Matrix-Format aufbereitet. Als zweites werden die relevanten Attribute
ausgewählt. Drittens wird für jedes relevante Attribut eine
Zugehörigkeitsfunktion
zur
Zielklasse
mittels
normalisierten
Likelihood-Vergleichen induziert. Viertens werden die Attributwerte in
Zugehörigkeitsgrade
zur
induktiven
unscharfen
Zielklasse
transformiert. Fünftens werden die Zugehörigkeitsgrade der einzelnen
Attributwerte jedes Datensatzes in einen Zugehörigkeitsgrad in der
Zielklasse aggregiert. Schliesslich wird die Vorhersageleistung der
induktiven Zugehörigkeitsgrade im Vergleich mit der effektiven
Zielvariable evaluiert.
Die vorgeschlagenen Methoden können in der Analytik für
Selektion, Visualisierung und Prognose angewendet werden. Erstens
bietet die Transformation von Attributen in Zugehörigkeitswerte zu
induktiven unscharfen Mengen ein Mittel, um die Stärke der
Zielassoziation von nummerischen und kategorischen Variablen mit der
gleichen Metrik zu testen. So können relevante Attribute mit einer
hohen Korrelation mit dem Ziel selektiert werden. Zweitens können die
resultierenden Zugehörigkeitsfunktionen als Graphen visualisiert
werden. Das bietet eine intuitive Darstellung der Zielassoziation von
9
verschiedenen Werten von relevanten Attributen, und ermöglicht es
menschlichen Entscheidungsträgern, die interessanten Teilbereiche der
Attributdomänen bezüglich des Ziels zu erkennen. Drittens kann die
Transformation der Attributwerte in Zugehörigkeitswerte zu
induktiven unscharfen Klassen die Prognose von statistischen Modellen
verbessern,
weil
nicht-lineare
Zusammenhänge
in
den
Zugehörigkeitsfunktionen abgebildet werden können.
In der Marketing-Analytik können diese Methoden in
verschiedenen Bereichen angewendet werden. Kundenanalytik
aufgrund von bestehenden Daten mittels Attributselektion und
Visualisierung von induktiven Zugehörigkeitsfunktionen liefert
Erkenntnisse über verschiedene Kundenklassen. Produktanalytik kann
verbessert werden, indem die Wahrscheinlichkeit der Produktnutzung
in den Daten für verschiedene Kundenmerkmale mit induktiven
Zugehörigkeitsfunktionen evaluiert wird. Zielgruppenselektion im
individualisierten Marketing kann optimiert werden, indem
Kundenattribute in induktive Zugehörigkeitsgrade transformiert
werden, um die Rücklaufrate von Kampagnen zu erhöhen, welche auf
prädiktiven Modellen basieren. Dies kann für individualisiertes
Marketing in die integrierte Analytik eingebettet werden.
Eine Fallstudie wird vorgestellt, in der die induktive unscharfe
Klassifikation bei einem Schweizer Finanzdienstleister im online
Marketing angewendet wurde. Die unscharfe Klassifikation wurde mit
der scharfen Klassifikation und einer Zufallsauswahl verglichen. Dies
zeigte, dass im individualisierten Marketing ein Scoring-Ansatz zu
besseren Rücklaufquoten führen kann als ein Segmentierungsansatz,
weil Schwellenwerteffekte kompensiert werden können.
Ein Prototyp wurde implementiert, welcher alle Schritte der
multivariaten IFC-Methode unterstützt. Er basiert auf einem SkriptInterpreter, welcher Aussagen der Sprache für die induktive unscharfe
Klassifikation (Inductive Fuzzy Classification Language, IFCL) in
entsprechende
SQL-Kommandos
übersetzt
und
auf
einem
Datenbankserver ausführt. Diese Software unterstützt sämtliche
Schritte
der
vorgeschlagenen
Methode,
einschliesslich
Datenvorbereitung,
Induktion
von
Zugehörigkeitsfunktionen,
Attributselektion, multivariate Aggregation, Datenklassifikation,
Prognose, und die Evaluation von Vorhersagemodellen.
Die Software wurde in einem Experiment angewendet, um die
Eigenschaften der vorgeschlagenen Methoden der Induktion von
Zugehörigkeitsfunktionen zu testen. Diese Algorithmen wurden bei
sechzig Dateien mit realen Daten angewendet, davon bei dreissig mit
binärer Zielvariable und dreissig mit einer graduellen Zielvariable. Sie
wurden mit existierenden Methoden verglichen. Verschiedene
Parameter wurden getestet, um eine optimale Konfiguration der
10
induktiven unscharfen Klassifikation zu induzieren. Die Experimente
zeigten, dass die durchschnittliche Prognoseleistung der logistischen
Regression für binäre Zielvariablen und der Regressionsbäume für
graduelle Zielvariablen signifikant verbessert werden kann, wenn vor
der Berechnung der Prognosemodelle die Attribute mit normalisierten
Likelihood-Verhältnissen respektive mit normalisierten LikelihoodDifferenzen in Zugehörigkeitsgrade zu induktiven unscharfen Mengen
transformiert werden.
11
12
Table of Contents
Acknowledgements ..................................................................................... 5 Abstract ....................................................................................................... 7 Zusammenfassung ...................................................................................... 9 Table of Contents ...................................................................................... 13 List of Figures ........................................................................................... 17 List of Tables ............................................................................................. 19 1 A Gradual Concept of Truth ............................................................... 21 1.1 Research Questions ...................................................................... 22 1.2 Thesis Structure ........................................................................... 24 2 Fuzziness & Induction ........................................................................ 27 2.1 Deduction ...................................................................................... 27 2.1.1 Logic ....................................................................................... 27 2.1.2 Classification.......................................................................... 30 2.2 Fuzziness ...................................................................................... 33 2.2.1 Fuzzy Logic ............................................................................ 35 2.2.2 Fuzzy Classification............................................................... 40 2.3 Induction ....................................................................................... 43 2.3.1 Inductive Logic....................................................................... 43 2.3.2 Inductive Classification ......................................................... 46 2.4 Inductive Fuzzy Classification ☼ ................................................ 51 2.4.1 Univariate Membership Function Induction ....................... 52 2.4.2 Multivariate Membership Function Induction ☺ ................ 60 3 Analytics & Marketing ....................................................................... 63 3.1 Analytics ☼ ................................................................................... 63 3.1.1 Selection ................................................................................. 67 3.1.2 Visualization .......................................................................... 69 3.1.3 Prediction ............................................................................... 71 3.2 Marketing Analytics ☼ ................................................................ 75 3.2.1 Customer Analytics ............................................................... 76 3.2.2 Product Analytics .................................................................. 78 13
3.2.3 Fuzzy Target Groups ............................................................. 80 3.2.4 Integrated Analytics .............................................................. 82 3.2.5 Case Study: IFC in Online Marketing at PostFinance ☺ ... 85 4 Prototyping & Evaluation .................................................................. 93 4.1 Software Prototypes ..................................................................... 93 4.1.1 IFCQL..................................................................................... 93 4.1.2 IFCL ....................................................................................... 94 4.1.3 IFC-Filter for Weka ☼ ........................................................... 99 4.2 Empirical Tests .......................................................................... 105 4.2.1 Experiment Design .............................................................. 105 4.2.2 Results .................................................................................. 107 5 Precisiating Fuzziness by Induction ................................................ 115 5.1 Summary of Contribution .......................................................... 115 5.2 Discussion of Results ................................................................. 117 5.3 Outlook and Further Research .................................................. 119 6 Literature References ....................................................................... 123 A. Appendix .......................................................................................... 129 A.1. Datasets used in the Experiments ........................................... 129 A.1.1. Attribute Selection Filters ................................................. 133 A.1.2. Record Selection Filters ..................................................... 134 A.2. Database Queries...................................................................... 135 A.2.1. Database Queries for Experiments with Sharp Target
Variables .......................................................................................... 135 A.2.2. Database Queries for Experiments with Numerical Target
Variables .......................................................................................... 138 A.2.3. Database Queries for Both Types of Experiments ........... 143 A.3. Experimental Results Tables ................................................... 149 A.4. IFCL Syntax and Application .................................................. 157 A.4.1. Grammar Notation............................................................. 157 A.4.2. IFCL File Structure ........................................................... 157 A.4.3. Executing IFCL Files in Batch Mode ................................ 158 A.4.4. Connecting to the Database .............................................. 158 A.4.5. Droping a Database Table ................................................. 159 14
A.4.6. Executing an SQL Script ................................................... 159 A.4.7. Loading Data into The Database ...................................... 159 A.4.8. Inducing a Membership Function ..................................... 161 A.4.9. Classification of Data ......................................................... 162 A.4.10. Aggregating Multiple Variables ...................................... 162 A.4.11. Evaluating Predictions .................................................... 163 A.4.12. Data Preparation ............................................................. 164 A.4.13. Attribute Selection ........................................................... 165 A.5. Curriculum Vitae ...................................................................... 167 A.6. Key Terms and Definitions ...................................................... 167 ☺ This section is based on a publication by Kaufmann & Meier (2009)
☼ This section is based on a publication by Kaufmann & Graf (2012)
15
16
List of Figures
Figure 1: The motivation of the current research was to develop
and evaluate inductive methods for the automated derivation of
membership functions for fuzzy classification and to propose
possible applications to marketing analytics. ......................................... 23 Figure 2: Thesis structure. This thesis has three thematic
categories that are reflected in the structure: theory, application,
and technology. ......................................................................................... 25 Figure 3: There is no square. Adapted from “Subjective Contours”
by G. Kaniza, 1976, ................................................................................... 33 Figure 4: Black or white: conventional yin and yang symbol with a
sharp distinction between opposites, representing metaphysical
dualism. ..................................................................................................... 34 Figure 5: Shades of grey: fuzzy yin and yang symbol with a
gradation between opposites, representing metaphysical monism. ...... 34 Figure 6: A visualization of a classical set with sharp boundaries. ....... 36 Figure 7: A visualization of a fuzzy set.................................................... 36 Figure 8: Fuzzy set theory applied to the sorites paradox. .................... 37 Figure 9: Unsupervised IFC by percentile rank. .................................... 53 Figure 10: Computation of membership functions for numerical
variables. ................................................................................................... 60 Figure 11: Proposed method for multivariate IFC. ................................. 60 Figure 12: Visualization of relevant attributes and their
association with the target as an inductive membership function. ....... 70 Figure 14: Proposed schema for multivariate IFC for prediction. ......... 72 Figure 15: IFC prediction with binary target classes based on
categorical attributes. ............................................................................... 73 Figure 16: IFC prediction with Zadehan target classes based on
numerical attributes. ................................................................................ 73 Figure 17: Proposed areas of application of IFC to marketing
analytics. ................................................................................................... 76 Figure 18: Schema of a fuzzy customer profile based on IFC-NLR,
together with two examples of a fuzzy customer profile. ....................... 78 17
Figure 19: Inductive fuzzy classification for individual marketing. ...... 84 Figure 20: Analytics applied to individual marketing in the online
channel at PostFinance. ........................................................................... 86 Figure 21: Membership function induction for a categorical
attribute. ................................................................................................... 88 Figure 22: Membership function induction for a continuous
attribute. ................................................................................................... 89 Figure 23: Architecture of the iFCQL interpreter. ................................. 94 Figure 24: Inductive fuzzy classification language (IFCL) system
architecture. .............................................................................................. 95 Figure 25: Software architecture of the IFC-Filter............................... 100 Figure 26: Knowledge flow with IFC-Filters. ........................................ 101 Figure 27: Visualization of membership functions with the IFCFilter. ....................................................................................................... 103 Figure 28: Average prediction correlation for different aggregation
methods for binary targets. .................................................................... 107 Figure 29: Average prediction correlation for different aggregation
methods for fuzzified numerical targets. ............................................... 108 Figure 30: Average prediction correlation for combinations of
membership function induction and aggregation methods for
binary targets. ......................................................................................... 109 Figure 31: Average prediction correlation of predictions with
linear versus percentile ITF, given regression trees as the
aggregation method. ............................................................................... 109 Figure 32: Average prediction correlation for combinations of
membership function induction and aggregation methods for
fuzzified numerical targets. ................................................................... 110 Figure 33: Relationship between target linearity and IAF benefit
for binary target variables. .................................................................... 111 Figure 34: Relationship between target linearity and IAF benefit
for fuzzified numerical target variables. ............................................... 112 18
List of Tables
Table 1 Proposed Formulae for Induction of Membership Degrees ....... 57 Table 2 Example of an Attribute Ranking regarding NLR/Target
Correlation ................................................................................................ 68 Table 3 Conditional Probabilities and NLRs for a Categorical
Attribute .................................................................................................... 87 Table 4 Optimization of the Gamma Parameter for Multivariate
Fuzzy Aggregation .................................................................................... 90 Table 5 Resulting Product Selling Ratios per Target Group ................. 92 Table 6 Different Aggregation Methods used for Prediction
Experiments ............................................................................................ 106 Table 7 Correlation and p-Value of Dataset Parameters and
Improvement of Logistic Regression with IAF for Binary Target
Variables ................................................................................................. 111 Table 8 Correlation and p-Value of Dataset Parameters and
Improvement of Regression Trees with IAF for Zadean Target
Variables ................................................................................................. 112 Table 9 Data Sets with Categorical Target Variable, Transformed
into a Binary Target ............................................................................... 129 Table 10 Data Sets with Numerical Target Variable, Transformed
into a Fuzzy Proposition µ (see Formula 46 and Formula 47) ............ 131 ↑
Table 11 Average Prediction Correlations per Dataset/Aggregation
Method for Binary Targets ..................................................................... 149 Table 12 Average Prediction Correlations per Dataset and
Aggregation Method for Numerical Target Variables .......................... 150 Table 13: Average Prediction Correlations for IAF on Data with
Numerical Target Variables using Linear versus Percentile ITF ....... 151 Table 14 Average Prediction Correlations per Dataset and Three
Different Combinations of Supervised Aggregation and IAF for
Binary Target Variables ......................................................................... 152 Table 15 Average Prediction Correlations per Dataset and Three
Different Combinations of Supervised Aggregation and IAF for
Numerical Target Variables ................................................................... 153 19
Table 16 Relationship between Target Linearity and Improvement
of Logistic Regression by IAF NLR for Binary Target Variables ......... 154 Table 17 Relationship between Target Linearity and Improvement
of Regression Trees by IAF NLD for Numerical Target Variables ...... 155 20
1 A Gradual Concept of Truth
“Would you describe a single grain of wheat as a heap? No. Would
you describe two grains of wheat as a heap? No. … You must admit the
presence of a heap sooner or later, so where do you draw the line?”
(Hyde, 2008)
How many grains does it take to constitute a heap? This question
is known as the sorites paradox (Hyde, 2008). It exemplifies that our
semantic universe is essentially vague, and with any luck, this
vagueness is ordinal and gradual. This applies to all kinds of
statements. Especially in science, different propositions or hypotheses
can only be compared to each other with regard to their relative
accuracy or predictive power. Fuzziness is a term that describes
vagueness in the form of boundary imprecision. Fuzzy concepts are
those that are not clearly delineated, such as the concept of a “heap of
grain.”
The classical notion of truth claims metaphysical dualism, and
thus divides thought into exactly two categories: true and false. A
gradual concept of truth leads our consciousness toward a metaphysical
monism: all possible statements belong to the same class; but there is a
gradation of degree. The continuum of propositions, ranging from
completely false to completely true, contains all the information that is
in-between.
Fuzzy set theory provides a tool for mathematically precise
definitions of fuzzy concepts, if those concepts can be ordered: assigning
gradual membership degrees to their elements. This gradual concept of
truth is the basis for fuzzy logic or approximate reasoning, as proposed
by Zadeh (1975a) and Bellmann and Zadeh (1977). Fuzzy logic, based on
the concept of fuzzy sets introduced by Zadeh (1965), allows
propositions with a gradual truth-value and, thus, supports
approximate reasoning, gradual and soft consolidation.
Fuzzy propositions define fuzzy classes, which allow gradual,
fuzzy class boundaries. In data analysis, or “the search for structure in
data” (Zimmermann H. J., 1997), fuzzy classification is a method for
gradation in data consolidation, as presented by Meier, Schindler, and
Werro (2008) and Del Amo, Montero, and Cutello (1999). The
application of fuzzy classification to marketing analytics (Spais &
Veloutsou, 2005) has the advantage of precisiation (sic; Zadeh, 2008) of
fuzzy concepts in the context of decision support for direct customer
contact, as proposed by Werro (2008). This precisiation can be achieved
by inducing membership functions to fuzzy target classes (Setnes,
Kaymak, & van Nauta Lemke, 1998).
21
Inductive, or probabilistic, inference and fuzzy, or gradual, logic
have been seen as incompatible, for example, by Elkan (1993). Whereas
probabilistic induction can amplify experience, membership functions
can precisiate vagueness. Combined, measured probabilities can be
applied to precisely define the semantics of vague or fuzzy concepts by
membership function induction. This thesis, eventually, demonstrates
how probabilistic and fuzzy logics can be synthesized to constitute a
method of inductive gradual reasoning for classification.
1.1
Research Questions
Werro (2008) and Meier et al. (2008) proposed the application of
Fuzzy classification to customer relationship management (CRM). A
suggestion for further research by Werro (2008), namely the integration
of data mining techniques into fuzzy classification software, has
inspired the present thesis. As illustrated in Figure 1, the motivation of
the current research was to develop and evaluate inductive methods for
the automated derivation of membership functions for fuzzy
classification and to propose possible applications to marketing
analytics.
The first motivation for research on “inductive fuzzy classification”
(IFC) was the development of methods for automated derivations of
understandable and interpretable membership functions to fuzzy
classes. The aim was to develop and evaluate algorithms and methods
to induce functions that indicate inductively inferred target class
memberships.
The second motivation was to improve marketing analytics with
IFC. Kaufmann and Meier (2009) have presented a methodology and a
case study for the application of IFC to predictive product affinity
scoring for target selection. This thesis extends the application of this
methodology to marketing in the field of integrated customer and
product analytics in order to provide means to deal with fuzziness in
marketing decisions and to enhance accountability by application of
analytics to precisiate fuzzy marketing concepts, as proposed by Spais
and Veloutsou (2005).
The third motivation was to develop a prototype implementation
of software for computing IFCs, showing the feasibility of the proposed
algorithms as a proof of concept and allowing an evaluation of the
proposed methodology.
Seven research questions were developed at the beginning of the
current research (Kaufmann, 2008). These questions guided the
development of the thesis from the beginning. The answers are
22
presented in the subsequent document and summarized in the
conclusions of the thesis.
1. What is the theoretical basis of IFC and what is its relation to
inductive logic?
2. How can membership functions be derived inductively from data?
3. How can a business enterprise apply IFC in practice?
4. How can the proposed methods be implemented in a computer
program?
5. How is IFC optimally applied for prediction?
6. Which aggregation methods are optimal for the multivariate
combination of fuzzy classes for prediction?
7. Can it be statistically supported that the proposed method of IFC
improves predictive performance?
Figure 1: The motivation of the current research was to develop and evaluate inductive
methods for the automated derivation of membership functions for fuzzy classification
and to propose possible applications to marketing analytics.
Following a constructive approach to business informatics and
information systems research (Oesterle, et al., 2010), this thesis
presents a design of new methods for IFC and proposes applications of
23
these new methods to analytics in marketing. Thus, the methodology for
developing the thesis includes the following: The theoretical background
is analyzed prior to presentation of the constructive design, a case study
exemplifies the designed approach, software prototyping shows
technical feasibility and enables experimental evaluation, and empirical
data collection in combination with statistical inference is applied to
draw conclusions about the proposed method.
1.2
Thesis Structure
As shown in Figure 2, this thesis has three thematic categories
that are reflected in the structure. First, the theoretical foundations of
IFC are examined. This theory is based on a synthesis of fuzzy (gradual)
and inductive (probabilistic) logics. Second, business applications of this
approach are analyzed. A possible application is studied for analytic
quantitative decision support in marketing. Third, technological aspects
of the proposed method are examined by software prototypes for
evaluation of the proposed constructs.
The second chapter, which addresses theory, analyzes the
theoretical foundations of IFC by approaching it from the viewpoints of
logic, fuzziness, and induction. It presents proposals of likelihood-based
methods for membership function induction. In Section 2.1, the basic
concepts of logic and classification are analyzed. In Section 2.2,
cognitive and conceptual fuzziness are discussed and approaches for
resolution of conceptual fuzziness, fuzzy logic, and fuzzy classification
are outlined. In Section 2.3, inductive classification, and induction
automation are examined. In Section 2.4, the application of induction to
fuzzy classification is explained, and a methodology for membership
function induction using normalized ratios and differences of empirical
conditional probabilities and likelihoods is proposed.
The third chapter, which deals with designing applications of IFC,
presents a study of analytics and examines the application of
membership function induction to this discipline in marketing. In
Section 3.1, the general methodology of logical data analysis, called
analytics, is analyzed, and applications of IFC to three sub-disciplines,
selection, visualization, and prediction, are proposed. In Section 3.2,
applications of IFC to analytic marketing decision support, or
marketing analytics (MA), are listed. A case study is presented, in
which the proposed methods are applied to individual online marketing.
The fourth chapter, which presents designs of technology for IFC,
explains prototype implementations of the proposed IFC methodology
and shows evaluation results. In Section 4.1, three prototype
implementations of IFC are presented. The prototype of an inductive
24
fuzzy classification language (IFCL) is described. This description
encompasses the architecture of the software and its functionality. Two
additional prototypes that were developed in collaboration with master’s
students guided by the author, iFCQL (Mayer, 2010) and IFC-Filter
(Graf, 2010), are briefly discussed. In Section 4.2, a systematic
experimental application of the IFCL prototype is described and the
empirical results of testing the predictive performance of different
parameters of the proposed methodology are statistically analyzed.
Figure 2: Thesis structure. This thesis has three thematic categories that are reflected
in the structure: theory, application, and technology.
The fifth chapter, which concludes the thesis, summarizes the
scientific contributions of the current research, discusses the results,
and proposes further research topics for IFC. Literature references are
indicated in the sixth chapter. Research details can be found in Chapter
A, the appendix at the end of the thesis.
25
26
2 Fuzziness & Induction
This chapter examines the foundations of IFC by analyzing the
concepts of deduction, fuzziness, and induction. The first subsection
explains the classical concepts of sharp and deductive logic and
classification; in this section, it is presupposed that all terms are clearly
defined. The second section explains what happens when those
definitions have fuzzy boundaries and provides the tools, fuzzy logic and
fuzzy classification, to reason about this. However, there are many
terms that do not only lack a sharp boundary of term definition but also
lack a priori definitions. Therefore, the third subsection discusses how
such definitions can be inferred through inductive logic and how such
inferred propositional functions define inductive fuzzy classes. Finally,
this chapter proposes a method to derive precise definitions of vague
concepts—membership functions—from data. It develops a methodology
for membership function induction using normalized likelihood
comparisons, which can be applied to fuzzy classification of individuals.
2.1
Deduction
This subsection discusses deductive logic and classification,
analyzes the classical as well as the mathematical (Boolean) approaches
to propositional logic, and shows their application to classification.
Deduction provides a set of tools for reasoning about propositions with a
priori truth-values—or inferences of such values. Thus, in the first
subsection, the concepts of classical two-valued logic and algebraic
Boolean logic are summarized. The second subsection explains how
propositional functions imply classes and, thus, provide the mechanism
for classification.
2.1.1
Logic
In the words of John Stuart Mill (1843), logic is “the science of
reasoning, as well as an art, founded on that science” (p. 18). He points
out that the most central entity of logic is the statement, called a
proposition:
The answer to every question which it is possible to
frame, is contained in a proposition, or assertion. Whatever
can be an object of belief, or even of disbelief, must, when put
into words, assume the form of a proposition. All truth and
all error lie in propositions. What, by a convenient
misapplication of an abstract term, we call a truth, is simply
a true proposition. (p. 27)
27
The central role of propositions indicates the importance of
linguistics in philosophy. Propositions are evaluated for their truth, and
thus, assigned a truth-value because knowledge and insight is based on
true statements.
Consider the universe of discourse in logic: The set of possible
statements or propositions, 𝒫𝒫. Logicians believe that there are different
levels of truth, usually two (true or false); in the general case, there is a
set, 𝒯𝒯, of possible truth-values that can be assigned to propositions.
Thus, the proposition 𝑝𝑝 ∈ 𝒫𝒫 is a meaningful piece of information to
which a truth-value, 𝜏𝜏(𝑝𝑝) ∈ 𝒯𝒯, can be assigned. The corresponding
mapping of 𝜏𝜏: 𝒫𝒫 ⟶ 𝒯𝒯 from propositions 𝒫𝒫 to truth-values 𝒯𝒯 is called a
truth function.
In general logic, operators can be applied to propositions. A unary
operator, 𝑂𝑂 : 𝒫𝒫 ⟶ 𝒯𝒯, maps a single proposition into a set of
transformed truth-values. Accordingly, a binary operator, 𝑂𝑂 : 𝒫𝒫 ×𝒫𝒫 ⟶
𝒯𝒯, assigns a truth-value to a combination of two propositions, and an nary operator, 𝑂𝑂 : 𝒫𝒫 × ⋯×𝒫𝒫 ⟶ 𝒯𝒯, is a mapping of a combination of n
propositions to a new truth-value.
The logic of two-valued propositions is the science and art of
reasoning about statements that can be either true or false. In the case
of two-valued logic, or classical logic (𝐶𝐶𝐶𝐶), the set of possible truth
values, 𝒯𝒯 : = {true, false}, contains only two elements, which partitions
the class of imaginable propositions 𝒫𝒫 into exactly two subclasses: the
class of false propositions and the class of true ones.
With two truth-values, there is only one possible unary operator
other than identity: A proposition, 𝑝𝑝 ∈ 𝒫𝒫, can be negated (not 𝑝𝑝), which
inverts the truth-value of the original proposition. Accordingly, for a
combination of two propositions, 𝑝𝑝 and 𝑞𝑞, each with two truth-values,
there are 16 (2 ) possible binary operators. The most common binary
logical operators are disjunction, conjunction, implication, and
equivalence: A conjunction of two propositions, 𝑝𝑝 and 𝑞𝑞, is true if both
propositions are true. A disjunction of two propositions, 𝑝𝑝 or 𝑞𝑞, is true if
one of the propositions is true. An implication of 𝑞𝑞 by 𝑝𝑝 is true if,
whenever 𝑝𝑝 is true, 𝑞𝑞 is true as well. An equivalence of two propositions
is true if 𝑝𝑝 implies 𝑞𝑞 and 𝑞𝑞 implies 𝑝𝑝.
Classical logic is often formalized in the form of a propositional
calculus. The syntax of classical propositional calculus is described by
the concept of variables, unary and binary operators, formulae, and
truth functions. Every proposition is represented by a variable (e.g., 𝑝𝑝);
every proposition and every negation of a proposition is a term; every
combination of terms by logical operators is a formula; terms and
formulae are themselves propositions; negation of the proposition 𝑝𝑝 is
represented by ¬𝑝𝑝; conjunction of the two propositions 𝑝𝑝 and 𝑞𝑞 is
represented by 𝑝𝑝 ∧ 𝑞𝑞; disjunction of the two propositions 𝑝𝑝 and 𝑞𝑞 is
28
represented by 𝑝𝑝 ∨ 𝑞𝑞; implication of the proposition 𝑞𝑞 by the proposition
𝑝𝑝 is represented by 𝑝𝑝 ⇒ 𝑞𝑞; equivalence between the two propositions 𝑝𝑝
and 𝑞𝑞 is represented by 𝑝𝑝 ≡ 𝑞𝑞; and there is a truth function, 𝜏𝜏 : 𝒫𝒫 ⟶
𝒯𝒯 , mapping from the set of propositions 𝒫𝒫 into the set of truth values
𝒯𝒯. The semantics of propositional calculus are defined by the values of
the truth function, as formalized in Formula 1 through Formula 5.
𝜏𝜏 ¬ 𝑝𝑝 ∶= 𝜏𝜏 𝑝𝑝 ∧ 𝑞𝑞 ∶= if 𝜏𝜏 𝑝𝑝 = true
else false
true.
Formula 1
if (𝜏𝜏 𝑝𝑝 = 𝜏𝜏 𝑞𝑞 = true)
else true
false.
Formula 2
if (𝜏𝜏 𝑝𝑝 = 𝜏𝜏 𝑞𝑞 = false)
else false
true.
Formula 3
𝜏𝜏 𝑝𝑝 ∨ 𝑞𝑞 ∶= 𝜏𝜏 𝑝𝑝 ⇒ 𝑞𝑞 ∶= 𝜏𝜏 ¬ 𝑝𝑝 ∨ 𝑞𝑞
𝜏𝜏 𝑝𝑝 ≡ 𝑞𝑞 ∶= 𝜏𝜏 (𝑝𝑝 ⇒ 𝑞𝑞 ∧ 𝑞𝑞 ⇒ 𝑝𝑝)
Formula 4
Formula 5
George Boole (1847) realized that logic can be calculated using the
numbers 0 and 1 as truth values. His conclusion was that logic is
mathematical in nature:
I am then compelled to assert, that according to this
view of the nature of Philosophy, Logic forms no part of it.
On the principle of a true classification, we ought no longer
to associate Logic and Metaphysics, but Logic and
Mathematics. (p. 13)
In Boole’s mathematical definition of logic, the numbers 1 and 0
represents the truth-values and logical connectives are derived from
arithmetic operations: subtraction from 1 as negation and
multiplication as conjunction. All other operators can be derived from
these two operators through application of the laws of logical
equivalence. Thus, in Boolean logic (𝐵𝐵𝐵𝐵), the corresponding
propositional calculus is called Boolean algebra, stressing the
29
conceptual switch from metaphysics to mathematics. Its syntax is
defined in the same way as that of 𝐶𝐶𝐶𝐶, except that the Boolean truth
function, 𝜏𝜏 : 𝒫𝒫 ⟶ 𝒯𝒯 , maps from the set of propositions into the set of
Boolean truth values, 𝒯𝒯 ∶= 0,1 , that is, the set of the two numbers 0
and 1.
The Boolean truth function 𝜏𝜏 defines the semantics of Boolean
algebra. It is calculated using multiplication as conjunction and
subtraction from 1 as negation, as formalized in Formula 6 through
Formula 8. Implication and equivalence can be derived from negation
and disjunction in the same way as in classical propositional calculus.
𝜏𝜏 ¬ 𝑝𝑝 ∶= 1 − 𝜏𝜏 𝑝𝑝
Formula 6
𝜏𝜏 𝑝𝑝 ∧ 𝑞𝑞 ∶= 𝜏𝜏 𝑝𝑝 ∙ 𝜏𝜏 𝑞𝑞
Formula 7
𝜏𝜏 𝑝𝑝 ∨ 𝑞𝑞 ∶= ¬ ¬p ∧ ¬ q = 1 − 1 − 𝜏𝜏 𝑝𝑝 ∙ 1 − 𝜏𝜏 𝑞𝑞
2.1.2
Formula 8
Classification
Class logic, as defined by Glubrecht, Oberschelp, and Todt (1983),
is a logical system that supports statements applying a classification
operator. Classes of objects can be defined according to logical
propositional functions. According to Oberschelp (1994), a class,
𝐶𝐶 = 𝑖𝑖 ∈ 𝑈𝑈 | 𝛱𝛱(𝑖𝑖) , is defined as a collection of individuals, 𝑖𝑖, from a
universe of discourse, 𝑈𝑈, satisfying a propositional function, 𝛱𝛱, called
the classification predicate. The domain of the classification operator,
. | . ∶ ℙ ⟶ 𝑈𝑈 ∗ , is the class of propositional functions ℙ and its range is
the powerclass of the universe of discourse 𝑈𝑈 ∗ , which is the class of
possible subclasses of 𝑈𝑈. In other words, the class operator assigns
subsets of the universe of discourse to propositional functions. A
universe of discourse is the set of all possible individuals considered,
and an individual is a real object of reference. In the words of Bertrand
Russell (1919), a propositional function is “an expression containing one
or more undetermined constituents, such that, when values are
assigned to these constituents, the expression becomes a proposition” (p.
155).
30
In contrast, classification is the process of grouping individuals
who satisfy the same predicate into a class. A (Boolean) classification
corresponds to a membership function, 𝜇𝜇 ∶ 𝑈𝑈 ⟶ {0,1}, which indicates
with a Boolean truth-value whether an individual is a member of a
class, given the individual’s classification predicate. As shown by
Formula 9, the membership 𝜇𝜇 of individual 𝑖𝑖 in class 𝐶𝐶 = 𝑖𝑖 ∈ 𝑈𝑈 | 𝛱𝛱(𝑖𝑖) is defined by the truth-value 𝜏𝜏 of the classification predicate 𝛱𝛱(𝑖𝑖). In
Boolean logic, the truth-values are assumed to be certain. Therefore,
classification is sharp because the truth values are either exactly 0 or
exactly 1.
𝜇𝜇 𝑖𝑖 ≔ 𝜏𝜏 𝛱𝛱 𝑖𝑖
∈ 0,1 Formula 9
Usually, the classification predicate that defines classes refers to
attributes of individuals. For example, the class “tall people” is defined
by the predicate “tall,” which refers to the attribute “height.” An
attribute, 𝑋𝑋, is a function that characterizes individuals by mapping
from the universe of discourse 𝑈𝑈 to the set of possible characteristics 𝜒𝜒
(Formula 10).
𝑋𝑋 ∶ 𝑈𝑈 ⟶ 𝜒𝜒
Formula 10
𝜇𝜇 𝑖𝑖 ∶= 𝜏𝜏 𝑋𝑋 𝑖𝑖 = 𝑐𝑐
Formula 11
There are different types of values encoding characteristics.
Categorical attributes have a discrete range of symbolic values.
Numerical attributes have a range of numbers, which can be natural or
real. Boolean attributes have Boolean truth-values {0,1} as a range.
Ordinal attributes have a range of categories that can be ordered.
On one hand, the distinction between univariate and multivariate
classification, the variety, depends on the number of attributes
considered for the classification predicate. The dimensionality of the
classification, on the other hand, depends on the number of dimensions,
or linearly independent attributes, of the classification predicate
domain.
In a univariate classification (𝑈𝑈𝑈𝑈), the classification predicate
𝛱𝛱 refers to one attribute, 𝑋𝑋, which is true for an individual, 𝑖𝑖, if the
feature 𝑋𝑋(𝑖𝑖) equals a certain characteristic, 𝑐𝑐 ∈ 𝜒𝜒.
In a multivariate classification (𝑀𝑀𝑀𝑀𝑀𝑀), the classification predicate
refers to multiple element attributes. The classification predicate is true
31
for an individual, 𝑖𝑖, if an aggregation, 𝑎𝑎, of several characteristic
constraints has a given value, 𝑐𝑐 ∈ 𝜒𝜒.
𝜇𝜇 𝑖𝑖 ∶= 𝜏𝜏 𝑎𝑎 𝑋𝑋 𝑖𝑖 , ⋯ , 𝑋𝑋 𝑖𝑖
= 𝑐𝑐
Formula 12
A multidimensional classification (𝑀𝑀𝑀𝑀𝑀𝑀) is a special case of a
multivariate classification that refers to 𝑛𝑛-tuples of attributes, such that
the resulting class is functionally dependent on the combination of all n
attributes.
𝜇𝜇 𝑖𝑖 ∶= 𝜏𝜏 𝐶𝐶
𝑋𝑋 𝑖𝑖
⋮
= ⋮
𝐶𝐶
𝑋𝑋 (𝑖𝑖)
Formula 13
This distinction between multivariate and multidimensional
classification is necessary for the construction of classification functions.
Multivariate classifications can be derived as functional aggregates of
one-dimensional membership functions, in which the influence of one
attribute to the resulting aggregate does not depend on the other
attributes. In contrast, in multidimensional classification, the
combination of all attributes determines the membership value, and
thus, one attribute has different influences on the membership degree
for different combinations with other attribute values. Therefore,
multidimensional classifications need multidimensional membership
functions that are defined on 𝑛𝑛-tuples of possible characteristics.
32
2.2
Fuzziness
“There are many misconceptions about fuzzy logic. Fuzzy logic is not
fuzzy. Basically, fuzzy logic is a precise logic of imprecision and
approximate reasoning.” (Zadeh, 2008, p. 2751)
Fuzziness, or vagueness (Sorensen, 2008), is an uncertainty
regarding concept boundaries. In contrast to ambiguous terms, which
have several meanings, vague terms have one meaning, but the extent
of it is not sharply distinguishable. For example, the word tall can be
ambiguous, because a tall cat is usually smaller than a small horse.
Nevertheless, the disambiguated predicate “tall for a cat” is vague,
because its linguistic concept does not imply a sharp border between tall
and small cats.
Our brains seem to love boundaries. Perhaps, making sharp
distinctions quickly was a key cognitive ability in evolution. Our brains
are so good at recognizing limits, that they construct limits where there
are none. This is what many optical illusions are based on: for example,
Kaniza’s (1976) Illusory Square (Figure 3).
Figure 3: There is no square. Adapted from “Subjective Contours” by G. Kaniza, 1976,
Copyright 1976 by Scientific American, Inc.
An ancient symbol of sharp distinction between classes is the yin
and yang symbol (Figure 4). It symbolizes a dualistic worldview—the
cosmos divided into light and dark, day and night, and so on.
33
Figure 4: Black or white: conventional yin and yang symbol with a sharp distinction
between opposites, representing metaphysical dualism.
Adapted from http://www.texample.net/tikz/examples/yin-and-yang/ (accessed 02.2012)
with permission (creative commons license CC BY 2.5).
Figure 5: Shades of grey: fuzzy yin and yang symbol with a gradation between
opposites, representing metaphysical monism.
34
Nevertheless, in reality, the transition between light and dark is
gradual during the 24 hours of a day. This idea of gradation of our
perceptions can be visualized by a fuzzy yin and yang symbol (Figure 5).
Sorensen (2008) explains that many-valued logics have been proposed to
solve the philosophical implications of vagueness. One many-valued
approach to logic is fuzzy logic, which allows infinite truth-values in the
interval between 0 and 1.
In the next section, introducing membership functions, fuzzy sets,
and fuzzy propositions are discussed; these are the bases for fuzzy logic,
which in fact, is a precise logic for fuzziness. Additionally, it is shown
how fuzzy classifications are derived from fuzzy propositional functions.
2.2.1
Fuzzy Logic
Lotfi Zadeh (2008) said, “Fuzzy logic is not fuzzy” (p. 2751).
Indeed, it is a precise mathematical concept for reasoning about fuzzy
(vague) concepts. If the domain of those concepts is ordinal, membership
can be distinguished by its degree. In classical set theory, an individual,
𝑖𝑖, of a universe of discourse, 𝑈𝑈, is either completely a member of a set or
not at all. As previously explained, according to Boolean logic, the
membership function 𝜇𝜇 : 𝑈𝑈 ⟶ {0,1}, for a crisp set 𝑆𝑆, maps from
individuals to sharp truth-values. As illustrated in Figure 6, a sharp set
(the big dark circle) has a clear boundary, and individuals (the small
bright circles) are either a member of it or not. However, one individual
is not entirely covered by the big dark circle, but is also not outside of it.
In contrast, a set is called fuzzy by Zadeh (1965) if individuals can have
a gradual degree of membership to it. In a fuzzy set, as shown by Figure
7, the limits of the set are blurred. The degree of membership of the
elements in the set is gradual, illustrated by the fuzzy gray edge of the
dark circle. The membership function, 𝜇𝜇 : 𝑈𝑈 → [0,1], for a fuzzy set, 𝐹𝐹,
indicates the degree to which individual 𝑖𝑖 is a member of 𝐹𝐹 in the
interval between 0 and 1. In Figure 6, the degree of membership of the
small circles 𝑖𝑖 is defined by a normalization 𝑛𝑛 of their distance 𝑑𝑑 from
the center 𝑐𝑐 of the big dark circle 𝑏𝑏, 𝜇𝜇 𝑖𝑖 = 𝑛𝑛 𝑑𝑑 𝑖𝑖, 𝑐𝑐 .In the same way
as in classical set theory, set operators can construct complements of
sets and combine two sets by union and intersection. Those operators
are defined by the fuzzy membership function. In the original proposal
of Zadeh (1965), the set operators are defined by subtraction from 1,
minimum and maximum. The complement, 𝐹𝐹, of a fuzzy set, 𝐹𝐹, is
derived by subtracting its membership function from 1; the union of two
sets, 𝐹𝐹 ∪ 𝐺𝐺, is derived from the maximum of the membership degrees;
and the intersection of two sets, 𝐹𝐹 ∩ 𝐺𝐺, is derived from the minimum of
the membership degrees.
35
Figure 6: A visualization of a classical set with sharp boundaries.
Figure 7: A visualization of a fuzzy set.
36
Accordingly, fuzzy subsets and fuzzy power sets can be
constructed. Consider the two fuzzy sets 𝐴𝐴 and 𝐵𝐵 on the universe of
discourse 𝑈𝑈. In general, 𝐴𝐴 is a fuzzy subset of 𝐵𝐵 if the membership
degrees of all its elements are smaller or equal to the membership
degrees of elements in 𝐵𝐵 (Formula 14). Thus, a fuzzy power set, 𝐵𝐵 ∗ , of a
(potentially fuzzy) set B is the class of all its fuzzy subsets (Formula 15).
Formula 14
𝐴𝐴 ⊆ 𝐵𝐵 ∶= ∀𝑥𝑥 ∈ 𝑈𝑈: 𝜇𝜇 𝑥𝑥 ≤ 𝜇𝜇 𝑥𝑥
𝐵𝐵 ∗ ≔ 𝐴𝐴 ⊆ 𝑈𝑈 𝐴𝐴 ⊆ 𝐵𝐵
Formula 15
With the tool of fuzzy set theory in hand, the sorites paradox cited
in the introduction (Chapter 1) can be tackled in a much more satisfying
manner. A heap of wheat grains can be defined as a fuzzy subset,
𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻 ⊆ ℕ, of natural numbers ℕ of wheat grains. A heap is defined in
the English language as “a great number or large quantity” (merriamwebster.com, 2012b). For instance, one could agree that 1,000 grains of
wheat is a large quantity, and between 1 and 1,000, the “heapness” of a
grain collection grows logarithmically. Thus, the membership function
of the number of grains 𝑛𝑛 ∈ ℕ in the fuzzy set 𝐻𝐻𝐻𝐻𝐻𝐻𝐻𝐻 can be defined
according to Formula 16. The resulting membership function is plotted
in Figure 8.
𝜇𝜇Heap 𝑛𝑛 ≔
0 if 𝑛𝑛 = 0
1 if 𝑛𝑛 > 1000
0.1448 ln 𝑛𝑛 else.
Formula 16
Figure 8: Fuzzy set theory applied to the sorites paradox.
37
Based on the concept of fuzzy sets, Zadeh (1975a) derived fuzzy
propositions (𝐹𝐹𝐹𝐹) for approximate reasoning: A fuzzy proposition has
the form “𝑥𝑥 𝑖𝑖𝑖𝑖 𝐿𝐿,” where 𝑥𝑥 is an individual of a universe of discourse 𝑈𝑈
and 𝐿𝐿 is a linguistic term, defined as a fuzzy set on 𝑈𝑈. As stated by
Formula 17, the truth-value 𝜏𝜏 of a fuzzy proposition is defined by the
degree of membership 𝜇𝜇 of 𝑥𝑥 in the linguistic term 𝐿𝐿.
𝜏𝜏 𝑥𝑥 𝑖𝑖𝑖𝑖 𝐿𝐿 ∶= 𝜇𝜇 (𝑥𝑥)
Formula 17
If 𝐴𝐴 is an attribute of 𝑥𝑥, a fuzzy proposition can also refer to the
corresponding attribute value, such as 𝑥𝑥 𝑖𝑖𝑖𝑖 𝐿𝐿 ∶= 𝐴𝐴 𝑥𝑥 𝑖𝑖𝑖𝑖 𝐿𝐿 . The fuzzy set 𝐿𝐿
on 𝑈𝑈 is equivalent to the fuzzy set 𝐿𝐿 on the domain of the attribute, or
dom(𝐴𝐴). In fact, the set can be defined on arbitrarily deep-nested
attribute hierarchies concerning the individual. As an example, let us
look at the fuzzy proposition, “Mary is blond.” In this sentence, the
linguistic term “blond” is a fuzzy set on the set of people, which is
equivalent to a fuzzy set blond on the color of people’s hair (Formula
18).
𝜏𝜏 "Mary is blond" = 𝜇𝜇 Mary
≡ 𝜇𝜇 color hair Mary
Formula 18
Fuzzy propositions (𝐹𝐹𝐹𝐹) can be combined to construct fuzzy
formulae using the usual logic operators not (¬), and (∧), and or (∨), for
which the semantics are defined by the fuzzy truth function 𝜏𝜏 : ℱ ⟶
[0,1], mapping from the class of fuzzy propositions ℱ into the set of
Zadehan truth values in the interval between 1 and 0. Let “𝑥𝑥 is 𝑃𝑃” and
“𝑥𝑥 is 𝑄𝑄” be two fuzzy propositions on the same individual. Then their
combination to fuzzy formulae is defined as follows (Formula 19
through Formula 21): negation by the inverse of the corresponding fuzzy
set, conjunction by intersection of the corresponding fuzzy sets, and
disjunction by union of the corresponding fuzzy sets.
𝜏𝜏 ¬ 𝑥𝑥 is 𝑃𝑃
∶= 𝜇𝜇 𝑥𝑥 ; 𝜏𝜏 𝑥𝑥 is 𝑃𝑃 ∧ 𝑥𝑥 is 𝑄𝑄 ∶= 𝜇𝜇∩ (𝑥𝑥);
𝜏𝜏 𝑥𝑥 is 𝑃𝑃 ∨ 𝑥𝑥 is 𝑄𝑄 ∶= 𝜇𝜇∪ (𝑥𝑥) 38
Formula 19
Formula 20
Formula 21
Zadeh’s fuzzy propositions are derived from statements of the form
“𝑋𝑋 is 𝑌𝑌.” They are based on the representation operator 𝑖𝑖𝑖𝑖 ∶ 𝑈𝑈×𝑈𝑈 ∗ ⟶ ℱ
mapping from the universe 𝑈𝑈 of discourse and its fuzzy powerset 𝑈𝑈 ∗ to
the class of fuzzy propositions ℱ. Consequently, fuzzy propositions in
the sense of Zadeh are limited to statements about degrees of
membership in a fuzzy set.
Generally, logic with fuzzy propositions—or more precisely, a
propositional logic with Zadehan truth values in the interval between 0
and 1, a “Zadehan Logic” (𝑍𝑍𝑍𝑍)—can be viewed as a generalization of
Boole’s mathematical analysis of logic to a gradual concept of truth. In
that sense, 𝑍𝑍𝑍𝑍 is a simple generalization of Boolean logic (𝐵𝐵𝐵𝐵), in which
the truth value of any proposition is not only represented by numbers,
but also can be anywhere in the interval between 0 and 1.
According to the Stanford Encyclopedia of Philosophy (Hajek,
2006), fuzzy logic, in the narrow sense, is a “symbolic logic with a
comparative notion of truth developed fully in the spirit of classical
logic” (“Fuzzy Logic,” paragraph 3). If 𝑍𝑍𝑍𝑍 is viewed as a generalization of
𝐵𝐵𝐵𝐵, fuzzy propositions of the form “𝑋𝑋 𝑖𝑖𝑖𝑖 𝑌𝑌” are a special case, and
propositions and propositional functions of any form can have gradual
values of truth. Accordingly, 𝑍𝑍𝑍𝑍 is defined by the truth function
𝜏𝜏 : 𝒫𝒫 ⟶ 𝒯𝒯 mapping from the class of propositions 𝒫𝒫 to the set of
Zadehan truth-values 𝒯𝒯 ≔ [0,1]. Consequently, fuzzy set membership
is a special case of fuzzy proposition, and the degree of membership of
individual 𝑥𝑥 in another individual 𝑦𝑦 can be defined as the value of truth
of the fuzzy proposition 𝑥𝑥 ∈ 𝑦𝑦 (Formula 22).
𝜇𝜇 𝑥𝑥 ∶= 𝜏𝜏 (𝑥𝑥 ∈ 𝑦𝑦) Formula 22
𝜏𝜏 ¬ 𝑝𝑝 ∶= 1 − 𝜏𝜏 𝑝𝑝
Formula 23
𝜏𝜏 𝑝𝑝 ∧ 𝑞𝑞 ∶= 𝜏𝜏 𝑝𝑝 ∙ 𝜏𝜏 𝑞𝑞
Formula 24
The Zadehan truth function 𝜏𝜏 defines the semantics of 𝑍𝑍𝑍𝑍. As in
Boolean algebra, its operators can be defined by subtraction from 1 as
negation, and multiplication as conjunction, as formalized in Formula
23 and Formula 24. Disjunction, implication, and equivalence can be
derived from negation and conjunction in the same way as in Boolean
logic.
39
In that light, any proposition with an uncertain truth-value
smaller than 1 or greater than 0 is a fuzzy proposition. Additionally,
every function with the range [0,1] can be thought of as a truth function
for a propositional function. For example, statistical likelihood 𝐿𝐿(𝑦𝑦|𝑥𝑥)
can be seen as a truth function for the propositional function, “y is likely
if x,” as a function of x. This idea is the basis for IFC proposed in the
next section. The usefulness of this generalization is shown in the
chapter on applications, in which fuzzy propositions such as “customers
with characteristic X are likely to buy product Y” are assigned truthvalues that are computed using quantitative prediction modeling.
2.2.2
Fuzzy Classification
A fuzzy class, 𝐶𝐶 : = ∼ 𝑖𝑖 ∈ 𝑈𝑈 | 𝛱𝛱(𝑖𝑖) , is defined as a fuzzy set 𝐶𝐶 of
individuals 𝑖𝑖, whose membership degree is defined by the Zadehan
truth-value of the proposition 𝛱𝛱 𝑖𝑖 . The classification predicate, 𝛱𝛱, is a
propositional function interpreted in 𝑍𝑍𝑍𝑍. The domain of the fuzzy class
operator, ∼ . | . ∶ ℙ ⟶ 𝑈𝑈 ∗ , is the class of propositional functions, ℙ, and
the range is the fuzzy power set, 𝑈𝑈 ∗ (the set of fuzzy subsets) of the
universe of discourse, 𝑈𝑈. In other words, the fuzzy class operator assigns
fuzzy subsets of the universe of discourse to propositional functions.
Fuzzy classification is the process of assigning individuals a
membership degree to a fuzzy set, based on their degrees of truth of the
classification predicate. It has been discussed, for example, by
Zimmermann (1997), Del Amo et al. (1999), and Meier et al. (2008). A
fuzzy classification is achieved by a membership function, 𝜇𝜇 : 𝑈𝑈 ⟶ [0,1],
that indicates the degree to which an individual is a member of a fuzzy
class, 𝐶𝐶, given the corresponding fuzzy propositional function, 𝛱𝛱. This
membership degree is defined by the Zadehan truth-value of the
corresponding proposition, 𝛱𝛱 𝑖𝑖 , as formalized in Formula 25.
𝜇𝜇 𝑖𝑖 ≔ 𝜏𝜏 𝛱𝛱 𝑖𝑖 Formula 25
In the same way as in crisp classification, the fuzzy classification
predicate refers to attributes of individuals. Additionally, Zadehan logic
introduces two new types of characteristics. Zadehan attributes have a
range of truth values represented by 𝒯𝒯 ≔ [0,1]. Linguistic attributes
have a range of linguistic terms (fuzzy sets) together with the Zadehan
truth-value of membership in those terms (Zadeh, 1975b).
In a univariate fuzzy classification (𝑈𝑈𝑈𝑈), the fuzzy classification
predicate 𝛱𝛱 refers to one attribute, 𝑋𝑋, and it corresponds to the
membership degree of the attribute characteristic 𝑋𝑋(𝑖𝑖) in a given fuzzy
40
restriction (Zadeh, 1975a), 𝑅𝑅 ∈ 𝜒𝜒 ∗ , which is a fuzzy subset of possible
characteristics 𝜒𝜒 (Formula 26).
𝜇𝜇 𝑖𝑖 ≔ 𝜏𝜏 𝑋𝑋 𝑖𝑖 𝑖𝑖𝑖𝑖 𝑅𝑅 Formula 26
𝜇𝜇 𝑖𝑖 ≔ 𝑎𝑎 𝜏𝜏 𝑋𝑋 i 𝑖𝑖𝑖𝑖 𝑅𝑅 , … , 𝑋𝑋 i 𝑖𝑖𝑖𝑖 𝑅𝑅 Formula 27
In a multivariate fuzzy classification (𝑀𝑀𝑀𝑀𝑀𝑀), 𝛱𝛱 refers to multiple
attributes. The truth function of the classification predicate for an
individual, 𝑖𝑖, equals to an aggregation, 𝑎𝑎, of several fuzzy restrictions of
multiple attribute characteristics, 𝑋𝑋 𝑖𝑖 , 𝑗𝑗 = 1 … 𝑛𝑛 (Formula 27).
In a multidimensional fuzzy classification (𝑀𝑀𝑀𝑀𝑀𝑀), 𝛱𝛱 refers to 𝑛𝑛tuples of functionally independent attributes. The membership degree
of individuals in a multidimensional class is based on an 𝑛𝑛-dimensional
fuzzy restriction, 𝑅𝑅 (Formula 28), which is a multidimensional fuzzy
set on the Cartesian product of the attribute ranges with a
multidimensional membership function of 𝜇𝜇 : 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑋𝑋 × … ×
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑋𝑋 ⟶ [0,1].
𝜇𝜇 𝑖𝑖 ≔ 𝜏𝜏 𝑋𝑋 𝑖𝑖
⋮
𝑋𝑋 𝑖𝑖
is 𝑅𝑅 Formula 28
41
42
2.3
Induction
Given a set of certainly true statements, deduction works fine. The
problem is that the only certainty philosophy can offer is Descartes’s “I
think therefore I am” proposition; however, postmodern philosophers
are not so sure about the I anymore (Precht, 2007, p. 62 ff). Therefore,
one should be given a tool to reason under uncertainty, and this tool is
induction. In this chapter, inductive logic is analyzed, the application of
induction to fuzzy classification is discussed, and a methodology for
membership function induction using normalized ratios and differences
of empirical conditional probabilities and likelihoods is proposed.
2.3.1
Inductive Logic
Traditionally, induction is defined as drawing general conclusions
from particular observations. Contemporary philosophy has shifted to a
different view because, not only are there inductions that lead to
particular conclusions, but also there are deductions that lead to
general conclusions. According to Vickers (2009) in the Stanford
Encyclopedia of Philosophy (SEP), it is agreed that induction is a form
of inference that is “contingent” and “ampliative” (“The contemporary
notion of induction”, paragraph 3), in contrast to deductive inference,
which is necessary and explicative. Induction is contingent, because
inductively inferred propositions are not necessarily true in all cases.
And it is ampliative because, in Vickers words, “induction can amplify
and generalize our experience, broaden and deepen our empirical
knowledge” (“The contemporary notion of induction”, paragraph 3). In
another essay in the SEP, inductive logic is defined as “a system of
evidential support that extends deductive logic to less-than-certain
inferences” (Hawthorne, 2008, “Inductive Logic,” paragraph 1).
Hawthorne admits that there is a degree of fuzziness in induction: In an
inductive inference, “the premises should provide some degree of
support for the conclusion” (“Inductive Logic,” para. 1). The degree of
support for an inductive inference can thus be viewed as a fuzzy
restriction of possible inferences, in the sense of Zadeh (1975a). Vickers
(2009) explains that the problem of induction is two-fold: The epistemic
problem is to define a method to distinguish appropriate from
inappropriate inductive inference. The metaphysical problem is to
explain in what substance the difference between reliable and
unreliable induction actually exists.
Epistemologically, the question of induction is to find a suitable
method to infer propositions under uncertainty. State of the art
methods rely on empirical probabilities or likelihoods. There are many
interpretations of probability (Hájek, 2009). For the context of this
43
thesis, one may agree that a mathematical probability,
𝑃𝑃 𝐴𝐴 , numerically represents how probable it is that a specific
proposition 𝐴𝐴 is true: 𝑃𝑃 𝐴𝐴 ≡ 𝜏𝜏 "𝐴𝐴 is probable" ; and that the
disjunction of all possible propositions, the probability space Ω, is
certain, i.e., 𝑃𝑃 Ω = 1.
In practice, probabilities can be estimated by relative frequencies,
or sampled empirical probabilities 𝑝𝑝 in a sample of 𝑛𝑛 observations,
defined by the ratio between the number of observations, 𝑖𝑖, in which the
proposition 𝐴𝐴 is true, and the total number of observations (Formula
29).
𝑃𝑃(𝐴𝐴) ≈ 𝑝𝑝 𝐴𝐴 ∶= 𝜏𝜏(𝐴𝐴 )
𝑛𝑛
Formula 29
A conditional probability (Weisstein, 2010a) is the probability for
an outcome 𝑥𝑥, given that 𝑦𝑦 is the case, as formalized in Formula 30.
𝑃𝑃(𝑥𝑥 | 𝑦𝑦) = 𝑃𝑃(𝑥𝑥 ∧ 𝑦𝑦)
𝑃𝑃(𝑦𝑦)
Formula 30
Empirical sampled conditional probabilities can be applied to
compute likelihoods. According to James Joyce, “in an unfortunate, but
now unavoidable, choice of terminology, statisticians refer to the inverse
probability PH(E) as the ‘likelihood’ of H on E” (Joyce, 2003,
“Conditional Probabilities and Bayes' Theorem,” paragraph 5). The
likelihood of the hypothesis 𝐻𝐻 is an estimate of how probable the
evidence or known data 𝐸𝐸 is, given that the hypothesis is true. Such a
probability is called a “posterior probability” (Hawthorne, 2008,
“inductive Logic,” paragraph 5), that is, a probability after
measurement, shown by Formula 31.
𝐿𝐿 𝐻𝐻 𝐸𝐸 ∶= 𝑝𝑝 𝐸𝐸 | 𝐻𝐻 Formula 31
In the sense of Hawthorne (2008), the general law of likelihood
states that, for a pair of incompatible hypotheses 𝐻𝐻 and 𝐻𝐻 , the
evidence 𝐸𝐸 supports 𝐻𝐻 over 𝐻𝐻 , if and only if 𝑝𝑝 𝐸𝐸 | 𝐻𝐻 > 𝑝𝑝 𝐸𝐸 | 𝐻𝐻 . The
likelihood ratio (𝐿𝐿𝐿𝐿) measures the strength of evidence for 𝐻𝐻 over 𝐻𝐻
(Formula 32). Thus, the “likelihoodist” (sic; Hawthorne, 2008,
“Likelihood Ratios, Likelihoodism, and the Law of Likelihood,”
paragraph 5) solution to the epistemological problem of induction is the
likelihood ratio as measure of support for inductive inference.
44
𝐿𝐿𝐿𝐿 𝐻𝐻 > 𝐻𝐻 | 𝐸𝐸 ∶=
𝐿𝐿 𝐻𝐻 𝐸𝐸
𝐿𝐿 𝐻𝐻 𝐸𝐸
Formula 32
According to Hawthorne, the prior probability of a hypothesis,
𝑝𝑝 𝐻𝐻 , that is, an estimated probability prior to measurement of
evidence 𝐸𝐸, plays an important role for inductive reasoning.
Accordingly, Bayes’ theorem can be interpreted and rewritten using
measured posterior likelihood and prior probability in order to apply it
to the evaluation of scientific hypotheses. According to Hawthorne
(2008), the posterior probability of hypothesis 𝐻𝐻 conditional to evidence
𝐸𝐸 is equal to the product of the posterior likelihood of 𝐻𝐻 given 𝐸𝐸 and the
prior probability of 𝐻𝐻, divided by the (measured) probability of 𝐸𝐸
(Formula 33).
𝑝𝑝 𝐻𝐻 𝐸𝐸 =
𝐿𝐿(𝐻𝐻|𝐸𝐸) ∙ 𝑝𝑝 𝐻𝐻
𝑝𝑝(𝐸𝐸)
Formula 33
What if there is fuzziness in the data, in the features of
observations, or in the theories? How is likelihood measured when the
hypothesis or the evidence is fuzzy? If this fuzziness is ordinal, that is, if
the extent of membership in the fuzzy terms can be ordered, a
membership function can be defined, and an empirical probability of
fuzzy events can be calculated. Analogous to Dubois and Prade (1980), a
fuzzy event 𝐴𝐴 in a universe of discourse 𝑈𝑈 is a fuzzy set on 𝑈𝑈 with a
membership function 𝜇𝜇 : 𝑈𝑈 ⟶ [0,1]. For categorical elements of 𝑈𝑈, the
estimated probability after n observations is defined as the average
degree of membership of observations 𝑖𝑖 in 𝐴𝐴, as formalized in Formula
34.
𝑃𝑃 𝐴𝐴 ≈ 𝑝𝑝 𝐴𝐴 =
𝜇𝜇 (𝑖𝑖)
𝑛𝑛
Formula 34
By application of Formula 34 to Formula 31, the likelihood of
ordinal fuzzy hypothesis 𝐻𝐻, given ordinal fuzzy evidence 𝐸𝐸, can be
defined as a conditional probability of fuzzy events, as shown in
Formula 35.
45
𝐿𝐿 𝐻𝐻 𝐸𝐸) = 𝑝𝑝(𝐸𝐸|𝐻𝐻) =
𝜇𝜇∩ (𝑖𝑖)
𝜇𝜇 (𝑖𝑖)
Formula 35
The question of the metaphysical problem of induction is: what is
the substance of induction? In what kind of material does the difference
between reliable and unreliable inductive inference exist? The
importance of this question cannot be underestimated, since reliable
induction enables prediction. A possible answer could be that the
substance of an induction is the amount of information contained in the
inference. This answer presupposes that information is a realist
category, as suggested by Chmielecki (1998). According to Shannon’s
information theory (Shannon, 1948), the information contained in
evidence 𝑥𝑥 about hypothesis 𝑦𝑦 is equal to the difference between the
uncertainty (entropy), 𝐻𝐻(𝑦𝑦), about the hypothesis 𝑦𝑦 and the resulting
uncertainty, 𝐻𝐻 (𝑦𝑦), after observation of the evidence 𝑥𝑥, 𝐼𝐼 𝑥𝑥, 𝑦𝑦 =
𝐻𝐻 𝑦𝑦 − 𝐻𝐻 𝑦𝑦 = Σ Σ 𝑝𝑝(𝑥𝑥 ∧ 𝑦𝑦)log 𝑝𝑝 𝑥𝑥 ∧ 𝑦𝑦 / 𝑝𝑝 𝑥𝑥 𝑝𝑝 𝑦𝑦 . Shannon’s quantity
of information is defined in terms of joint probabilities. However, by
application of Shannon’s theory, the metaphysical problem of induction
is transferred to a metaphysical problem of probabilities because,
according to Shannon, the basic substance of information is the
probability of two signals occurring simultaneously compared to the
probability of occurring individually. (One could link this solution to the
concept of quantum physical particle probability waves [Greene, 2011],
but this would go beyond the scope of this thesis and would be highly
speculative; therefore, this link is not explored here. Suffice it to state
that probability apparently is a fundamental construct of matter and
waves as well as of information and induction.)
2.3.2
Inductive Classification
Inductive classification is the process of assigning individuals to a
set based on a classification predicate derived by an inductive inference.
Inductive classification can be automated as a form of supervised
machine learning (Witten & Frank, 2005): a class of processes
(algorithms or heuristics) that learn from examples to decide whether
an individual, 𝑖𝑖, belongs to a given class, 𝑦𝑦, based on its attributes.
Generally, supervised machine learning processes induce a model from
a dataset, which generalizes associations in the data in order to provide
support for inductive inference. This model can be used for predicting
the class membership of new data elements. Induced classification
models, called classifiers, are first trained using a training set with
known class membership. Then, they are applied to a test or prediction
set in order to derive class membership predictions. Examples of
46
classification learning algorithms that result in classifications are
decision trees, classification rules, and association rules. In those cases,
the model consists of logical formulae of attribute values, which predict
a crisp class value.
Data are signs (signals) that represent knowledge such as
numbers, characters, or bits. The basis for automated data analysis is a
systematic collection of data on individuals. The most frequently used
data structure for analytics is the matrix, in which every individual, 𝑖𝑖 (a
customer, a transaction, a website, etc.), is represented by a row, and
every attribute, 𝑋𝑋 , is represented by a column. Every characteristic,
𝑋𝑋 (𝑖𝑖), of individual 𝑖𝑖 for attribute 𝑋𝑋 is represented by one scalar value
within the matrix.
A training dataset 𝑑𝑑 is an 𝑚𝑚× 𝑛𝑛 + 1 matrix with 𝑚𝑚 rows, 𝑛𝑛
columns for 𝑋𝑋 , … , 𝑋𝑋 and a column 𝑌𝑌 indicating the actual class
membership. The columns 𝑋𝑋 , 1 ≤ 𝑘𝑘 ≤ 𝑛𝑛 are called analytic variables,
and 𝑌𝑌 is called the target variable, which indicates membership in a
target class 𝑦𝑦. In case of a binary classification, for each row index 𝑖𝑖, the
label 𝑌𝑌(𝑖𝑖) is equal to 1 if and only if individual 𝑖𝑖 is in class 𝑦𝑦 (Formula
36).
𝑌𝑌(𝑖𝑖) ∶=
1
0
if 𝑖𝑖 ∈ 𝑦𝑦
else.
Formula 36
A machine learning process for inductive sharp classification
generates a model 𝑀𝑀 (𝑖𝑖), mapping from the Cartesian product of the
analytic variable ranges into the set {0,1}, indicating inductive support
for the hypothesis that 𝑖𝑖 ∈ 𝑦𝑦. As discussed in the section on induction,
the model should provide support for inductive inferences about an
individual’s class membership: Given 𝑀𝑀 (𝑖𝑖) = 1, the likelihood of 𝑖𝑖 ∈ 𝑦𝑦
should be greater than the likelihood of 𝑖𝑖 ∉ 𝑦𝑦.
The inductive model 𝑀𝑀 can be applied for prediction to a new
dataset with unknown class indicator, which is either a test set for
performance evaluation or a prediction set, where the model is applied
to forecast class membership of new data. The test set or prediction set
𝑑𝑑′ has the same structure as the training set 𝑑𝑑, except that the class
membership is unknown, and thus, the target variable 𝑌𝑌 is empty. The
classifier 𝑀𝑀 , derived from the training set, can be used for predicting
the class memberships of representations of individuals 𝑖𝑖 ∈ 𝑑𝑑. The
model output prediction 𝑀𝑀 (𝑖𝑖) yields an inductive classification defined
by 𝑖𝑖 𝑀𝑀 𝑖𝑖 = 1}.
In order to evaluate the quality of prediction of a crisp classifier
model, several measures can be computed. In this section, likelihood
47
ratio and Pearson correlation are mentioned. The greater the ratio
between likelihood for target class membership, given a positive
prediction, and the likelihood for target class membership, given a
negative prediction, the better the inductive support of the model. Thus,
the predictive model can be evaluated by the likelihood ratio of target
class membership given the model output (Formula 37). 𝐿𝐿𝐿𝐿 𝑌𝑌 𝑖𝑖 = 1 | 𝑀𝑀 𝑖𝑖 = 1 ∶=
𝑝𝑝 𝑀𝑀 𝑖𝑖 = 1 | 𝑌𝑌 𝑖𝑖 = 1
𝑝𝑝 𝑀𝑀 𝑖𝑖 = 1 | 𝑌𝑌 𝑖𝑖 = 0
Formula 37
Working with binary or Boolean target indicators and model
indicators allows the evaluation of predictive quality by a measure of
correlation of the two variables 𝑀𝑀 and 𝑌𝑌 (Formula 38). The correlation
between two numerical variables can be measured by the Pearson
correlation coefficient as the ratio between the covariance of the two
variables and the square root of the product of individual variances
(Weisstein, 2010b).
corr 𝑀𝑀 , 𝑌𝑌 =
𝐸𝐸
𝑀𝑀 − avg 𝑀𝑀
𝑌𝑌 − avg 𝑌𝑌
stddev 𝑀𝑀 ∙ stddev(𝑌𝑌)
Formula 38
The advantage of the correlation coefficient is its availability in
database systems. Every standard SQL (structured query language)
database has an implementation of correlation as an aggregate
function. Thus, using the correlation coefficient, evaluating the
predictive performance of a model in a database is fast and simple.
However, it is important to stress that evaluating predictions with a
measure of correlation is only meaningful if the target variable as well
as the predictive variable are Boolean, Zadehan, or numeric in nature.
It will not work for ordinal or categorical target classes, except if they
are transformed into a set of Boolean variables.
For example, in database marketing, the process of target group
selection uses classifiers to select customers who are likely to buy a
certain product. In order to do this, a classifier model can be computed
in the following way: Given set of customers 𝐶𝐶, we know whether they
have bought product 𝐴𝐴 or not. Let 𝑐𝑐 be an individual customer, and 𝐶𝐶
be the set of customers who bought product 𝐴𝐴. Then, the value 𝑌𝑌(𝑐𝑐) of
target variable 𝑌𝑌 for customer 𝑐𝑐 is defined in Formula 39.
48
𝑌𝑌(𝑐𝑐) =
1
0
if 𝑐𝑐 ∈ 𝐶𝐶
else.
Formula 39
The analytic variables for customers are selected from every known
customer attribute, such as age, location, transaction behavior, recency,
frequency, and monetary value of purchase. The aim of the classifier
induction process is to learn a model, 𝑀𝑀 , that provides a degree of
support for the inductive inference that a customer is interested in the
target product 𝐴𝐴. This prediction, 𝑀𝑀 (c) ∈ {0,1}, should provide a better
likelihood to identify potential buyers of product 𝐴𝐴, and it should
optimally correlate with the actual product usage of existing and future
customers.
49
50
2.4
Inductive Fuzzy Classification
The understanding of IFC in the proposed research approach is an
inductive gradation of the degree of membership of individuals in
classes In many interpretations, the induction step consists of learning
fuzzy rules (e.g., Dianhui, Dillon, & Chang, 2001; Hu, Chen, & Tzeng,
2003; Roubos, Setnes, & Abonyi, 2003; Wang & Mendel, 1992). In this
thesis, IFC is understood more generally as inducing membership
functions to fuzzy classes and assigning individuals to those classes. In
general, a membership function can be any function mapping into the
interval between 1 and 0. Consequently, IFC is defined as the process of
assigning individuals to fuzzy sets for which membership functions are
generated from data so that the membership degrees are based on an
inductive inference.
An inductive fuzzy class, 𝑦𝑦′, is defined by a predictive scoring
model, 𝑀𝑀 : 𝑈𝑈 → [0,1], for membership in a class, 𝑦𝑦. This model
represents an inductive membership function for 𝑦𝑦′, which maps from
the universe of discourse 𝑈𝑈 into the interval between 0 and 1 (Formula
40).
𝜇𝜇 : 𝑈𝑈 → [0,1] ∶= 𝑀𝑀 Formula 40
𝜏𝜏 “𝑖𝑖 𝑖𝑖𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 𝑎𝑎 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑜𝑜𝑜𝑜 𝑦𝑦” ≔ 𝑀𝑀 (𝑖𝑖) Formula 41
Consider the following fuzzy classification predicate 𝑃𝑃 𝑖𝑖, 𝑦𝑦 : =
“𝑖𝑖 is likely a member of 𝑦𝑦. ” This is a fuzzy proposition (Zadeh, 1975a) as a
function of 𝑖𝑖 and 𝑦𝑦, which indicates that there is inductive support for
the conclusion that individual 𝑖𝑖 belongs to class 𝑦𝑦. The truth function,
𝜏𝜏 , of this fuzzy propositional function can be defined by the
membership function of an inductive fuzzy class, 𝑦𝑦′. Thus, 𝑃𝑃(𝑖𝑖, 𝑦𝑦) is a
fuzzy restriction on 𝑈𝑈 defined by 𝜇𝜇 (Formula 41).
In practice, any function that assigns values between 0 and 1 to
data records can be used as a fuzzy restriction. The aim of IFC is to
calculate a membership function to a fuzzy set of likely members in the
target class. Hence, any type of classifier with a normalized numeric
output can be viewed as an inductive membership function to the target
class, or as a truth function for the fuzzy proposition 𝑃𝑃 𝑖𝑖, 𝑦𝑦 . State of the
art methods for IFC in that sense include linear regression, logistic
regression, naïve Bayesian classification, neural networks, fuzzy
classification trees, and fuzzy rules. These are classification methods
51
yielding numerical predictions that can be normalized in order to serve
as a membership function to the inductive fuzzy class 𝑦𝑦 (Formula 42).
2.4.1
𝑦𝑦 ≔ 𝑖𝑖 ∈ 𝑈𝑈 𝑖𝑖 𝑖𝑖𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 𝑎𝑎 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑜𝑜𝑜𝑜 𝑦𝑦 Formula 42
Univariate Membership Function Induction
This section describes methods to derive membership functions for
one variable based on inductive methods. First, unsupervised methods
are described, which do not require learning from a target class
indicator. Second, supervised methods for predictive membership
functions are proposed.
Numerical attributes can be fuzzified in an unsupervised way,
that is, without a target variable, by calculating a membership function
to a fuzzy class x is a large number, denoted by the symbol ↑: the fuzzy
set of attribute values that are large relative to the available data. This
membership function, 𝜇𝜇↑ : dom 𝐶𝐶 ⟶ 0,1 , maps from the attribute
domain of the target variable into the set of Zadehan truth values. This
unsupervised fuzzification serves two purposes. First, it can be used to
automatically derive linguistic interpretations of numerical data, such
as “large” or “small.” Second, it can be used to transform numerical
attributes into Zadehan target variables in order to calculate likelihoods
of fuzzy events. There are two approaches proposed here to compute a
membership function to this class: percentile ranks and linear
normalization based on minimum and maximum.
For a numeric or ordinal variable 𝑋𝑋 with a value 𝑥𝑥 𝜖𝜖 dom(𝑋𝑋), the
percentile rank (PR) is equal to the sampled probability that the value
of the variable 𝑋𝑋 is smaller than 𝑥𝑥. This sampled probability is
calculated by the percentage of values in dom(𝑋𝑋) that are smaller than
or equal to 𝑥𝑥. This sampled probability can be transformed into a degree
of membership in a fuzzy set. Inductively, the sampled probability is
taken as an indicator for the support of the inductive inference that a
certain value, 𝑋𝑋 , is large in comparison to the distribution of the other
attribute values. The membership degree of 𝑥𝑥 in the fuzzy class
“relatively large number”, symbolized by ↑, is then defined as specified
in Formula 43.
𝜇𝜇↑ 𝑥𝑥 ∶= 𝑝𝑝 𝑋𝑋 < 𝑥𝑥
Formula 43
For example, customers can be classified by their profitability. The
percentile rank of profitability can be viewed as a membership function
52
of customers in the fuzzy set ↑ of customers with a high profitability.
Figure 9 shows an example of an IFC-PR of customer profitability for a
financial service provider.
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
-1'000
-500
0
500
1'000
Customer Profitability
High
Low
Figure 9: Unsupervised IFC by percentile rank.
Adapted from “An Inductive Approach to Fuzzy Marketing Analytics,” by M. Kaufmann,
2009, In M. H. Hamza (Ed.), Proceedings of the 13th IASTED International Conference
on Artificial Intelligence and Soft Computing: Copyright 2009 by Publisher.
A simpler variant of unsupervised fuzzification for generating a
membership function for a relatively large number (↑) is linear
normalization (IFC-LN). For a numerical attribute 𝐶𝐶, it is defined as the
relative distance to the minimal attribute value, as specified in Formula
44.
𝜇𝜇↑ 𝐶𝐶 𝑖𝑖
∶= 𝐶𝐶 𝑖𝑖 − min 𝐶𝐶
max 𝐶𝐶 − min 𝐶𝐶
Formula 44
For the membership function induction (MFI) methods in the
following sections, the target variable for supervised induction must be
a Zadehan variable, 𝑌𝑌: 𝑈𝑈 ⟶ 0,1 , mapping from the universe of
discourse (the set of possible individuals) into the interval of Zadehan
53
truth values between 0 and 1. Thus, 𝑌𝑌(𝑖𝑖) indicates the degree of
membership of individual 𝑖𝑖 in the target class 𝑦𝑦. In the special case of a
Boolean target class, 𝑌𝑌(𝑖𝑖) is equal to 1 if 𝑖𝑖 ∈ 𝑦𝑦, and it is equal to 0 if 𝑖𝑖 ∉
𝑦𝑦. In the analytic training data, a target class indicator 𝑌𝑌 can be
deduced from data attributes in the following way:
o
If an attribute, 𝐴𝐴, is Zadehan with a range between 0 and 1, it can
be defined directly as the target variable. In fact, if the variable is
Boolean, this implies that it is also Zadehan, because it is a special
case (Formula 45).
Zadehan 𝐴𝐴 ⟹ 𝜇𝜇 𝑖𝑖 ∶= 𝐴𝐴(𝑖𝑖)
o
If an attribute, 𝐵𝐵, is categorical with a range of 𝑛𝑛 categories, it can
be transformed into 𝑛𝑛 Boolean variables 𝜇𝜇 ( k = 1, 2, … , n ), where
𝜇𝜇 (𝑖𝑖) indicates whether record 𝑖𝑖 belongs to class 𝑘𝑘, as specified by
Formula 46.
categorical 𝐵𝐵 ⟹ 𝜇𝜇 (𝑖𝑖) ∶=
o
Formula 45
1
0
if 𝐵𝐵(𝑖𝑖) = 𝑘𝑘 else.
Formula 46
If an attribute, 𝐶𝐶, is numeric, this thesis proposes application of an
unsupervised fuzzification, as previously specified, in order to derive
a Zadehan target variable, as formalized in Formula 47. This is
called an inductive target fuzzification (ITF).
numerical 𝐶𝐶 ⟹ 𝜇𝜇 𝑖𝑖 ∶= 𝜇𝜇↑ 𝐶𝐶 𝑖𝑖
Formula 47
The second approach for univariate membership function
induction is supervised induction based on a target variable. In order to
derive membership functions to inductive fuzzy classes for one variable
based on the distribution of a second variable, it is proposed to
normalize comparisons (ratios and differences) of likelihoods for
membership function induction. For example, a normalized likelihood
ratio can represent a membership degree to an inductive fuzzy class.
54
The basic idea of inductive fuzzy classification based on
normalized likelihood ratios (IFC-NLR) is to transform inductive
support of target class membership into a membership function with
the following properties: The higher the likelihood of 𝑖𝑖 ∈ 𝑦𝑦 in relation to
𝑖𝑖 ∉ 𝑦𝑦, the greater the degree membership of 𝑖𝑖 in 𝑦𝑦 . For an attribute 𝑋𝑋,
the NLR function calculates a membership degree of a value 𝑥𝑥 𝜖𝜖 dom(𝑋𝑋)
in the predictive class 𝑦𝑦 , based on the likelihood of target class
membership. The resulting membership function is defined as a relation
between all values in the domain of the attribute 𝑋𝑋 and their NLRs.
As discussed in Section 2.3.1, following the principle of likelihood
(Hawthorne, 2008), the ratio between the two likelihoods is an indicator
for the degree of support for the inductive conclusion that 𝑖𝑖 ∈ 𝑦𝑦, given
the evidence that 𝑋𝑋(𝑖𝑖) = 𝑥𝑥. In order to transform the likelihood ratio into
a fuzzy set membership function, it can be normalized in the interval
between 0 and 1. Luckily, for every ratio, R = A/B, there exists a
normalization, N = A/(A+B), having the following properties:
o
N is close to 0 if R is close to 0.
o
N is close to 1 if R is a large number.
o
N is equal to 0.5 if and only if R is equal to 1.
This kind of normalization is applied to the aforementioned
likelihood ratio in order to derive the NLR function. Accordingly, the
membership µ of an attribute value x in the target class prediction y’ is
defined by the corresponding NLR, as formalized in Formula 48.
µ y ' ( x) := NLR( y | x) =
L( y | x )
L( y | x) + L(¬y | x)
Formula 48
In fact, one can demonstrate that the NLR function is equal to the
posterior probability of y, conditional to x, if both hypotheses y and ¬y
are assumed to be of equal prior probability (Formula 52), by
application of the second form of Bayes’ theorem (Joyce, 2003), as
presented in Formula 50. The trick is to express the probability of the
evidence p(x) in terms of a sum of products of prior probabilities, p0, and
measured likelihoods, L, of the hypothesis and its alternative by
application of Formula 33.
55
Theorem.
NLR ( y | x) = p( y | x) ⇔ p0 ( y) = p0 (¬y)
Proof.
Formula 49
p0 (y)L(y | x)
(c.f. Formula 36)
p(x)
p0 (y)L(y | x)
=
p0 (y)L(y | x) + p0 (¬y)L(¬y | x)
p(y | x) =
[if p(x) = p(y)p(x | y) + p(¬y)p(x |¬y)
and p(y) := p0 (y) and p(x | y) := L(y | x)]
Formula 50
L(y | x)
[if p0 (y) :=: p0 (¬y)]
L(y | x) + L(¬y | x)
=: NLR(y | x), q.e.d.
=
Alternatively, two likelihoods can be compared by a normalized
difference, as shown in Formula 51. In that case, the membership
function is defined by a normalized likelihood difference (NLD), and its
application for classification is called inductive fuzzy classification by
normalized likelihood difference (IFC-NLD). In general, IFC methods
based on normalized likelihood comparison can be categorized by the
abbreviation IFC-NLC.
µ y ' ( x) := NLD ( y | x) =
L( y | x) − L(¬y | x) + 1
2
Formula 51
If a target attribute is continuous, it can be mapped into the
Zadehan domain of numeric truth-values between 0 and 1, and
membership degrees can be computed by a normalized ratio of
likelihoods of fuzzy events. If the target class is fuzzy, for example
because the target variable is gradual, the likelihoods are calculated by
fuzzy conditional relative frequencies based on fuzzy set cardinality
(Dubois & Prade, 1980). Therefore, the formula for calculating the
likelihoods is generalized in order to be suitable for both sharp and
fuzzy characteristics. Thus, in the general case of variables with fuzzy
truth-values, the likelihoods are calculated as defined in Formula 52.
56
n
∑ µ
L( y | x) :=
∑
i =1
x
n
i =1
(i ) µ y (i )
µ y (i)
Formula 52
n
L(¬y | x) :=
∑ µ (i)(1 − µ (i))
∑ (1 − µ (i))
i =1
x
y
n
i =1
y
Accordingly, the calculation of membership degrees using the NLR
function (Formula 52) works for both categorical and fuzzy target
classes and for categorical and fuzzy analytic variables. For numerical
attributes, the attribute values can be discretized using quantiles, and a
piecewise linear function can be approximated to average values in the
quantiles and the corresponding NLR. A membership function for
individuals based on their attribute values can be derived by
aggregation, as explained in Section 2.4.2.
Following the different comparison methods for conditional
probabilities described by Joyce (2003), different methods for the
induction of membership degrees using conditional probabilities are
proposed in Table 1. They have been chosen in order to analytically test
different Bayesian approaches listed by Joyce (2003) for their predictive
capabilities. Additionally, three experimental measures were
considered: logical equivalence, normalized correlation, and a measure
based on minimum and maximum. In those formulae, 𝑥𝑥 and 𝑦𝑦 are
assumed to be Zadehan with a domain of [0,1] or Boolean as a special
case. These formulae are evaluated as parameters in the meta-induction
experiment described in Section 4.2.
Table 1
Proposed Formulae for Induction of Membership Degrees
Method
Formula
Likelihood of y given x (L)
𝐿𝐿 𝑦𝑦 𝑥𝑥 = 𝑝𝑝(𝑥𝑥|𝑦𝑦)
Normalized likelihood ratio (NLR)
Normalized likelihood ratio
unconditional (NLRU)
Normalized likelihood difference
(NLD)
Normalized likelihood difference
unconditional (NLDU)
𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑥𝑥 𝑦𝑦
𝑝𝑝 𝑥𝑥 𝑦𝑦 + 𝑝𝑝 𝑥𝑥 ¬𝑦𝑦
𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑥𝑥 𝑦𝑦 − 𝑝𝑝 𝑥𝑥 ¬𝑦𝑦 + 1
2
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑥𝑥 𝑦𝑦
𝑝𝑝 𝑥𝑥 𝑦𝑦 + 𝑝𝑝(𝑥𝑥)
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑥𝑥 𝑦𝑦 − 𝑝𝑝(𝑥𝑥) + 1
2
57
Conditional probability of y given x
(CP)
Normalized probability ratio
(NPR)
Normalized probability ratio
unconditional (NPRU)
Normalized probability difference
(NPD)
Normalized probability difference
unconditional (NPDU)
Equlivalence - if and only if (IFF)
𝑝𝑝(y|𝑥𝑥)
𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑦𝑦 𝑥𝑥
𝑝𝑝 𝑦𝑦 𝑥𝑥 + 𝑝𝑝 𝑦𝑦 ¬𝑥𝑥
𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑦𝑦 𝑥𝑥 − 𝑝𝑝 𝑦𝑦 ¬𝑥𝑥 + 1
2
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑦𝑦 𝑥𝑥
𝑝𝑝 𝑦𝑦 𝑥𝑥 + 𝑝𝑝(𝑦𝑦)
𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑦𝑦 𝑥𝑥 =
𝑝𝑝 𝑦𝑦 𝑥𝑥 − 𝑝𝑝(𝑦𝑦) + 1
2
Minimum – maximum (MM)
avg
1−x∙ 1−y
Normalized correlation (NC)
min∈() 𝑝𝑝 𝑦𝑦 𝑧𝑧
∙ 1−y∙ 1−x
𝑝𝑝 𝑦𝑦 𝑥𝑥 + min∈() 𝑝𝑝 𝑦𝑦 𝑧𝑧
corr(x, 𝑦𝑦) + 1
2
+ max∈() 𝑝𝑝 𝑦𝑦 𝑧𝑧
A method for discretization of a numerical range is the calculation
of quantiles or n-tiles for the range of the analytical variable. A quantile
discretization using n-tiles partitions the variable range into n intervals
having the same number of individuals. The quantile 𝑄𝑄 𝑖𝑖 for an
attribute value 𝑍𝑍(𝑖𝑖), of a numeric attribute 𝑍𝑍 and an individual 𝑖𝑖, is
calculated using Formula 53, where 𝑛𝑛 is the number of quantiles, 𝑚𝑚 is
the total number of individuals or data records, 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝑖𝑖) is the position
of the individual in the list of individuals sorted by their values of
attribute 𝑍𝑍, and 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡(𝑟𝑟) is the closest integer that is smaller than the
real value 𝑟𝑟.
𝑄𝑄 𝑖𝑖 ∶= 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
𝑛𝑛
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑖𝑖 − 1
𝑚𝑚
Formula 53
The rank of individual 𝑖𝑖 relative to attribute 𝑍𝑍, 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑖𝑖 , in a
dataset 𝑆𝑆 is the number of other individuals, ℎ, that have higher values
in attribute 𝑍𝑍, calculated using Formula 54.
𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝑖𝑖) ∶= ℎ ∈ 𝑆𝑆 ∀𝑖𝑖 ∈ 𝑆𝑆: 𝑍𝑍(ℎ) > 𝑍𝑍(𝑖𝑖)
Formula 54
In order to approximate a linear function, the method of twodimensional piecewise linear function approximation (PLFA) is
proposed. For a list of points in ℝ , ordered by the first coordinate,
𝑥𝑥 , 𝑦𝑦 , 𝑥𝑥 , 𝑦𝑦 , … , 𝑥𝑥 , 𝑦𝑦 , for every point 𝑥𝑥 , 𝑦𝑦 except the last one
58
(𝑖𝑖 = 1,2, … , 𝑛𝑛 − 1), a linear function, 𝑓𝑓 𝑥𝑥 = 𝑎𝑎 𝑥𝑥 + 𝑏𝑏 , can be interpolated
to its neighbor point, where 𝑎𝑎 is the slope (Formula 55) and 𝑏𝑏 is the
intercept (Formula 56) of the straight line.
𝑎𝑎 ∶=
𝑦𝑦 − 𝑦𝑦
𝑥𝑥 − 𝑥𝑥
𝑏𝑏 ∶= 𝑦𝑦 −𝑎𝑎 𝑥𝑥
Formula 55
Formula 56
For the calculation of membership degrees for quantiles, the input
is a list of points with one point for every quantile k. The first
coordinate is the average of the attribute values in k. The second
coordinate is the inductive degree of membership µ y ' in target class y,
given Z(i) is in quantile k, for example derived using the NLR function.
y k := µ y ' (k )
xk := avg{Z (i ) | QnZ (i ) = k}
Formula 57
Finally, a continuous, piecewise affine membership function can
be calculated, truncated below 0 and above 1, and is composed of
straight lines for every quantile k = 1, …, n − 1; n ≥ 2 of the numeric
variable Z (Formula 58).
⎧0
⎪a x + b
1
⎪ 1
⎪
⎪
µ y ( x) := ⎨ak x + bk
⎪
⎪
⎪an−1 x + bn−1
⎪
⎩1
| a1 x + b1 ≤ 0 ∨ an−1 x + bn−1 ≤ 0
| x ≤ x2

| xk < x ≤ xk +1
Formula 58

| x > xn−1
| a1 x + b1 ≥ 1 ∨ an−1 x + bn−1 ≥ 1
The number of quantiles can be optimized, so that the correlation
of the membership function with the target variable is optimal, as
illustrated in Figure 10.
59
Figure 10: Computation of membership functions for numerical variables.
2.4.2
Multivariate Membership Function
Induction
As shown in Figure 11, the proposed process for inducing a
multivariate inductive fuzzy class consists of preparing the data,
inducing univariate membership functions for the attributes,
transforming the attribute values into univariate target membership
degrees, classifying individuals by aggregating the fuzzified attributes
into a multivariate fuzzy classification, and evaluating the predictive
performance of the resulting model.
Figure 11: Proposed method for multivariate IFC.
The idea of the process is to develop a fuzzy classification that
ranks the inductive membership of individuals, 𝑖𝑖, in the target class 𝑦𝑦
gradually. This fuzzy classification will assign individuals an inductive
membership degree to the predictive inductive fuzzy class 𝑦𝑦′ using the
multivariate model 𝜇𝜇 . The higher the inductive degree of membership
𝜇𝜇 (𝑖𝑖) of an individual in 𝑦𝑦′, the greater the degree of inductive support
for class membership in the target class 𝑦𝑦.
60
In order to accomplish this, a training set is prepared from source
data, and the relevant attributes are selected using an interestingness
measure. Then, for every attribute 𝑋𝑋 , a membership function,
𝜇𝜇
: dom 𝑋𝑋 ⟶ [0,1], is defined. Each 𝜇𝜇
is induced from the data such
that the degree of membership of an attribute value 𝑋𝑋 (𝑖𝑖) in the
inductive fuzzy class 𝑦𝑦′ is proportional to the degree of support for the
inference that 𝑖𝑖 ∈ 𝑦𝑦. After that, in the univariate classification step,
each variable, 𝑋𝑋 , is fuzzified using 𝜇𝜇
. The multivariate fuzzy
classification step consists of aggregating the fuzzified attributes into
one multivariate model, 𝜇𝜇 , of data elements that represents the
membership function of individual 𝑖𝑖 in 𝑦𝑦′. This inductive fuzzy class
corresponds to an IFC that can be used for predictive ranking of data
elements. The last step of the process is model evaluation through
analyzing the prediction performance of the ranking. Comparing the
forecasts with the real class memberships in a test set does this. In the
following paragraphs, every step of the IFC process is described in
detail.
In order to analyze the data, combining data from various sources
into a single coherent matrix composes a training set and a test set. All
possibly relevant attributes are merged into one table structure. The
class label 𝑌𝑌 for the target variable has to be defined, calculated, and
added to the dataset. The class label is restricted to the Zadehan
domain, as defined in the previous section. For multiclass predictions,
the proposed process can be applied iteratively.
Intuitively, the aim is to assign to every individual a membership
degree in the inductive fuzzy class 𝑦𝑦′. As explained in Section 2.3.1, this
degree indicates support for the inference that an individual is a
member of the target class 𝑦𝑦. The membership function for 𝑦𝑦′ will be
derived as an aggregation of inductively fuzzified attributes. In order to
accomplish this, for each attribute, a univariate membership function in
the target class is computed, as described in the previous section.
Once the membership functions have been induced, the attributes
can be fuzzified by application of the membership function to the actual
attribute values. In order to do so, each variable, 𝑋𝑋 , is transformed
into an inductive degree of membership in the target class. The process
of mapping analytic variables into the interval [0, 1] is an attribute
fuzzification. The resulting values can be considered a membership
degree to a fuzzy set. If this membership function indicates a degree of
support for an inductive inference, it is called an inductive attribute
fuzzification (IAF), and this transformation is denoted by the symbol ⇝
in Formula 59.
𝑋𝑋 i ⇝ 𝜇𝜇 𝑋𝑋 i Formula 59 61
The most relevant attributes are selected before the IFC core
process takes place. The proposed method for attribute selection is a
ranking of the Pearson correlation coefficients (Formula 38) between
the inductively fuzzified analytic variables and the (Zadehan) target
class indicator Y. Thus, for every attribute, 𝑋𝑋 , the relevance regarding
target 𝑦𝑦 is defined as the correlation of its inductive fuzzification with
the target variable (see Section 3.1.1).
In order to obtain a multivariate membership function for
individuals 𝑖𝑖 derived from their fuzzified attribute values 𝜇𝜇 𝑋𝑋 𝑖𝑖 ,
their attribute value membership degrees are aggregated. This
corresponds to a multivariate fuzzy classification of individuals.
Consequently, the individual’s multivariate membership function
𝜇𝜇 : 𝑈𝑈 ⟶ [0,1] to the inductive fuzzy target class 𝑦𝑦′ is defined as an
aggregation, 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎, of the membership degrees of 𝑛𝑛 attributes,
𝑋𝑋 , 𝑘𝑘 = 1,2, … , 𝑛𝑛 (Formula 60).
𝜇𝜇 𝑖𝑖 ∶= 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝜇𝜇 𝑋𝑋 𝑖𝑖 , … , 𝜇𝜇 𝑋𝑋 𝑖𝑖
Formula 60
By combining the inductively fuzzified attributes into a
multivariate fuzzy class of individuals, a multivariate predictive model,
𝜇𝜇 , is obtained from the training set. This corresponds to a
classification of individuals by the fuzzy proposition “𝑖𝑖 is likely a
member of 𝑦𝑦,” for which the truth value is defined by an aggregation of
the truth values of fuzzy propositions about the individual’s attributes,
𝑋𝑋 𝑖𝑖 is 𝑦𝑦′. This model can be used for IFC of unlabeled data for
predictive ranking. Applying an alpha cutoff, 𝑖𝑖 𝜇𝜇 (𝑖𝑖) ≥ 𝛼𝛼 , an
𝛼𝛼 ∈ [0,1] leads to a binary classifier.
There are different possibilities for calculating the aggregation,
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎. Simpler methods use an average of the attribute membership
degrees, logical conjunction (minimum, algebraic product), or logical
disjunction (maximum, algebraic sum). More sophisticated methods
involve the supervised calculation of a multivariate model. In this
thesis, normalized or cutoff linear regression, logistic regression, and
regression trees are considered. These different aggregation methods
were tested as a parameter in the meta-induction experiment described
in Section 4.2 in order to find an optimal configuration.
Finally, in order to evaluate predictive performance, the classifier
is applied to a hold-out test set, and the predictions 𝜇𝜇 𝑖𝑖 are compared
with the actual target variable (𝑖𝑖). The correlation between the
prediction and the target, corr 𝜇𝜇 , 𝑌𝑌 , can be used to compare the
performance of different IFC models.
62
3 Analytics & Marketing
3.1
Analytics
Analytics is “the method of logical data analysis” (merriamwebster.com, 2012a). According to Zimmermann (1997), data analysis is
the “search for structure in data”. The more data is available, the more
complex it becomes to find relevant information. Consequently,
organizations and individuals analyze their data in order to gain useful
insights. Business analytics is defined as “a broad category of
applications and techniques for gathering, storing, analyzing and
providing access to data to help enterprise users make better business
and strategic decisions” (Turban, Aronson, Liang, & Sharda, 2007, p.
256).
The ability of enterprises to analyze the potentially infinite space
of available data—their capacity of business analytics—is a major
competitive advantage. Companies that use analytics as key strategies
are called “analytics competitors” by Davenport (2006, p. 3). They can
differentiate themselves through a better customer understanding in a
time when products and technologies are becoming more and more
comparable. Analytics competitors apply predictive modeling to a wide
range of fields, such as customer relationship management, supply
chain management, pricing, and marketing. Their top management
understands and advocates that most business functions can benefit
from quantitative optimization. In fact, business analytics can be
applied to almost any area that concerns an enterprise:
o
Customer relationship management (CRM): Prediction of the most
appropriate customer relationship activity.
o
Web analytics: Optimization of the website according to click stream
analysis.
o
Compliance: Prediction of illegal behavior such as fraud or money
laundering.
o
Risk management: Prediction of credit worthiness.
o
Strategic management: Visualization of customer profiles for
product or market strategies.
o
Marketing: Prediction of customer product affinity.
Kohavi,
Rothleder,
and
Simoudis
(2002)
identified
comprehensibility and integration as the driving forces of emerging
trends in business analytics. Today, good business analytics is either
comprehensible or integrated: Human decision makers need
63
interpretable, visual, or textual models in order to understand and
apply analytic insights in their daily business. Contrarily, information
systems demand machine readable, integrated automated interfaces
from analytic applications to operational systems in order to apply
scorings or classifications in automated processes.
In order to enhance the data basis for analytics, large data pools,
such as data warehouses, have been built. Today, the problem is no
longer the lack of data, but the abundance of it. Data mining (in the
broader sense) is “the process of discovering patterns in data”. In a more
narrow sense, those “structural patterns” help not only to explain the
data but also to make “nontrivial predictions on new data” (Witten &
Frank, 2005, p. 5). The common aspect of the two definitions is the
discovery of patterns in data. In contrast, the second definition
emphasizes the inductive aspect of data mining, namely prediction.
Data mining, in its broader sense, means any form of discovery of
knowledge in data, whereas in the narrower sense, it means model
induction for prediction using statistics and machine learning. One aim
of data mining is the induction of models for datasets. If applied
correctly, models not only describe the past, but also help to forecast the
future. Successful data mining is a practical demonstration of inductive
logic. There are two categories of machine learning: non-supervised and
supervised learning. The former extracts models such as clusters or
association rules from data without labels. The latter induces a model
for labeled data, representing the relationship between the data and the
label. Continuous labels lead to regression models, and categorical ones
lead to classification models or classifiers.
Descriptive data analysis presents data as is, that is, it describes
the data without generalization or conclusion. It is completely
deductive, because the numbers are true without need for inductive
support. In a business context, this is often called reporting. A report is
an understandable tabular or graphical representation of relevant data.
Usually, methods of classification and aggregation are applied to
consolidate data into meaningful information. Frequently applied
techniques for descriptive data analysis are deductive classification,
aggregation, and grouping. For deductive classification, data records are
classified according to an a priori predicate. Therefore, the class
membership is known in advance. Aggregation is the calculation of a
scalar value from a vector or set, such as sums, averages, or extrema.
Often, these aggregations are grouped by one or more variables. In such
cases, the aggregated values are calculated for each combination of
values in the grouping variables For example, the average cost and
benefit per customer segment could be an interesting data description
for many companies. An important part of descriptive data analysis is
the presentation of the results. The raw output of an analysis is usually
64
a matrix or table, but human decision makers are usually managers
that want to decide quickly and intuitively based on insights that can be
understood. Therefore, visualization of data descriptions can help
increase the understanding of analytic insights.
Data descriptions might indicate certain conclusions or inferences
that cannot be directly deduced from the data. For example, if twice as
many women than men have bought a certain product, an inference
could be that women are more likely to buy this product in the future.
The question, therefore, is, what kind of inductive inferences can be
made from the given data descriptions, and how high is the degree of
support in the data for this kind of inductive inference? There are two
important inductive techniques in data analysis: attribute selection and
prediction. In data abundance, not only the number of available records
is incredibly large but also the number of available attributes.
Automated attribute selection applies statistical and machine learning
techniques for a ranking of association between attributes and a
prediction target, and thus filters out the most relevant attributes for
inductive inferences regarding a target class. Prediction means
inference of an unknown feature (a target variable) from known
features (the analytic variables). This can be accomplished to a certain
degree of likelihood using a model induced by methods from statistics or
machine learning. There are two kinds of predictions: inductive
classification and regression. Inductive classification means predicting a
categorical target variable, and regression means predicting a
numerical target variable. Fuzzy classification is a special case in which
the prediction consists of a numeric value indicating the gradual
membership in a category.
Insights from analytics are traditionally presented to human
decision makers as tables and graphics. Today, more and more analytic
results are loaded automatically from analytic systems into operational
systems, a concept that is called integrated analytics. Those systems
can automate certain decision processes or display analytic insights to
users of operational systems. Marketing decisions can be automated,
such as choosing an individualized advertisement message in the online
channel (Kaufmann & Meier, 2009). The process of integrated analytics
can be described in five steps. First, analytic data is collected from
different sources. Second, a predictive model is induced from the data,
either in a supervised form using a target variable or in an
unsupervised form of clustering or association analysis. Third, a
prediction, classification, or score is assigned to the original data based
on the induced model. Fourth, these predictions are transmitted to the
operational systems where they are applied to decision support. Finally,
outcomes of decision support—for example, sales decisions and
65
actions—are fed back into the analytic data pool for meta-analysis and
iterative optimization.
Application of fuzzy logic techniques to analytics is called fuzzy
data analysis. This permits gradual relationships between data and
predictions. The application of fuzzy logic to analytics is appropriate
when there is fuzziness in the data, the prediction target, or in the
association between the two (Zimmermann H. J., 1997). The advantage
of fuzzy classification methods is that the class boundary is not sharp:
Patterns detected by fuzzy data mining provide soft partitions of the
original data. Furthermore, an object can belong to several classes with
different degrees of membership (Zimmermann H. J., 2000). Two main
advantages of fuzzy logic techniques pointed out by Hüllermeier (2005)
are the elimination of certain threshold effects because of gradual soft
data partitioning, and the comprehensibility and interpretability of
resulting models containing linguistic terms. Hüllermeier discusses
possible fields of application of fuzzy set theory to data mining: Fuzzy
cluster analysis partitions a set of objects into several fuzzy sets with
similar properties in an unsupervised manner. Learning of fuzzy rules
computes decision rules with fuzzy propositions. Fuzzy decision trees
are a special case of fuzzy rules in which every node of the tree
partitions the data into a fuzzy subset with an optimal distinction
criterion. Fuzzy association analysis computes association rules
between fuzzy restrictions on variables. Fuzzy classification partitions
sharp data into fuzzy sets according to a classification predicate (Meier,
Schindler, & Werro, 2008). If this predicate is inferred by induction, the
process is called inductive fuzzy classification, or IFC (Kaufmann &
Meier, 2009).
In business analytics, scoring methods are used for ranking data
records according to desirable features. Multivariate methods based on
multiple attributes, for instance linear or logistic regression, can predict
targets such as customer profitability, product affinity, or credit
worthiness. The scores are used in daily decision making. Record
scoring based on data in an information system is an inference about
unknown target variables, which is not deductive. Rather, it is a form of
inductive inference. This introduces fuzziness into decision support
because there is only some degree of support for the hypothesis that a
given record belongs to a certain class. In fact, statistical models can
increase the likelihood of correct inductive classification. Accordingly,
for every scored record, there is fuzziness in the membership to the
target class. Consequently, fuzzy logic is the appropriate tool for
reasoning about those fuzzy target classes. The solution is to compute a
continuous membership function mapping from the data into the fuzzy
target class to which every record has a degree of membership. Ranking
or scoring models yield continuous predictions instead of crisp
66
classifications. Those models can serve as target membership functions.
Thus, scoring methods correspond to an IFC.
The proposed IFC method provides a means for computing
inductive membership functions for a target class. These membership
functions can be visualized. Furthermore, they can be used for inductive
fuzzification of attributes, which can be applied for attribute selection
and for improving predictive models. Thus, the proposed IFC methods
can be applied in the following areas of data analysis:
o
Selection: Attributes can be scored by the correlation of
automatically generated inductive membership degrees with a
Boolean or Zadehan target class in order to select relevant
attributes.
o
Visualization: Induced membership functions can serve as humanreadable diagrams of association between analytic and target
variables using a plot of automatically generated membership
functions.
o
Prediction: Datasets used for prediction can be inductively fuzzified
by transforming the original data into inductive target membership
degrees, which can improve the correlation of predictions with the
actual class membership in existing statistical methods.
3.1.1
Selection
In practice, there is an abundance of data available. The problem
is to find relevant attributes for a given data mining task. Often,
thousands of variables or more are available. Most machine learning
algorithms are not suited for such a great number of inputs. Also,
human decision makers need to know which of those customer
attributes are relevant for their decisions. Therefore, the most relevant
attributes need to be selected before they can be used for visualization
or prediction.
In order to find relevant attributes for predicting a target class y,
an attribute selection method can be derived from membership
functions induced by IFC, using the method proposed in Section 2.4. All
possible analytic variables can be ranked by the correlation coefficient
of their NLR with the target variable. For every attribute, Xk, the
membership function in the predictive target class y’, denoted by µy’(Xk),
is computed. The membership in target y is indicated by a Zadehan
variable Y, which is a variable with a domain of gradual truth values in
the interval between 0 and 1. Then, the Pearson sample correlation
coefficients between the NLR-based membership degrees and the
Zadehan values of Y are calculated. Thus, the relevance of attribute Xk
regarding target y is defined as the correlation of its inductive
67
fuzzification with the target variable (Formula 61). The most relevant
variables are those with the highest correlations.
relevance( X k ) := corr ( µ y ' ( X k ), Y )
Formula 61
The fuzzification of analytic variables prior to attribute relevance
ranking has the advantage that all types of analytic variables—
linguistic, categorical, Boolean, Zadehan, and numeric variables—can
be ranked using the same measure. Choosing correlation as a measure
of association with the target has the advantage that it is a standard
aggregate in SQL, and thus readily available as a well-performing
precompiled function in major database systems.
To illustrate the proposed method, the attributes of the German
credit data are ranked regarding the target class of customers with a
good credit rating (Table 2). Those attributes have been inductively
fuzzified with NLRs. The Pearson coefficient of correlation between the
resulting membership degrees and the Boolean target variable good
credit rating has been calculated. One can see that, for a credit rating,
checking status, credit history, or duration of customer relationship are
quite correlated attributes, whereas for example, the number of existing
credits is less relevant.
This kind of attribute selection can be used as an input for
visualization and prediction. When a target class is defined, the
relevant correlated variables can be identified. A visualization of five to
ten most relevant variables gives good insights about associations in the
data.
Table 2
Example of an Attribute Ranking regarding NLR/Target Correlation1
1
Attribute
Correlation of NLR with target
Checking status
0.379304635
Credit history
0.253024544
Duration
0.235924759
Purpose
0.190281416
Savings status
0.189813415
Credit amount
0.184522282
Housing
0.173987212
Property magnitude
0.157937117
Age
0.136176815
http://archive.ics.uci.edu/ml/datasets/Statlog+(German+Credit+Data)
68
Employment
0.128605942
Personal status
0.093525855
Installment commitment
0.091375548
Other payment plans
0.079139824
Foreign worker
0.077735317
Job
0.061875096
Other parties
0.060046535
Own telephone
0.051481771
Existing credits
0.042703728
Residence since
0.034584104
Num. dependents
0.007769441
3.1.2
Visualization
The IFC-NLR method can be applied to create visualizations of
associations between an analytic variable (a factor) and a class (a
target). A visualization of variable association is a human-readable
presentation that allows the reader to see the distribution of target
likelihood in tabular form or as a graph.
Using IFC for visualization, the table consists of relations between
factor values and normalized target likelihood ratios, and the graph
results in a plot of the inductive membership function. For categorical
factors, the graph is a bar chart. For numerical factors, the membership
function is plotted as a continuous line. The advantage of this method is
that the notion of membership of a factor X in a target Y is semantically
clear and intuitively understandable by readers. Furthermore, the
semantics of the NLR function is mathematically clearly defined.
As an example, for two variables from the German credit dataset,
checking status and duration, the inductive fuzzy membership function
in the target class of customers with a good credit rating can be
visualized as shown in Figure 12. This graphic can be interpreted as
follows: Customers without checking accounts or with a balance of more
than $200 are more likely to have good credit histories than not.
Customers with a negative balance are more likely to have bad credit
ratings. For the duration of the credit (in months), the shorter the
duration is, the higher the likelihood of good credit history. Credits with
durations of more than 19 months have higher likelihoods of being
associated with bad credit histories, and accordingly, the NLR is less
than 0.5.
69
Figure 12: Visualization of relevant attributes and their association with the target as
an inductive membership function.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 1731 Copyright 2012 by
Publisher.
70
3.1.3
Prediction
A method for the application of likelihood-based IFC to prediction
has been introduced by Kaufmann and Meier (2009). The basic idea is to
create a multivariate inductive model for target class membership with
a combination of inductively fuzzified attributes derived by membership
function induction (see Section 2.4). The proposed approach for
application of likelihood-based IFC to prediction consists of a univariate
inductive fuzzification of analytic variables prior to a multivariate
aggregation. This has the advantage that non-linear associations
between analytic variables and target membership can be modeled by
an appropriate membership function. As presented in Figure 13, the
following steps are applied in order to derive an inductive membership
degree of individual i in the prediction y’ for class y based on the
individual’s attributes:
o
A: The raw data consists of sharp attribute values.
o
B: An inductive definition of the membership function for the
attribute values in the predictive fuzzy class y’ is calculated using
the previously described IFC-NLC methods.
o
C: The attribute values are fuzzified using the derived membership
functions from step B. This step is called inductive attribute
fuzzification (IAF), defined as supervised univariate fuzzy
classification of attribute values.
o
D: After that, the dataset consists of fuzzified attribute values in the
interval [0,1], indicating the inductive support for class membership.
o
E: The several fuzzified analytic variables are aggregated into a
membership degree of individuals in the predictive class. This can
be a simple conjunction or disjunction, a fuzzy rule set, or a
statistical model derived by supervised machine learning algorithms
such as logistic or linear regression.
o
F: This results in a multivariate membership function that outputs
an inductive membership degree for individual i in class y, which
represents the resulting prediction.
It is proposed to preprocess analytic data with IFC methods in
order to improve prediction accuracy. More specifically, attributes used
for data mining can be transformed into inductive membership degrees
in the target class, which can improve the performance of existing data
mining methods. The basic idea is to create a multivariate model for a
target variable with a combination of inductively fuzzified attributes
derived by membership function induction.
71
Figure 13: Proposed schema for multivariate IFC for prediction.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 172. Copyright 2012 by
Publisher.
Empirical tests (Section 4.2) have shown that using the proposed
method of IAF significantly improves the average correlation between
prediction and target. For binary target variables, a combination of IAF
with an NLR, and the subsequent application of a logistic regression,
turned out to be optimal. This configuration is illustrated in Figure 14.
For numerical targets, a linear fuzzification (LF) of the target, IAF
using NLD, and calculation of a regression tree have turned out to be
optimal. This configuration is illustrated in Figure 15.
However, this improvement by IAF can be shown only in average
prediction correlation. There are instances of data in which the
application of IAF lowers the predictive performance. Therefore, an IAF
is worth a try, but it has to be tested whether it really improves the
prediction or not in the given data domain. IAF provides a tool for finetuning predictive modeling, but answering the question about the best
classification algorithm in a specific context takes place in the data
mining process in which the algorithm with the best results is selected
(Küsters, 2001).
72
Figure 14: IFC prediction with binary target classes based on categorical attributes.
Figure 15: IFC prediction with Zadehan target classes based on numerical attributes.
73
74
3.2
Marketing Analytics
“Analytics competitors understand that most business functions—
even those, like marketing, that have historically depended on art
rather than science—can be improved with sophisticated quantitative
techniques.” (Davenport, 2006, p. 4)
The application of the method of data analysis to marketing is
called marketing analytics. This can optimize the marketing mix and
individual marketing on multiple channels. An important challenge in
marketing is its financial accountability: making costs and benefits of
marketing decisions explicit. Spais and Veloutsou (2005) proposed that
marketing analytics could increase the accountability of marketing
decisions. They point out the need of incorporating marketing analytics
in daily marketing decision making, which is a conceptual shift for
marketers to work with mathematical tools. Furthermore, they suggest
that the problem of fuzziness in consumer information should be
addressed by fuzzy logic techniques.
Individual marketing is capable of accounting for all costs and
benefits of a marketing campaign to daily CRM decisions. Because all
customers, advertisements, and sales are recorded in information
systems, the benefit of marketing activities can be measured. For
example, in an individual marketing campaign, the sales ratio of
targeted customers can be compared to average or random customers,
and the sales increase can be directly linked to the corresponding
marketing campaigns.
The application of IFC methods to data analysis can be used in
marketing analytics. Specifically, customer and product analytics,
target selection, and integrated analytics are proposed as possible
applications for the methods that have been developed in this thesis.
Using IFC methods for selection, visualization, and prediction can
support these four fields in marketing analytics. The benefits of IFC
include automated generation of graphics with a linguistic
interpretation, clear model semantics for visualization by membership
function induction, and a possibly better target selection because of
optimized predictive models using IAF. As shown in Figure 16, general
application of IFC to the field of analytics for selection, visualization,
and prediction can be applied to marketing analytics, specifically.
Visualizations of induced membership functions of attributes in
marketing targets provide fuzzy customer and product profiles.
Furthermore, MFI allows fuzzy definitions of target groups based on
customer characteristics. Finally, scoring methods can represent
membership functions to fuzzy target groups defined as fuzzy sets of
customers to which every individual customer has a degree of
75
membership. These can be integrated into automated analytics
processes for individual marketing.
Figure 16: Proposed areas of application of IFC to marketing analytics.
3.2.1
Customer Analytics
The aim of customer analytics is to make associations between
customer characteristics and target classes of interesting customers
with desirable features understandable to marketing decision makers.
The question is, which customer features are associated with the target?
A customer report visualizes the likelihood of target class membership
given different values of an attribute in order to show relevant features
that distinguish the target class from the rest of the customers. In the
context of CRM, there are several target customer classes that are
interesting for profiling:
o
Product affinity: Customers who have bought a given product.
o
Recency: Customers who are active buyers because they have
recently bought a product.
o
Frequency: Customers who buy frequently.
o
Loyalty: Customers who have used products or services for a long
period of time.
o
o
76
Profitability: Customers who are profitable to the enterprise.
Monetary value: Customers who generate a high turnover.
A customer profile for the mentioned target customer classes
should answer the question, what distinguishes customers in this class
from other customers? Thus, the association between class membership
and customer attributes is analyzed. The customer attributes that can
be evaluated for target likelihood include all of the aforementioned
characteristics, and additionally, socio-demographic data, geographic
data, contract data, and transaction behavior recorded in operational
computer systems, such as CRM interactions and contacts in direct
marketing channels.
If one looks more closely at the target customer classes, one can
see that, often, these classes are fuzzy. For example, to generate a “high
turnover” is a fuzzy proposition, and the corresponding customer class is
not sharply defined. Of course, one could discretize the class using a
sharp boundary, but this does not reflect reality in an appropriate
manner. Therefore, it is proposed to visualize fuzzy customer classes as
fuzzy sets with membership functions.
In order to generate customer profiles based on a target class and
the relevant customer attributes, the method of MFI by NLRs presented
in Section 2.4.1 can be applied. It is proposed to select the relevant
customer attributes first, using the method previously described. After
that, the relevant attributes are called profile variables. For a profile
variable X, the values xk ∈ dom(X) are called profile characteristics.
They are assigned an inductive membership degree for the predictive
target class y’ defined by µy’(xk) := NLR(y|xk). A plot of the
corresponding membership function represents a visual fuzzy customer
profile for variable X, as illustrated by Figure 17.
Two instances of a fuzzy customer profile using NLRs are shown
in Figure 17, in which the customers of investment funds products of a
financial service provider are profiled by customer segment and
customer age. Interpretation of the graphs suggests that customers
above the age of 50 and customers in the segments Gold and Premium
are more likely to buy investment funds than the average customer.
A fuzzy customer profile derived with the IFC-NLR method
creates a visual image of the customer class to be analyzed by showing
degrees of membership of customer features in the target characteristic.
Marketing managers can easily interpret the resulting reports.
77
Figure 17: Schema of a fuzzy customer profile based on IFC-NLR, together with two
examples of a fuzzy customer profile.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 173. Copyright 2012 by
Publisher.
3.2.2
Product Analytics
Marketing target groups can be defined by typical characteristics
of existing product users. This method is based on analysis of existing
customer data in the information systems of a company, which is an
instance of secondary market research. The data of customers that are
product users (the test group) is compared to the data of customers that
do not use the product (the control group). The attributes that separate
test and control group most significantly are used for defining the target
group of potential customers. These attributes are the most selective
ones. Accordingly, the target group is derived inductively by similarity
to the set of existing product users. As an example, a customer profile
78
could show that customers between 30 and 40 years of age in urban
areas are twice as likely to be product users as other customers. In such
a case, the target group for this particular product could be defined
accordingly.
Analytic target group definitions are based on a set of customer
characteristics. However, these characteristics differ in relevance. Thus,
the set of relevant customer characteristics for a product target group is
a fuzzy set because the boundary between relevant and non-relevant
characteristics is gradual. The corresponding membership degree can be
precisiated by a relevance or selectivity metric. It is proposed that an
NLR be used as a measure of selectivity.
The likelihood of product usage, UP (the notation is for product P),
given that a customer record, i, in the existing customer database d has
characteristic c, can be computed accurately as a sampled conditional
probability, p(c|UP)d, defined as the number of product users that have
this feature divided by the total number of product users (Formula 62).
In this formula, c(i) and UP(i) are Boolean truth values that indicate
presence or absence of a customer characteristic and the usage of the
product.
p(c | U P ) d :=
| {i ∈ d | c(i ) ∧ U P (i )} |
| {i ∈ d | U P (i )} |
Formula 62
This product usage likelihood can be compared to the likelihood,
p(c |¬UP)d, of the opposite hypothesis that a customer is not a product
user, given characteristic c, calculated as the number of non-users of the
product that have characteristic c divided by the total number of nonusers (Formula 63).
p(c | ¬U P ) d :=
| {i ∈ d | c(i ) ∧ ¬U P (i )} |
| {i ∈ d | ¬U P (i )} |
Formula 63
Thus, the selectivity of a customer characteristic c can be
expressed by the ratio between the likelihood of product usage, given c,
and likelihood of product non-usage, given c (Formula 64). This ratio
can be normalized as formalized in Formula 65.
LR (U P | c) d :=
p (c | U P ) d
p ( c | ¬U P ) d
Formula 64
79
NLR (U P | c) d :=
p (c | U P ) d
p(c | U P ) d + p(c | ¬U P ) d
Formula 65
Finally, the selectivity of characteristic c, expressed as an NLR,
represents an inductive degree of membership of this characteristic in
the fuzzy target group definition for product P. An inductive fuzzy
target group definition, t, based on analysis of database d is a fuzzy set
of customer features and its membership function is defined by the
corresponding NLR (Formula 66).
µt (c) d := NLR (u P | c) d
Formula 66
A customer characteristic may or may not be defined by a
granular attribute value. Characteristics can also be computed by
functional combination of several basic attribute values. Furthermore,
the customer characteristic indicator c(i) can indicate a fuzzy truth
value in the interval [0,1] if the corresponding characteristic c is a fuzzy
proposition such as “high turnover.” In that case, the definition of the
NLR for fuzzy truth values from Section 2.4.1 can be applied. However,
the aim of analytic target group definition is to find or construct optimal
defining target customer characteristic indicators with the highest
possible degree of membership in the target group definition. These
indicators can be constructed by logical connections or functions of
granular customer characteristics.
This kind of analytic target group definition is a conceptual one,
suited for presentation to human decision makers, because it is
intuitively understandable. It results in a ranking of customer
characteristics that define typical product customers. However, for
integrated analytics in analytic CRM, a scoring approach is more
promising because it yields better response rates, although the resulting
models may be less comprehensible.
3.2.3
Fuzzy Target Groups
Contemporary information technology provides the means to
individualize marketing campaigns. Each customer is targeted not only
directly but also individually with an advertisement message, decision,
or activity. The behavior of an organization toward an individual
customer is analytically customized according to the customer’s
classification, and customers with different characteristics are targeted
with different behaviors. Individual customers are assigned a score for
different CRM targets based on a predictive multivariate model. IFC
80
methods can be applied to customer data for individual marketing in
order to calculate a predictive scoring model for product usage. This
model can be applied on the data to score customers for their product
affinity, which corresponds to an IFC of customers in which the
predictive model is a multivariate membership function.
An aim of customer analytics is the application of prediction to the
selection of a target group with likelihood of belonging to a given class
of customers. For example, potential buyers or credit-worthy customers
can be selected by applying data analysis to the customer database.
This is done either by segmentation or scoring (Sousa, Kaymak, &
Madeira, 2002). By applying a segmentation approach, sharp sets of
customers with similar characteristics are calculated. Those segments
have a given size. For individual marketing, a scoring approach is more
promising, in which every customer is assigned a score predicting a
likelihood of response. The score is calculated by application of a
predictive multivariate model with numeric output, such as neural
networks, linear regression, logistic regression, or regression trees.
When a numeric score can be normalized, it represents a membership
function to a fuzzy set of customers with a high response likelihood—a
fuzzy target group—to which every customer has a degree of
membership. This scoring process can be applied for every product.
Thus, for every individual customer, the degree of membership to all
possible cross-selling target groups is known, and in direct customer
contact, the customer can be assigned the advertisement message with
the highest score.
There are different methods for fuzzy customer scoring. For
example, fuzzy clustering for product affinity scoring was presented by
Setnes, Kaymak, and van Nauta Lemke (1998), in which the reason for
using fuzzy systems instead of neural networks is declared as the
comprehensibility of the model. Kaufmann and Meier (2009) evaluated
a supervised fuzzy classification approach for prediction using a
combination of NLRs with an algebraic disjunction. In this thesis, a
synthesis of probabilistic modeling and approximate reasoning applying
fuzzy set theory is proposed for prediction (as previously explained),
using univariate inductive membership functions for improving the
target correlations of logistic regressions or regression trees. However,
all inductive scoring methods that yield a numeric value representing
response likelihood can be normalized in order to represent an inductive
membership function to a fuzzy set of customers, and can, therefore, be
categorized as IFC.
A classical sharp CRM target group T ⊂ C is a subset of all
customers C defined by a target group definition t: T := {i ∈ C | t(i)}. In
the case of scoring methods for target selection, the output of the model
application is numeric and can be normalized in the interval [0,1] to
81
represent a membership function. In that case, the target group is a
fuzzy set T’ and the (normalized) customer score is a membership
function, µT’:C→[0,1]. This score is defined by a predictive model, M, as
a multivariate combination of n customer attributes, X1, …, Xn, as
shown in Formula 67.
µT' (i) := M ( X 1 (i),..., X n (i))
Formula 67
Because T’ is a fuzzy set, it has to be defuzzified when a campaign
requires a binary decision. For example, the decision about contacting a
customer by mail has to be sharp. An alpha cutoff of the fuzzy set allows
the definition of individual marketing target groups of optimal size
regarding budget and response rate, and leads to a sharp target group
T , as formalized in Formula 68.
α
Tα := {i ∈ C | µT ' (i) ≥ α }
3.2.4
Formula 68
Integrated Analytics
For individual marketing, the fuzzy target group membership
degrees are predictive scores that are processed via the CRM system,
which dispatches them to the distribution channel systems for mapping
customers to individualized advertisement messages. In all computer
supported channels with direct customer contact, inbound or outbound,
as soon as the customer is identified, a mapping can be calculated to the
product to which the customer has the highest score. Based on that
mapping, the advertisement message is chosen. In the online channel,
logged-in customers are displayed individual advertisement banners.
Customers are sent individual letters with product advertisement. In
the call center and in personal sales, if the customer can be identified,
the agent can try cross-selling for the next best product.
In analytic customer relationship management (aCRM), customer
data is analyzed in order to improve customer interactions in multiple
channels (Turban et al., 2007). This can be applied to individual
marketing. Target groups for aCRM campaigns are derived analytically
and individually by application of classification and regression models
to individual customer data records. The aim is to increase campaign
response rates with statistical methods. Cross-selling campaigns
analyze product affinity of customers given their features. Churn
82
(change and turn) campaigns target customers with a low loyalty
prediction in order to re-gain their trust. Figure 18 illustrates a
schematic information system that can enable aCRM processes.
In contemporary enterprises, due to intensive application of
computing machinery and electronic databases in business processes,
there exist large amounts of possibly useful customer data in various
software applications that can serve as data sources for aCRM. The
heterogeneity of data storage implies a necessity for data integration
before they can be analyzed. A system that automates this task is called
a customer data warehouse (Inmon, 2005). Based on this integrated
data pool, analytics provides inductions of predictive models for
desirable customer classes, such as product purchase, profitability, or
loyalty. If these models output a gradual degree of membership to the
target classes, their application to the customer database can be called
an IFC. These membership degrees are defined by a prediction of class
membership for individual customers, sometimes called lead detection.
The corresponding fuzzy sets of customers can be called fuzzy target
groups. These analytic results, called leads by aCRM managers, are
transferred into the operational CRM application, where their
presentation to human decision makers provides decision support for
direct customer contact. The utilization of analytic data output (leads)
takes place in individual marketing channels with direct customer
contact—for instance, web presence, mailing, call center, or personal
sales. The results of aCRM campaigns, such as sales, are electronically
collected and fed back into the customer data warehouse in order to
improve future aCRM campaigns by meta-analytics. This mechanism is
called a closed loop. Generally, aCRM is an instance of a business
intelligence process (Gluchowski, Gabriel, & Dittmar, 2008), which
involves data sourcing, integration, analytics, presentation, and
utilization. Furthermore, it is also an instance of integrated analytics,
in which analytics are integrated into operational systems with
feedback mechanisms.
In comparison to mass marketing, in individual marketing, every
customer is provided with the advertisement that fits best. Especially
when customer databases, such as a data warehouse, are present,
individual scoring is possible based on available data: The customer’s
target membership score is used to assign an individual advertisement
message to each customer. These messages are delivered electronically
via customer relationship management application to individual
marketing channels. Campaign target groups are individualized. All
customers in the target groups are known individually and are
contacted directly.
83
Figure 18: Inductive fuzzy classification for individual marketing.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 170. Copyright 2012 by
Publisher.
84
3.2.5
Case Study: IFC in Online Marketing at
PostFinance
The following case study shows the application of the proposed
methods in practice, from a point of view of information systems and
methodology. Membership function induction is applied to a real-world
online marketing campaign. A comparison with crisp classification and
random selection is made.
PostFinance Inc.2 is a profitable business unit of Swiss Post. Its
activities contribute significantly to the financial services market in
Switzerland. PostFinance is an analytic enterprise feeding business
processes with information gained from predictive analytics.
The online channel of PostFinance supports individual
advertisement banners for logged-in customers. Based on the customer
score, a customer is assigned the advertisement message for the product
with the highest response likelihood computed for that particular
customer. By clicking on the banner, the customer has the opportunity
to order the product directly.
The marketing process of PostFinance uses analytical target
groups and predictive scores on product affinity for individual
marketing. Target groups for online marketing campaigns are selected
using inductive classification on the customer data warehouse. Figure
19 shows the process from customer data to individualized online
marketing. Target groups are defined in the data warehouse. This is
done using a predictive model (e.g., logistic regression) or a crisp
classification (e.g., customers above the age of 50 with a balance higher
than CHF 10,000). A dataset is generated that assigns an individual
advertisement message to every single customer based on his or her
target group. This dataset is loaded into the online platform, where it
controls the online advertisements. In the online channel, for every
customer that logs in, the individual advertisement message is mapped
according to the previous target group selection.
A case study was conducted with PostFinance in 2008 in order to
test IFC in a real online marketing campaign. Inductive fuzzy
classification was applied in a PostFinance online marketing campaign
promoting investment funds. The aim of the prediction was to forecast
customers with an enhanced likelihood of buying investments funds.
The resulting fuzzy classification yielded a fuzzy target group for an
individual marketing campaign in which every customer had a gradual
degree of membership.
2
http://www.postfinance.ch
85
Figure 19: Analytics applied to individual marketing in the online channel at
PostFinance.
Adapted from “ An Inductive Fuzzy Classification Approach applied to Individual
Marketing.,” by M. Kaufmann and A. Meier, 2009, In Proceedings of the 28th North
American Fuzzy Information Processing Society Annual Conference, p. 4. Copyright
2009 by Publisher.
The same advertisement message was shown to three groups of
customers: a test group of 5,000 customers with the highest degree of
membership to a fuzzy target group, a control group of 5,000 customers
selected by classical crisp target group definition, and a second control
group with randomly selected customers. The hypothesis was that a
gradual scoring approach (an inductive fuzzy customer classification)
would provide better response rates than a crisp Boolean target
selection (a segmentation approach), and that a fuzzy target group
selection using typical characteristics of product users would classify
customers more softly and eliminate certain threshold effects of Boolean
classification because attributes can be compensated for with a
multivariate gradual approach. It was tested whether customers in a
fuzzy target group selected with a scoring approach had a higher
response rate than customers in a crisp target group selected with a
classical segmentation approach. Furthermore, the two groups were
compared with a random selection of customers. In the following, it is
shown how the data mining methodology presented in Section 2.4 has
been applied to a real-world direct marketing campaign.
86
In order to prepare a test set for model induction, a sample
customer dataset was selected from the customer data warehouse.
Because of the excellent quality of the cleaned and integrated data in
the data warehouse, the data preparation step was accomplished by a
simple database query. The dataset contained anonymous customer
numbers associated with customer specific attributes. As the target
variable, the class label was set to 1 if the customer had investment
funds in his product portfolio and to 0 if else. Mutual information of the
dependent variables with the target variable was chosen as a ranking
method. The following attributes were selected as relevant:
o
Customer segment (CS; 0.047 bit),
o
Overall balance (OB; 0.035 bit),
o
Customer group (CG; 0.016 bit),
o
Age (A; 0.013 bit).
o
Number of products (NP; 0.036 bit),
o
Loyalty (L; 0.021 bit),
o
Balance on private account (BP; 0.014 bit), and
Table 3
Conditional Probabilities and NLRs for a Categorical Attribute
X1:
customer
segment
Y=1
Y=0
p(X1 | Y=1) p(X1 | Y=0) NLRY=1(X)
Basis
SMB
Gold
Premium
Total
11'455
249
12'666
5'432
29'802
308'406
2'917
54'173
10'843
376'339
0.38
0.01
0.43
0.18
0.82
0.01
0.14
0.03
0.32
0.52
0.75
0.86
Using the method presented in Section 2.4, for each of the relevant
attributes, the fuzzy restriction corresponding to the likelihood of
having investment funds was induced. In the following section, the
induction processes for a categorical and a continuous attribute are
described in detail.
As the first example, in the domain of the categorical attribute
customer segment (X1), there are four values: Basis, SMB, Gold, and
Premium. For customers who have not yet bought investment funds,
the aim is to define a degree of membership in a fuzzy restriction on the
customer segment domain in order to classify them for their likelihood
to buy that product in the future.
The frequencies and conditional probabilities are presented in
Table 3. The first column indicates the customer segment. Column Y=1
87
contains the number of customers of each segment who have bought
investment funds. Column Y=0 contains the number of customers of
each segment who have not bought that product. According to Formula
69, the membership degree of segment Basis in the fuzzy restriction y1
was induced as follows:
µ y (" Basis" ) =
1
p( X 1 = " Basis" | Y = 1)
p( X 1 = " Basis" | Y = 1) + p( X 1 = " Basis" | Y = 0)
0.38
=
= 0.32 = NLRY =1 (" Basis" )
0.38 + 0.82
Formula 69
The other degrees of the membership were induced analogically.
The resulting values are shown in column NLR of Table 3. The
corresponding membership function is illustrated in Figure 20.
Members hip degree to product
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
B as is
NL R
S MB
G old
P remium
C us tomer S egment
Figure 20: Membership function induction for a categorical attribute.
Adapted from “ An Inductive Fuzzy Classification Approach applied to Individual
Marketing.,” by M. Kaufmann and A. Meier, 2009, In Proceedings of the 28th North
American Fuzzy Information Processing Society Annual Conference, p. 4. Copyright
2009 by Publisher.
As a second example, for the continuous attribute overall balance,
the membership function was induced in the following way: First, the
NLR for deciles of the attribute’s domain was calculated analogically to
the previous example, represented by grey squares in Figure 21. Then, a
function, A / (1 + exp(B – C ln(x + 1)))+D, was fitted by optimizing the
parameters A to D using the method of John (1998). The resulting
membership function for the customer overall balance is shown as a line
in Figure 21.
88
Members hip degree to product
NL R
F (x) = 0.71/(1+ E X P (7.1-­‐0.80*L N(x+ 1)))+ 0.09
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
50'000
100'000
O verall B alance
150'000
Figure 21: Membership function induction for a continuous attribute.
Adapted from “ An Inductive Fuzzy Classification Approach applied to Individual
Marketing.,” by M. Kaufmann and A. Meier, 2009, In Proceedings of the 28th North
American Fuzzy Information Processing Society Annual Conference, p. 5. Copyright
2009 by Publisher.
Every relevant attribute of the original dataset was transformed
to a fuzzy membership degree using SQL directly in the database.
Categorical variables were transformed using a case differentiation
statement. For example, the attribute customer segment was
transformed using the following SQL command: select case
when customer_segment = 'Basis' then 0.32
when customer_segment = 'SMB' then 0.52
when customer_segment = 'Gold' then 0.75
when customer_segment = 'Premium' then 0.86
end as fuzzy_customer_segment
from ads_funds
Continuous variables were transformed using a function expression. For
example, the attribute overall balance was fuzzified using the following
SQL expression:
89
select
0.71/(1+EXP(7.1-0.8*LN(overall_balance + 1)))+0.09
as fuzzy_overall_balance
from ads_funds
The individual fuzzy attribute domain restrictions were then
aggregated to a multivariate fuzzy class of customers. This was done
using an SQL statement implementing the gamma operator
(Zimmermann & Zisno, 1980) defined in Formula 70.
Γγ ( µ y1 (e j ), … , µ yn (e j ))
:= µ  yi (e j ) γ ⋅ µ  yi (e j )1−γ
i
Formula 70
i
:= (1 − ∏ 1 − µ yi ( xij ))γ ⋅ (∏ µ yi ( xij ))1−γ
i
i
Table 4
Optimization of the Gamma Parameter for Multivariate Fuzzy Aggregation
I(Y;Y')
I(Y;Y'=1)
LR(Y'=1)
Gamma = 0
0.018486
0.4357629
6.7459743
Gamma = 0.5
0.0185358
0.4359879
6.7488694
Gamma = 1
0.0209042
0.4787567
7.2359172
In order to define the gamma parameter, different performance
measures were calculated. As shown in Table 4, a gamma of 1,
corresponding to an algebraic disjunction or full compensation, was
most successful. Thus, the multivariate fuzzy classification was
performed using the following SQL statement:
Select
(1(1-fuzzy_number_of_products)
* (1-fuzzy_customer_segment)
* (1-fuzzy_overall_balance)
90
* (1-fuzzy_loyalty)
* (1-fuzzy_age)
* (1-fuzzy_balance_on_private_account)
* (1-fuzzy_customer_group)
) as multivariate_fuzzy_classification
from f_ads_funds
As a result, a fuzzy class of customers was calculated, for whom
the degree of membership indicates the product affinity for investment
funds and represents a product affinity score. This can be used in
individual marketing for defining target groups for different products. In order to test the resulting fuzzy classifier, a pilot marketing
campaign was performed using the resulting fuzzy target group. First, a
target group of 5,000 customers with the highest membership degree
was selected from the fuzzy class using an alpha cutoff (Test Group 1).
Second, as a comparison, 5,000 other customers were selected using a
crisp classification (Test Group 2), using the following crisp constraints:
Select case when
Customer segment in (‘Gold’, ‘Premium’, ‘SMB’)
And Customer_group = 50 Plus
And Loyalty > 14
And Number_of_products > 1
And Age between 35 and 75
And Balance on private account > 3000
And Overall_balance > 20000
Then 1 else 0 end as multivariate_crisp_classification
From ads_funds
This conventional target group selection used the same seven
customer attributes as the fuzzy classification, but the classification
was done using a crisp constraints predicate. Third, 5,000 customers
were selected randomly (Test Group 3) in order to compare inductive
classification to random selection.
To each of those 15,000 customers, an online advertisement for
investment funds was shown. After the marketing campaign, the
product selling ratio during three months was measured. The results
are shown in Table 5. The column Y=1 indicates the number of
customers who have bought the product. The product selling ratio for
the target group selected by IFC was the most effective. This individual
marketing campaign using inductive fuzzy target group selection
91
showed that fuzzy classification can lead to a higher product selling
ratio than crisp classification or random selection. In the case study,
IFC predictions of product affinity were more accurate than those of the
crisp classification rules on exactly the same attributes. The conclusion
is that, in comparison to crisp classification, fuzzy classification has an
advantageous feature that leads to better response rates because it
eliminates certain threshold effects by compensation between
attributes.
Table 5
Resulting Product Selling Ratios per Target Group
Test group
1: Fuzzy classification
2: Crisp classification
3: Random selection
92
Y=1
31
15
10
Y=0
4939
5037
5016
Sales Rate
0.63%
0.30%
0.20%
4 Prototyping & Evaluation
Three software prototypes were programmed as proofs of concept
of the technological aspects of the proposed methods for automated MFI
in fuzzy data analysis. Master’s students developed two of them, iFCQL
and IFC-Filter for Weka. The author developed the inductive fuzzy
classification language IFCL. This implementation also allows
experiments on the implemented methods for experimental evaluation.
Using a meta-induction approach, the designed method was applied for
prediction in several real-world datasets in order to analyze
characteristics and optimal parameters of the constructed methodology
and to compare it to conventional predictive approaches. Classical
inductive statistical methods were applied to gain insights about
induction by IFC.
4.1
4.1.1
Software Prototypes
IFCQL
The first attempt of prototyping software for IFC was developed in
a master’s thesis by Mayer (2010). The basic idea was an extension of
the existing fuzzy classification and query language FCQL, for which a
prototype interpreter was built by Werro (2008).
In order to derive fuzzy membership functions directly from the
underlying data, the FCQL language was extended with commands to
induce membership functions in order to classify data based on them.
For example, it was intended to describe MFI using the syntax “induce
fuzzy membership of <dependent variable> in <target variable> from
<relation>.” For more detail on the iFCQL syntax, see the report by
Mayer (2010).
As shown in Figure 22, the architecture of iFCQL reflects the
language-oriented design approach. The user can enter commands in
iFCQL, which are translated into SQL, or he or she can enter SQL in
the command shell, which is transmitted directly to the database by a
database connector. Using language instead of a graphical user
interface, it is possible to save and reload scripts for data mining
processes—this was the intention. Nevertheless, the implementation of
the iFCQL language interpreter, consisting of lexer, parser, evaluator,
and SQL generator, turned out to be a complex task that is not directly
associated with IFC.
The original idea was to develop a tool to support the proposed
data mining methodology of this thesis (Section 2.4). Thus, there was
one statement class for each of the six steps: data preparation (audit),
93
MFI, fuzzification of attribute values, attribute selection, fuzzy
classification of data records, and model evaluation. Eventually, only
three syntax elements for the main tasks were implemented, namely
MFI, univariate classification of attributes, and multivariate
classification of records.
Figure 22: Architecture of the iFCQL interpreter.
Adapted from “Implementation of an Inductive Fuzzy Classification Interpreter,“
(master's thesis), by P. Mayer, 2010, p. 59. Copyright 2010 by Mayer, University of
Fribourg, Switzerland.
4.1.2
IFCL
This section presents an implementation of a tool that enables the
computation of inductive membership functions based on the
methodology proposed in Section 2.4. The inductive fuzzy classification
language (IFCL) is a research prototype. Its goal is to show the
94
feasibility of implementing the data mining methodology based on IFC.
It is a markup language designed for simplicity of description and
parsing; therefore, the development of the language interpreter was
kept simple. In addition to supporting the basic steps of the proposed
method, the scope is to support automated experiments for
benchmarking the predictive performance of the proposed methodology.
This meta-induction experiment and its results are presented in Section
4.2
Figure 23: Inductive fuzzy classification language (IFCL) system architecture.
As shown in Figure 23, the system architecture of IFCL consists of
an IFCL file, a command line shell, the IFCL program itself, Weka
regression algorithms, a relational database, and a data file. In the
IFCL file, the data mining process steps are described in the IFCL
language. In a command line shell, the IFCL program is invoked, and
the corresponding IFCL file is passed to the program. There is no
graphical user interface because the research concentrated on
evaluating automated inductive inferences. The IFCL actions
representing steps of the data mining process are interpreted by the
IFCL program, which translates them into SQL queries to the relational
database. For the supervised multivariate aggregation functions linear
regression, logistic regression and M5P regression trees, open source
implementations of mining algorithms from the Weka machine learning
workbench (Hall et al. 2009) are accessed by the IFCL program. The
IFC process steps that have been translated to SQL are executed on a
database server. Analytic data is loaded into the database via a data file
in comma-separated value format (CSV). The IFCL language supports a
simple load functionality that creates the necessary database tables
based on a data description and loads records from a file into database
tables. The data file contains the data to be analyzed, and the IFCL
program provides the means to compute predictive models based on the
data in the form of membership functions.
The functionality of the software encompasses all steps of the
proposed data mining process for IFC as presented in Section 2.4.2. This
95
includes data preparation, univariate MFI, fuzzification of attribute
values, attribute selection, multivariate aggregation, data classification,
prediction, and evaluation of predictions. This prototype provides the
possibility
o
to prepare data using arbitrary SQL statements,
o
to induce membership functions with all thirteen methods proposed
in Table 1 in Section 2.4.1,
o
to fuzzify attribute values using the induced membership functions
directly in the database,
o
to rank attribute relevance by sorting correlation
membership degrees and target class indicators,
o
to aggregate many inductively fuzzified attributes into a fuzzy class
membership for data records using eight different aggregation
methods shown in Table 6,
o
to assign membership degrees by fuzzy classification to data records
in the database,
o
and to evaluate predictive models with correlation analysis.
o
Executing IFCL script files in batch mode,
o
Dropping a database table,
o
Loading data into the database,
o
Classification of data,
between
All of these steps can be programmed in IFCL-syntax. IFCL is
fully automatable using scripting files. All models are displayed and
stored in SQL syntax. Therefore, it provides repeatability of
experiments and comprehensibility of models with respect to their
database implementation as queries. Details on the syntax and
application of IFCL can be found in the appendix (Section A.4) To
summarize, the functionality of IFCL encompasses the following points:
o
Connecting to the database,
o
Executing an SQL script,
o
Inducing a membership function,
o
Aggregating multiple variables,
o
Data preparation, and
o
Evaluating predictions,
o
Attribute selection.
96
Steps of the data mining process in Section 2.4.2 are invoked in so
called IFCL actions. A sequence of actions is listed in IFCL files. IFCprocesses can be described in a tree-like structure of IFCL files. There
are nodes and leaves of the IFCL tree. An IFCL file is either a node as a
list of calls to other IFCL files or a leaf containing actual execution
sequences. IFCL leaves consist of a database connection followed by a
sequence of IFCL actions. An IFCL file can call the execution of several
other predefined IFCL files in batch mode using the action execifcl. In
order to do so, one needs to indicate the correct operation system file
path of the corresponding IFCL file, either as an absolute path, or a
path relative to the location of the process that invoked the IFCL
program.
IFCL can connect to a database with the action connect. This is
necessary for operation and has to be done at the beginning of an IFCL
file leaf. One needs to know the hostname of the database server, the
service identifier, the port number, a valid username, and a password.
The user must be granted the rights to read tables. For data
classifications, the user needs grants to create tables. The action drop
allows the deletion of an existing database table. The IFCL program
tests whether the indicated table exists, and executes a drop command
to the database if and only if the table exists in the database. This is
useful in preventing an exception because of an attempt to drop a nonexisting table. IFCL allows executing SQL statements in the database
with the action execsql. A single SQL command can be indicated
directly within the IFCL syntax. Scripts (statement sequences) can be
called from a separate SQL script file. The file system path has to be
indicated in order to call an execution of an SQL file.
In order to load analytic data into the database server, there are
two possibilities. First, the SQL*Loader can be called, which is highly
parameterized. Second, a simpler load mechanism can be invoked
involving less code. The SQL*Loader has the advantage that it is
extremely configurable by a control file. The drawback is that there is a
large amount of code that needs to be written for simpler load tasks.
Therefore, for simpler loads that do not involve transformation, the
IFCL load utility is a faster method to load data into the database. The
call to the SQL*Loader can be made from an IFCL file using the action
sqlldr. In order to load data, one needs a data file containing the
analytic data and an SQL*Loader control file for the configuration of
the load process. Furthermore, one can indicate two SQL*Loader
parameters, namely skip and errors. (For more details, consult the
SQL*Loader documentation.) For simpler data loads, the IFCL program
provides a data loading utility with the action load. The advantage is
that the load action automatically creates or recreates a table for the
data load based on an enumeration of attributes and their type, where
97
the type is either n (numerical) or c (categorical). In the enumerations,
the number of the attribute and the number of its corresponding type
must be equal: Type 3 corresponds to Attribute 3. The attribute names
will correspond to table columns, so their syntax is syntactically
restricted to valid column names. The load action must also be provided
a path to a file containing the load data, and a delimiter that separates
data cells in the text file. White spaces and semicolons cannot be used
as delimiters because they are syntactical elements of the IFCL
language, and an escape mechanism has not been implemented.
In order to induce a univariate membership function for a single
attribute to the target class, the IFCL action inducemf can be used. A
database table is indicated on which the MFI will be based. One specific
table column is defined as a target variable. As analytic variables, a
specific table column can be defined, a list of columns can be
enumerated, or all columns can be chosen for MFI. One of several
different methods for MFI can be chosen (see Section 2.4.1). The output
containing the induced membership function in SQL syntax is written
to the indicated output file. This SQL classification template will be
used later for univariate fuzzification of input variables. When more
than one column is chosen as analytic variables, some of those variables
can be skipped. The IFCL action will skip columns if they are
enumerated in the corresponding action parameter. This means that
those columns are not considered for MFI and they are also not copied
into the fuzzified table. Alternatively, some columns can be left out by
indicating the IFCL action to leave columns in the corresponding action
parameters the way they are. This means that, for those columns, no
membership function is induced, but those columns are copied into the
fuzzified table in their original state. This can be useful to incorporate
original key columns in a fuzzified table.
In order to fuzzify the input variables using the membership
functions obtained, the action classify can be invoked using the
previously generated SQL classification template file. The classification
action can also apply multivariate fuzzy classification templates
obtained by the IFCL action aggregatemv, which aggregates multiple
variables into a multivariate fuzzy class. Template files generated by
the IFCL actions inducemf and aggregatemv contain an SQL query in
which the input and output tables are parameterized. A data
classification applies these two parameters (classified table and output
table) to this query template stored in the template file. Data is read
from an existing table, the classified table, and transformed using the
induced membership functions, and the resulting transformation is
written into the output table.
In order to aggregate multiple variables into a multivariate fuzzy
classification, the corresponding aggregation function can be computed
98
using the IFCL action aggregatemv. A base table is indicated on which
the aggregation takes place. This is usually a transformed table
containing inductively fuzzified attributes derived with the actions
inducemf and classify, but for Weka regression algorithms (linreg,
logreg, regtree) the aggregatemv action can also be applied to the
original data. Again, the analytic variables and the target variable are
indicated, which usually correspond to those variables used for MFI. As
aggregation operator, either an unsupervised aggregation, such as a
minimum (min), maximum (max), algebraic product (ap), algebraic sum
(as), or average (avg), can be chosen or a supervised aggregation, such
as a linear regression (linreg), a logistic regression (logreg), or a
regression tree (regtree), can be calculated. The column that contains
the aggregated membership value for each row must be given a column
alias. The resulting aggregation function is written as a query template
to an output file for which the path is defined in the action definition.
This query template represents the multivariate model as a
membership function in the target class. It can be used for data
classification and prediction with the classify action.
In order to evaluate correlations between columns, the IFCL
action evaluate can be applied. This is useful to evaluate predictive
performance, but also for attribute ranking and selection. Using IFCL,
an attribute selection can be accomplished by ranking the inductively
fuzzified analytic variables by their correlations with the target
variable. In order to do this, for all columns of the table, a membership
function to the target is induced in a training set with the inducemf
action and the columns are transformed into a membership degree with
the classify action. Then, the correlations of all fuzzified variables and
the target variable can be evaluated with the evaluate action.
4.1.3
IFC-Filter for Weka
The IFC-NLR data mining methodology introduced by Kaufmann
and Meier (2009) has been implemented by Graf (2010) as a supervised
attribute filter in the Weka machine learning workbench (Hall et al.,
2009). The aim of this implementation is the application of IFC-NLR in
a typical data mining process. This IFC-Filter allows the evaluation of
the algorithm.
Weka can be used for data mining in order to create predictive
models based on customer data. The IFC-Filter can facilitate
visualization of the association between customer characteristics and
the target class, and it can improve predictive performance of customer
scoring by transforming attribute values into inductive fuzzy
membership degrees, as previously explained. Thus, it can perform MFI
99
and IAF. As shown in Figure 24, the Weka software provides data
access to relational databases (RDB), comma separated values (CSV),
and attribute relation file format (ARFF); classification and regression;
and predictions. The IFC-Filter implements the functionality for MFI,
IAF, and visualizations based on the calculated membership functions.
Figure 24: Software architecture of the IFC-Filter.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 184. Copyright 2012 by
Publisher.
The IFC-Filter transforms sharp input data into membership
degrees that indicate the inductive support for the conclusion that the
data record belongs to the target class. In order to do so, first,
membership functions are induced from data and optionally displayed
to the screen. Then, these functions are applied to fuzzify the original
attributes. Visualization and prediction based on the concepts of MFI
and IAF are two main uses for the IFC-Filter software in the data
mining process.
In order to visualize associations between variables, inductive
membership functions can be plotted with the method IFC-NLR
described earlier. Thus, for every analytic variable, a function mapping
from the variable’s domain into a degree of membership in the inductive
fuzzy target class is displayed graphically. This plot gives intuitive
100
insights about associations between attribute values and the target
class.
For prediction, the transformation of crisp attribute values from a
data source into inductive membership degrees can enhance the
performance of existing classification algorithms. The IFC-Filter
transforms the original attribute values into inductive membership
indicating target class likelihood based on the original value. After that,
a classical prediction algorithm, such as logistic regression, can be
applied to the transformed data and to the original data in order to
compare the performance of IAF. It is possible and likely that IAF
improves prediction. When that is the case, IAF data transformation
can be applied to huge data volumes in relational databases using the
SQL code generated by the IFC-Filter. The software is available for
download3 in source and binary format for researchers and practitioners
for experimenting and practicing IFC.
Figure 25: Knowledge flow with IFC-Filters.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 186. Copyright 2012 by
Publisher.
As proposed in the section on application of IFC to analytics
(Section 3.1) the IFC-Filter can visualize membership functions that
indicate target membership likelihoods. Figure 25 shows two
screenshots of the membership function plots for the variables Duration
and Checking status from the German Credit dataset. The IFC-Filter
has the possibility to activate a frame containing a graphical
3
http://diuf.unifr.ch/is/ifc (accessed 11.2010)
101
illustration of the resulting membership functions. Each analytical
variable is represented by a tab in this frame. The presentation of
numerical analytical variables differs slightly from that of categorical
analytical variables, because it presents a continuous membership
function.
As shown in Figure 26, the illustration of numerical analytical
variables consists of four fields. The first field is a table containing the
NLRs with the corresponding quantiles and average quantile values
(AQVs). The second field shows a histogram containing the NLRs and
their corresponding AQVs. The third field shows the membership
function of the analytical variable. The fourth field shows the
membership function in SQL syntax for this particular analytical
variable, which can be used directly in a relational database for fuzzy
classification of variables.
The illustration of categorical analytical variables can be reduced
to three fields. The first field is a table containing the NLR with the
corresponding quantile and average value of the quantile. The second
field shows a histogram containing the NLRs corresponding to the
categorical values. The third field shows the membership function in
SQL syntax.
An additional tab, the SQL panel, contains the membership
functions of all analytic variables in SQL. It displays a concatenation of
the membership functions in SQL syntax for all analytical variables
that have been input for MFI by the IFC-Filter. This database script
can be applied in a database in order to transform large database tables
into inductive degrees of membership in a target class. This can
improve the predictive quality of multivariate models.
102
Figure 26: Visualization of membership functions with the IFC-Filter.
Adapted from “Fuzzy Target Groups in Analytic Customer Relationship Management,”
by M. Kaufmann and C. Graf, 2012, In A. Meier and L. Donzé (Eds.), Fuzzy Methods for
Customer Relationship Management and Marketing, p. 188. Copyright 2012 by
Publisher.
103
104
4.2
Empirical Tests
This section describes the experiments that have been conducted
using the IFCL prototype (Section 4.1.2). Sixty real-world datasets (see
the appendix in Section A.1) with categorical and numerical target
attributes have been used to build predictive models with different
parameters. The results of these experiments allow the answering of the
following research questions: Which is a suitable MFI method for the
application of fuzzy classification for prediction? Which aggregation
operators are suitable for the multivariate combination of fuzzy classes?
Can it be statistically supported that this method of IFC leads to higher
predictive performance?
In order to find the answers to these research questions, the aim is
to inductively infer conclusions about inductive methods, which is called
a meta-induction scheme. Thus, quantitative statistical methods are
applied to the data on the performance of IFC using different
parameters gained by running the automated prediction experiments.
4.2.1
Experiment Design
The aim of these experiments was to determine the optimal
parameters for a maximal predictive performance in order to gain
insights about IFC. Using the scripting functionality of the IFCL
prototype, a systematic benchmarking of predictive performance of
different IFC algorithms was conducted. From the UCI Machine
Learning Repository,4 Weka numeric datasets archive,5 and Weka UCI
datasets archive,6 60 statistical datasets were used for systematic
automated prediction experiments using different parameters. These
datasets are listed in Table 9 and Table 10 in Section A.1. They contain
real-world data from different processes that can be used for testing
machine learning algorithms. They have been chosen by their tabular
structure and by the type of the prediction target. Apart from that, they
were chosen randomly.
For categorical target classes, a Boolean target variable was
extracted with a simple predicate (see Table 9, column Target). For
numerical target classes, a Zadehan target variable was generated
using inductive target fuzzification using both linear and percentile
fuzzification (see Table 10).
http://archive.ics.uci.edu/ml/ (accessed 03.2010)
http://sourceforge.net/projects/WEKA/files/datasets/datasetsnumeric.jar/datasets-numeric.jar/download (accessed 08.2010)
6 http://sourceforge.net/projects/WEKA/files/datasets/datasets-UCI.jar/datasetsUCI.jar/download (accessed 08.2010)
4
5
105
The experimental parameters included 13 methods for IAF (Table
1), two types of base target classes (numerical or categorical), two types
of inductive target fuzzification for numerical target variables (ITF;
Formula 43 and Formula 44), and eight methods for multivariate
aggregation (Table 6). As benchmarks, three state of the art prediction
methods (regression trees, linear and logistic regression) were applied
for prediction without prior IAF.
Table 6
Different Aggregation Methods used for Prediction Experiments
Abbreviation
logreg
regtree
linreg
avg
min
ap
as
max
Description
Logistic regression
Regression trees
Linear regression
Average membership degree
Minimum membership degree
Algebraic product
Algebraic sum
Maximum membership degree
In order to test predictive performance, for every combination of
dataset, IAF method, ITF method, and aggregation method, the
correlation of the prediction with the actual class value was computed.
This has been done using a repeated hold-out cross-validation method:
For every parameter combination, the correlation result was calculated
10 times with different random splits of 66.7% training data versus
33.3% test data, and the results were averaged.
Prediction quality of model instances was measured with the
Pearson correlation coefficient (Weisstein, 2010b) between the (Boolean
or Zadehan) predictive and target variables in order to measure the
proximity of predictions and targets in two-dimensional space.
Significance of prediction correlation for experiment parameters
θ and θ was estimated with a non-parametric statistical test, the onesided Wilcoxon Signed-Rank (WSR) test (Weisstein, 2012a), where the
null hypothesis, H0, supposes that the average prediction correlation,
γ(θ1) := avg(corr(Y’,Y)| θ1), of experiments with parameter θ1 is smaller
or equal to γ(θ2), where Y is the target variable, Y’ is the predictive
model output, and both Y and Y’ are Boolean or Zadehan variables with
a range of truth values within [0,1]. The statistical test hypothesis
is formalized by H0: γ(θ1) ≤ γ(θ2). If the corresponding p-value returned
by WSR is significantly small, it is likely that θ1 performs better than
θ 2.
In addition to their statistical meaningfulness, the reason for
using these measures, Pearson correlation and WSR, is that both are
1
2
,
106
precompiled functions in certain database systems, and thus computed
very efficiently for large numbers of experiments.
4.2.2
Results
This subsection describes the results of the meta-inductive
prediction experiments. It describes the findings about the best
performing methods for MFI and multivariate class aggregation.
Additionally, it evaluates which dataset parameters correlate with the
performance improvement gained by IAF.
In order to evaluate the best performing aggregation method, the
average correlation between prediction and actual value of the target
variable has been calculated for every aggregation method for
categorical as well as numerical targets over all datasets and MFI
methods. In Figure 27, the result of this evaluation for categorical
variables is shown. Clearly, logistic regression is the best aggregation
method for binary targets with an average correlation of 0.71.7
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Average Correlation
logreg
regtree
linreg
avg
min
ap
as
max
0.71
0.69
0.69
0.64
0.53
0.52
0.45
0.45
Figure 27: Average prediction correlation for different aggregation methods for binary
targets.
A Wilcoxon Signed Rank (WSR)8 test returned a p-value of 0.0115 for
the hypothesis that the average correlation between prediction and
target for logistic regression is smaller than or equal to that of
regression trees. This means that, with a high probability, the
predictive performance of logistic regression is higher than that of
regression tree. The p-value for the difference in performance between
logistic and linear regression is even smaller: 0.00105. Therefore,
7 See
8
Query 2 in Chapter A
See Query 3 and data in Table 11 in Chapter A
107
logistic regression performs significantly better as an aggregation
method for prediction than the other methods.
As illustrated by Figure 28, the best performing method of
multivariate class aggregation for numerical targets is regression trees
with an average correlation of 0.66.9 The WSR significance test10
showed, with a p-value of 1.30152E-06, that the average correlation of
regression trees is smaller than or equal to the average correlation of
linear regression. Again, this result is, statistically, very significant.
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Average Correlation
regtree
linreg
avg
logreg
ap
max
min
as
0.66
0.58
0.52
0.52
0.45
0.42
0.41
0.39
Figure 28: Average prediction correlation for different aggregation methods for fuzzified
numerical targets.
In order to determine the best performing combination of
membership induction method for IAF and multivariate aggregation
method, the average correlation between prediction and actual value of
the target variable has been calculated for every combination. As shown
in Figure 29, the best combination for binary target variables is logistic
regression, with an NLR as MFI method for IAF, with an average
prediction correlation of 0.71.11 The best benchmark method without
IAF is regression trees on rank 24 of 107 with an average prediction
correlation of 0.699. The top 23 combinations all involved a form of IAF.
The best performing method was to combine an IAF using NLR with a
logistic regression.
The WSR significance test returned a p-value of 0.180012 for the
hypothesis that the average correlation between prediction and target
for logistic regression with NLR is smaller than or equal to that of
logistic regression with IAF using NLD. This means that, for logistic
See Query 7 in Chapter A
See Query 8 and data in Table 12 in Chapter A
11 See Query 4 in Chapter A
12 See Query 5 and data in Table 14 in Chapter A
9
10
108
regression with IAF, NLR does not perform significantly better than
NLD. However, the WSR showed, with a p-value of 0.04888, that a
logistic regression with IAF using NLR performs equally or less well
compared to regression trees without IAF. Consequently, a significant
performance improvement of logistic regression models can be achieved,
on average, with IAF.
0.720
0.710
0.700
0.690
0.680
0.670
0.660
0.650
0.640
Average Correlation
logreg(nlr):
Rank 1
logreg(nld):
Rank 2
logreg(cp):
Rank 3
0.712
0.710
0.710
regtree(no
logreg(npd): logreg(npdu)
IAF): Rank
Rank 4
: Rank 5
24
0.709
0.708
0.699
logreg(no
IAF): Rank
30
linreg(no
IAF): Rank
44
0.688
0.669
Figure 29: Average prediction correlation for combinations of membership function
induction and aggregation methods for binary targets.
0.69
0.68
0.67
0.66
0.65
0.64
0.63
0.62
0.61
Average Correlation
Linear
Percentile
0.69
0.64
Figure 30: Average prediction correlation of predictions with linear versus percentile
ITF, given regression trees as the aggregation method.
In order to apply MFI methods from Section 2.4, numerical target
variables need to be fuzzified. As proposed in this section, two
approaches for inductive target fuzzification (ITF) have been proposed.
Both were tested and compared in the experiments. In Figure 30, one
can see that the average correlation between prediction and target was
much higher for linear ITF.13 The WSR significance test14 returned a pvalue of 0.0000289. Thus, linear target fuzzification performs
significantly better for predicting fuzzy classes than percentile target
fuzzification.
13
14
See Query 9 in Chapter A
See Query 10 and data in Table 13 in Chapter A
109
0.800
0.700
0.600
0.500
0.400
0.300
0.200
0.100
0.000
Average Correlation
regtree(nld): regtree(nldu):
Rank 1
Rank 2
0.697
0.695
regtree(nc):
Rank 3
0.690
regtree(npdu) regtree(npr):
: Rank 4
Rank 5
0.690
regtree(no
IAF): Rank
14
linreg(no
IAF): Rank
25
logreg(no
IAF): Rank
38
0.621
0.591
0.569
0.689
Figure 31: Average prediction correlation for combinations of membership function
induction and aggregation methods for fuzzified numerical targets.
In Figure 31, the results for experiments with combinations of
different IAF and aggregation methods for data with numerical
targets15 are shown, given linear ITF of numerical target variables. In
this case, IAF with NLD, together with a supervised aggregation using
Weka M5P regression trees (regtree), is the best method with an
average correlation of 0.697. The difference between this combination
and the benchmark prediction using regression trees without IAF is
significant with a WSR p-value16 of 0.0026.
There are several parameters that distinguish datasets. These
include number of variables, number and percentage of categorical or
numerical variables, number of rows, ratio between number of rows and
number of columns, and the linearity of the prediction target. This
linearity was measured by the average correlation of the predictions of a
linear regression benchmark model, YLR, with the target variable Y:
Linearity := avg(corr(YLR,Y)). The question is whether any of these
parameters correlate significantly with the improvement of predictive
performance of regression models with IAF.
In order to answer this question for binary target predictions,
correlation and level of significance of correlation were calculated, using
a non-parametric correlation test, between the parameters and the
relative improvement of the predictive performance of logistic
regression if the analytic variables are fuzzified using NLR versus
logistic regression without IAF.17 As the statistical test, Spearman’s rho
(Weisstein, 2012b), with a z-score significance test was applied, which
exists as a precompiled function in the database.18
15
16
See Query 11 in Chapter A
See Query 12 and data in Table 15 in Chapter A
See Query 14 in Chapter A
18 http://docs.oracle.com/cd/B28359_01/server.111/b28286/functions029.htm
(6/2012)
17
110
Table 7
Correlation and p-Value of Dataset Parameters and Improvement of Logistic Regression
with IAF for Binary Target Variables
Parameter
Correlation with
Improvement with IAF
p-value
Number of categorical variables
0.335516011
0.069899788
Number of numeric variables
-0.097871137
0.60688689
Percentage of categorical variables
0.206624854
0.273290146
Percentage of numeric variables
-0.206624854
0.273290146
Number of columns
Average correlation of the prediction of
linear regression
0.228400654
0.224758897
-0.50567297
0.004362976
Number of rows (n)
Ratio number of rows / number of
columns
0.089665147
0.637498617
-0.001557286
0.993483625
Improvement of logistic
regression
by IAF NLR
0.3
0.2
0.1
0
-0.1
0
0.2
0.4
0.6
0.8
1
1.2
-0.2
-0.3
Average correlation of predictions of linear regression
Figure 32: Relationship between target linearity and IAF benefit for binary target
variables.
As shown in Table 7, the correlation between target linearity and
improvement by NLR fuzzification is negative and significant (p <
0.005). This means that the less linear the connection between the
analytic variables and the target variable is, the more a Boolean
prediction using logistic regression can be improved with an IAF using
NLRs. In Figure 32, this association is visualized in a scatter plot.19
19
See Query 15 and data in Table 16 in Chapter A
111
For fuzzified numerical prediction targets, Table 8 shows similar
results.20 The improvement of regression trees with IAF NLD correlates
negatively and significantly with the linearity of association between
analytic and target variables. The lower the performance of linear
regression, the more a predictive regression tree model can benefit from
IAF using NLD. This association is illustrated21 in Figure 33.
Table 8
Correlation and p-Value of Dataset Parameters and Improvement of Regression Trees
with IAF for Zadean Target Variables
Parameter
Correlation w/
Improvement by IAF p-value
Number of categorical variables
0.121461217
0.522578264
Number of numeric variables
0.03610839
0.84975457
Percentage of categorical variables
0.036497846
0.848152597
Percentage of numeric variables
-0.036497846
0.848152597
Number of columns
Average correlation of the
prediction of linear regression
0.195910119
0.299481536
-0.583537264
0.000712151
Number of rows (n)
Ratio number of rows / number of
columns
0.081432863
0.668807278
-0.043381535
0.81994093
Improvement of regression trees
by IAF NLD
1
0.8
0.6
0.4
0.2
0
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Average correlation of predictions of linear regression
Figure 33: Relationship between target linearity and IAF benefit for fuzzified numerical
target variables.
20
21
See Query 14 in Chapter A
See Query 15 and data in Table 17 in Chapter A
112
“Every species is vague, every term goes cloudy at its edges, and
so in my way of thinking, relentless logic is only another name for
stupidity—for a sort of intellectual pigheadedness. If you push a
philosophical or metaphysical enquiry through a series of valid
syllogisms—never committing any generally recognized fallacy—you
nevertheless leave behind you at each step a certain rubbing and
marginal loss of objective truth and you get deflections that are difficult
to trace, at each phase in the process. Every species waggles about in its
definition, every tool is a little loose in its handle, every scale has its
individual.” (H. G. Wells, 1908)
113
114
5 Precisiating Fuzziness by Induction
5.1
Summary of Contribution
A methodology for IFC has been introduced. This method allows
automated derivation of membership functions from data. The idea is to
translate sampled probabilities into fuzzy restrictions. Thus, a
conceptual switch from the probabilistic to the possibilistic view is
applied in order to reason about the data in terms of fuzzy logic.
Business applications of MFI have been proposed in the area of
marketing analytics in the fields of customer and product analytics,
fuzzy target groups, and integrated analytics for individual marketing.
A prototype inductive fuzzy classification language (IFCL) has been
implemented and applied in experiments with 60 datasets with
categorical and numerical target variables in order to evaluate
performance improvements by application of the proposed methodology.
Seven questions (Section 1.1) have guided the research conducted for
this thesis. The following list summarizes the answers that this thesis
proposes:
o
What is the theoretical basis of inductive fuzzy classification (IFC)
and what is its relation to inductive logic?
Inductive logic is defined as “a system of evidential support that
extends deductive logic to less-than-certain inferences”
(Hawthorne, 2008, “”Inductive Logic,” paragraph 1). Fuzzy
classes are fuzzy sets (Zadeh 1965) defined by propositional
functions (Russell, 1919) with a gradual truth-value (i.e.,
mapping to fuzzy propositions; Zadeh 1975). Their membership
functions are thus inductive if their defining propositional
functions are based on less-than-certain inferences (Section 2.4).
The inductive fuzzy class y’ = { i ∈U | i is likely a member of y }
is defined by a predictive model, µ, for membership in class y,
where “likely” is a fuzzy set of hypotheses. The truth function of
the fuzzy propositional function L(i,y) := “i is likely a member of
y” is a fuzzy restriction (Zadeh, 1975a) on U defined by µy‘.
o
How can membership functions be derived inductively from data?
Membership functions can be derived using ratios or differences
of sampled conditional probabilities. Different formulas have
been proposed in Section 2.4.1 and tested in Section 4.2
Normalized differences and ratios of likelihoods have shown to
provide optimal membership degrees. For numeric variables, a
continuous membership function can be defined by piecewise
115
linear function approximation. The numeric variable is
discretized using quantiles and a linear function is approximated
to the average values and the membership degrees for every
adjacent pair of quantiles. For numerical target variables,
normalization leads to a linear fuzzification with a membership
degree to a fuzzy set that can be used to calculate likelihoods of
fuzzy hypotheses.
o
How can a business enterprise apply IFC in practice?
o
How can the proposed methods be implemented in a computer
program?
Inductive fuzzy classes can support marketing analytics by
providing selection, visualization, and prediction in the fields of
customer analytics, product analytics, fuzzy target selection, and
integrated analytics for individual marketing. These techniques
improve decision support for marketing by generating visual,
intuitive profiles and improved predictive models, which can
enhance sales (Section 3.2). A case study shows a real-world
application with a financial service provider (Section 3.2.5).
The IFCL prototype uses a database server for data storage,
access, and computation (Section 4.1.2). The analytic process is
configured in a text file using a synthetic language called IFCL.
This language allows definition of all computations necessary for
MFI, IAF, target fuzzification, attribute selection, supervised
multivariate aggregation, fuzzy classification of data using
predictive models, and model evaluation. Additionally, the IFCL
prototype allows experiments on the proposed methods for MFI
in batch mode. Furthermore, student works have shown
implementations of IFC with graphical user interfaces.
o
How is IFC optimally applied for prediction?
116
First, data is prepared and relevant attributes are selected.
Then, each attribute is inductively fuzzified, so that the
membership degree indicates the degree of membership in the
target class. Finally, those multiple membership degrees are
aggregated into a single membership degree with a multivariate
regression, which represents the inductive class prediction.
Optimally, for binary targets, attributes are fuzzified with NLRs
and aggregated using logistic regression. For numerical targets,
the target attribute is fuzzified using a linear normalization, the
dependent attributes are fuzzified using NLD and then
aggregated using regression trees (Section 3.1.3).
o
Which aggregation methods are optimal for the multivariate
combination of fuzzy classes for prediction?
In the experimental phase, product, algebraic sum, minimum,
maximum, average, logistic regression, linear regression, and
regression trees have been tested for their predictive
performance using the prototype software IFCL. For the
aggregation of fuzzified attributes into a single prediction value
for data records, the experiments showed that, on average, a
supervised aggregation using multivariate regression models
(logistic regression for Boolean targets and regression trees for
Zadehan targets) are optimal (Section 4.2.2).
o
Can it be statistically supported that the proposed method of IFC
improves predictive performance?
Inductive attribute fuzzification using NLRs significantly
improved average prediction results of classical regression
models (Section 4.2.2). In fact, this improvement significantly
and negatively correlated with the linearity of association
between the data and the target class. Thus, the less linear this
association is, demonstrated by a smaller correlation of the
predictions of a linear regression model with the actual values,
the more a multivariate model can be improved by transforming
attributes into an inductive membership degree using the
methods proposed in this thesis.
5.2
Discussion of Results
Human perception and cognition is blurred by uncertainty and
fuzziness. Concepts in our minds cluster perceptions into categories that
cannot always be sharply distinguished. In some cases, binary
classifications are feasible, such as detecting the presence or absence of
light. Nevertheless, in many cases, our classification relies on fuzzy
terms such as bright. Some concepts are gradual and can be ordered by
their degree. In our previous example, brightness is a perception that is
ordinal because some visions are brighter than others. By mapping an
ordinal scale to Zadehan truth values, a membership function can be
defined in order to clearly precisiate the semantics of fuzzy words,
concepts, or terms, in the sense of Zadeh (2008). In fact, fuzzy set theory
has not yet been recognized enough for its most valuable contribution: It
provides a tool for assigning precise definitions to fuzzy statements! In
Section 2.2.1, the sorites paradox, an age-old philosophical puzzle, was
easily resolved by application of fuzzy set theory; such is the power of
Zadehan logic.
117
Conditioned by fuzziness of cognition, the human mind infers and
reasons based on fuzzy ideas, and often, successfully so. Induction is an
uncertain type of reasoning, and thus classifies inferences into a fuzzy
set: the fuzzy class of inductively supported conclusions. On one hand,
induction is always uncertain, and even the most unlikely event can
actually happen, even if induction indicates the contrary, such as the
observation of a black swan (Taleb, 2007). On the other hand, how can
we learn if not from past experience? Good inductive inferences are
reliable more often than not. Furthermore, gaining insights by
evaluating likelihoods of hypotheses using objective data is a
cornerstone of empiricism. Thus, induction and its support measure, the
likelihood ratio, can amplify past experience by leading to better and
more reliable inductive inferences. While many concepts are fuzzy,
induction provides a tool for precisiation.
The transformation of likelihood ratios into membership functions
to fuzzy sets by normalization is a tool for reasoning about uncertainty
by amplification of past experience, where the fuzziness of class
membership is precisiated by likelihood in the data. Accordingly, MFI
with NLRs applies the law of likelihood and fuzzy set theory to make
inductive inferences and to represent them as membership functions to
fuzzy classes.
There are three main advantages of applying likelihood-based
MFI in analytics. First, a membership function makes hidden
meaningful associations between attributes in the data explicit. It
allows visualizing these associations in order to provide insight into
automated inductive inferences. Second, a two-dimensional plot of these
membership functions is easily interpretable by humans. Those
visualizations are precisiations of inductive associations. Third, these
membership functions are models that allow for a better quality of
predictions and thus more accurate decisions.
Likelihood-based precisiation of fuzziness can be applied to
marketing analytics. Data analysis on customers, products, and
transactions can be applied for customer relationship management
(CRM) and individual marketing. This kind of analytic CRM provides
decision support to company representatives in marketing channels
with direct customer contact for offers, underwriting, sales,
communications, gifts, and benefits.
For the analytics process, it is proposed to classify customers
inductively and fuzzily in order to target individual customers with the
right relationship activities. First, the customers are classified by
assigning them to different targets (classes), such as product offers or
promotional gifts. Second, this classification is done inductively by
analyzing present data as evidence for or against target membership
based on customer characteristics. Third, the assignment of individual
118
customers to targets is fuzzy if the resulting classes have a gradual
membership function. Finally, those inductive membership degrees to
fuzzy targets provide decision support by indicating the likelihood of
desired response in customer activities based on customer
characteristics. These membership functions can be used in analytics to
select the most important characteristics regarding a target, visualize
the relationship between characteristics and a target, and predict target
membership for individual customers.
The proposed method for MFI is based on calculating likelihoods
of target membership in existing data. These likelihoods are turned into
a fuzzy set membership function by comparison and normalization
using an NLR or an NLD. This algorithm has been implemented in the
Weka machine learning workbench as a supervised attribute filter,
which inductively fuzzifies sharp data and visualizes the induced
membership functions. This software can be downloaded for
experimental purposes.
Sharp binary classification and segmentation lead to anomalies of
threshold, and fuzzy classification provides a solution with gradual
distinctions that can be applied to CRM, as pointed out by Werro (2008).
Gradual customer partitioning has the advantage that the size of target
groups can be varied by choosing a threshold according to conditions
such as budget or profitability. Of course, scoring methods for CRM are
state of the art; additionally, the present approach suggests that fuzzy
logic is the appropriate tool for reasoning about CRM targets because
they are essentially fuzzy concepts. Scoring provides a means for
precisiation of this fuzziness, which provides numerical membership
degrees of customers in fuzzy target groups.
The advantages of MFI for fuzzy classification of customers are
efficiency and precision. First, membership functions to CRM targets
can be derived automatically, which is an efficient way of definition.
And second, those inferred membership functions are precise because
they indicate an objective, measurable degree of likelihood for target
membership. Thus, the semantics of the membership function is clearly
grounded.
5.3
Outlook and Further Research
The first point of further research is further development of the
IFCL software. This prototype has fulfilled its purpose in answering the
research questions of this thesis, but it lacks many features to make it
suitable for productive use. Further development of this software could
address the following points:
119
o
refactoring of the source code with a clear object oriented design
methodology,
o
adding a graphical user interface,
o
redesign of certain syntactical elements that have turned out to be
inconvenient,
o
adding a syntax checker for the IFCL language,
o
o
o
adding support for all databases with a JDBC interface,
handling null values in the data,
and allowing multiclass predictions.
As a second research direction, the field of application of IFC will
be diversified. So far, applications have been proposed and evaluated in
the area of marketing. Providing applications for web semantics
analytics and biomedical analytics is envisioned.
Third, the logical framework of fuzzy classification by induction is
intended to be generalized. “Inductive approximate reasoning” can be
defined as a methodology for the application of logic with gradual truthvalues using less-than-certain inferences. Also, the epistemological
problem of induction will be tackled, because the likelihood ratio as a
measure of support is distorted for large values of 𝑝𝑝(𝐻𝐻 ∩ 𝐸𝐸)/𝑝𝑝(𝐻𝐻), which
is exemplified by the paradox of the raven (Vickers, 2009): Is the
conclusion that a Raven (𝐸𝐸) is black (𝐻𝐻) truly supported by countless
observations of white clouds (𝐻𝐻 ∩ 𝐸𝐸)? At least, it increases the likelihood
ratio by decreasing 𝑝𝑝(𝐸𝐸|𝐻𝐻); perhaps, there might be other useful
measures of inductive support.
The most important suggestion for further research is the
investigation of multidimensional MFI. This thesis is concentrated on
induction of membership functions for single variables and their
multivariate combination. As such, not all possible combinations of all
variables are evaluated (see Section 2.2.2). In a multivariate model, a
given attribute value can be assigned the same weight for every
combination with values of other variables, for example, in a linear
regression. In a multidimensional model, the same attribute value has
different weights in different contexts, because every vector of a possible
attribute value combination is assigned an individual weight or
membership degree. Therefore, it might be interesting for future
research to investigate the properties of multidimensional MFI. Instead
of a multivariate combination of univariate membership functions, a
multidimensional membership function assigns a membership degree to
vectors of attribute value combinations, and the influence of an
attribute value is not only dependent on itself, but also on the context of
the other attribute’s values. Thus, predictive performance could be
120
improved because the interdependency of attributes can be modeled.
This could resolve the “watermelon paradox” stated by Elkan (1993, p.
703), because the inductive membership function to the concept of
watermelon is measured in two dimensions, with “red inside” as a first
coordinate and “green outside” as a second.
The following research questions can be of interest: How are
multidimensional membership functions induced from data? How are
multidimensional membership degrees computed for vectors of
categorical values, for vectors of numerical values, and for vectors of
numerical and categorical values? How are continuous functions derived
from those discrete degrees? Can multidimensional membership
functions improve prediction? In what cases is it advisable to work with
multidimensional membership functions?
A possibility is to use n-linear interpolation using n-simplexes
(Danielsson & Malmquist, 1992), which are hyper-triangles or
generalizations of triangles in multidimensional space. Using
simplexes, for every n-tuple of possible dimension value combinations, a
corresponding likelihood measure (such as an NLR) could be sampled.
Then, n different membership degree samples would represent the
edges of a simplex. This hyper-triangle would represent a piece of the
continuous multidimensional membership function, which could be
approximated by combining simplexes to piecewise linear hyper
surfaces.
121
122
6 Literature References
Backus, J. W., Bauer, F. L., Green, J., Katz, C., McCarthy, J., Perlis, A.
J., et al. (1960). Report on the algorithmic language ALGOL 60.
(P. Naur, Ed.) Communications of the Association for Computing
Machinery , 3 (5), pp. 299-314.
Bellmann, R. E., & Zadeh, L. A. (1977). Local and Fuzzy Logics. In J. M.
Dunn, & G. Epstein (Eds.), Modern Uses of Multiple-Valued
Logic (pp. 283-335). Kluwer Academic Publishers.
Boole, G. (1847). The Matematical Analysis of Logic, being an Essay
towards a Calculus of Deductive Reasoning. London: George Bell.
Carrol, L. (1916). Alice's Adventures in Wonderland. New York: Sam'l
Gabriel Sons & Company.
Chmielecki, A. (1998). What is Information? Twentieth World Congress
of Philosophy. Boston, Massachusetts USA.
Danielsson, R., & Malmquist, G. (1992). Multi-Dimensional Simplex
Interpolation - An Approach to Local Models for Prediction.
Chemometrics and Intelligent Laboratory Systems , 14, pp. 115128.
Davenport, T. H. (2006). Competing on Analytics. Harvard Business
Review .
Del Amo, A., Montero, J., & Cutello, V. (1999). On the Principles of
Fuzzy Classification. Proc. 18th North American Fuzzy
Information Processing Society Annual Conf, (pp. 675 – 679).
Dianhui, W., Dillon, T., & Chang, E. J. (2001). A Data Mining Approach
for Fuzzy Classification Rule Generation. IFSA World Congress
and 20th NAFIPS International Conference, (pp. 2960-2964).
Dubois, D. J., & Prade, H. (1980). Fuzzy Sets and Systems. Theory and
Applications. New York: Academic Press.
Elkan, C. (1993). The Paradoxical Success of Fuzzy Logic. Proceedings
of the Eleventh National Conference on Artificial Intelligence,
(pp. 698–703). Washington D.C.
123
Glubrecht, J.-M., Oberschelp, A., & Todt, G. (1983). Klassenlogik.
Mannheim: Wissenschaftsverlag.
Gluchowski, P., Gabriel, R., & Dittmar, C. (2008). Management Support
Systeme und Business Intelligence. Berlin: Springer Verlag.
Graf, C. (2010). Erweiterung des Data-Mining-Softwarepakets WEKA
um induktive unscharfe Klassifikation (Master's Thesis).
Department of Informatics, University of Fribourg, Switzerland.
Greene, B. (2011). The Hidden Reality. Parallel Universes And The
Deep Laws Of The Cosmos. New York: Random House.
Hüllermeier, E. (2005). Fuzzy Methods in Machine Learning and Data
Mining: Status and Prospects. Fuzzy Sets and Systems , pp. 387406.
Hájek, A. (2009). Interpretations of Probability. In E. N. Zalta (Ed.),
Stanford Encyclopedia of Philosophy. Stanford: The Metaphysics
Research Lab, Stanford University.
Hajek, P. (2006). Fuzzy Logic. In E. N. Zalta (Ed.), Stanford
Encyclopedia of Philosophy. Stanford: The Metaphysics Research
Lab, Stanford University.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., &
Witten, I. H. (2009). The WEKA Data Mining Software: An
Update. SIGKDD Explorations , 11 (1).
Hawthorne, J. (2008). Inductive Logic. In E. N. Zalta (Ed.), Stanford
Encyclopedia of Philosophy. Stanford: The Metaphysics Research
Lab, Stanford University.
Hu, Y., Chen, R., & Tzeng, G. (2003). Finding Fuzzy Classification Rules
using Data Mining Techniques. Pattern recognition letters , 24
(1-3), pp. 509-514.
Hyde, D. (2008). Sorites Paradox. In E. N. Zalta (Ed.), The Stanford
Encyclopedia of Philosophy. Stanford : The Metaphysics
Research Lab, Stanford University.
Inmon, W. H. (2005). Building the Data Warehouse (fourth ed.). New
York: Wiley Publishing.
124
John, E. G. (1998). Simplified Curve Fitting Using Speadsheet Add-ins.
International Journal of Engineering Education , 14 (5), pp. 375380.
Joyce, J. (2003). Bayes' Theorem. In E. N. Zalta (Ed.), Stanford
Encyclopedia of Philosophy. Stanford: The Metaphysics Research
Lab, Stanford University.
Küsters, U. (2001). Data Mining Methoden: Einordung und Überblick.
In H. Hippner, U. Küsters, M. Meyer, & K. Wilde (Eds.),
Handbuch Data Mining im Marketing - Knowledge Discovery in
Marketing Databases (pp. 95-130). Wiesbaden: vieweg GABLER.
Kaniza, G. (1976). Subjective contours. Scientific American (234), pp.
48-52.
Kaufmann, M. A. (2009). An Inductive Approach to Fuzzy Marketing
Analytics. In M. H. Hamza (Ed.), Proceedings of the 13th
IASTED international conference on Artificial Intelligence and
Soft Computing. Palma, Spain.
Kaufmann, M. A. (2008). Inductive Fuzzy Classification. Thesis
Proposal . Internal Working Paper, University of Fribourg,
Switzerland.
Kaufmann, M. A., & Graf, C. (2012). Fuzzy Target Groups in Analytic
Customer Relationship Management. In A. Meier, & L. Donzé
(Eds.), Fuzzy Methods for Customer Relationship Management
and Marketing - Applications and Classification. London: IGI
Global.
Kaufmann, M. A., & Meier, A. (2009). An Inductive Fuzzy Classification
Approach Applied To Individual Marketing. Proc. 28th North
American Fuzzy Information Processing Society Annual Conf.
Kohavi, R., Rothleder, N. J., & Simoudis, E. (2002). Emerging Trends in
Business Analytics. Communications of the ACM , 45 (8), pp. 4548.
Mayer, P. (2010). Implementation of an Inductive Fuzzy Classification
Interpreter. Master's Thesis. University of Fribourg.
Meier, A., Schindler, G., & Werro, N. (2008). Fuzzy classification on
relational databases. In M. Galindo (Ed.), Handbook of Research
125
on Fuzzy Information Processing in Databases (Vol. II, pp. 586614). London: Information Science Reference.
merriam-webster.com. (2012a). Analytics. Retrieved from Merriam
Webster Online Dictionnary: http://www.merriamwebster.com/dictionary/analytics
merriam-webster.com. (2012b). Heap. Retrieved from Merriam Webster
Online Dictionnary: http://www.merriamwebster.com/dictionary/heap
Mill, J. S. (1843). A System of Logic. London: John W. Parker, West
Strand.
Oberschelp, A. (1994). Allgemeine Mengenlehre. Mannheim:
Wissenschaftsverlag.
Oesterle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar,
H., et al. (2010). Memorandum zur gestaltungsorientierten
Wirtschaftsinformatik. Multikonferenz Wirtschaftsinformatik
(MKWI). Göttingen.
Precht, R. D. (2007). Wer bin ich - und wenn ja wie viele? Eine
philosophische Reise. München: Goldmann.
Roubos, J. A., Setnes, M., & Abonyi, J. (2003). Learning Fuzzy
Classification Rules from Labeled Data. Information Sciences—
Informatics and Computer Science: An International Journal ,
Volume 150 (Issue 1-2).
Russell, B. (1919). Introduction to Mathematical Philosophy. London:
George Allen & Unwin, Ltd.
Setnes, M., Kaymak, H., & van Nauta Lemke, H. R. (1998). Fuzzy
Target Selection in Direct Marketing. Proc. of the
IEEE/IAFE/INFORMS Conf. On Computational Intelligence for
Financial Engineering (CIFEr).
Shannon, C. (1948). A Mathematical Theory of Communication. The
Bell Systems Technical Journal (Vol. 27), pp. 379–423, 623–656.
Sorensen, R. (2008). Vagueness. In E. N. Zalta (Ed.), The Stanford
Encyclopedia of Philosophy. Stanford: The Metaphysics Research
Lab, Stanford University.
126
Sousa, J. M., Kaymak, U., & Madeira, S. (2002). A Comparative Study
of Fuzzy Target Selection Methods in Direct Marketing.
Proceedings of the 2002 IEEE International Conference on Fuzzy
Systems (FUZZ-IEEE '02).
Spais, G., & Veloutsou, C. (2005). Marketing Analytics: Managing
Incomplete Information in Consumer Markets and the
Contribution of Mathematics to the Accountability of Marketing
Decisions. Southeuropean Review of Business Finance &
Accounting (3), pp. 127-150.
Taleb, N. N. (2007). The Black Swan. The Impact of the Highly
Improbable. New York: Random House.
Turban, E., Aronson, J. E., Liang, T.-P., & Sharda, R. (2007). Decision
Support and Business Intelligence Systems. New Jersey:
Pearson Education.
Vickers, J. (2009). The Problem of Induction. In E. N. Zalta (Ed.),
Stanford Encyclopedia of Philosophy. Stanford: The Metaphysics
Research Lab, Stanford University.
Wang, L., & Mendel, J. (1992). Generating Fuzzy Rules by Learning
from Examples. IEEE Transactions on Systems, Man and
Cybernetics (Issue 6), pp. 1414 – 1427.
Weisstein, E. W. (2010a). Conditional Probability. Retrieved from
MathWorld--A Wolfram Web Resource:
http://mathworld.wolfram.com/ConditionalProbability.html
Weisstein, E. W. (2010b). Correlation Coefficient. Retrieved from
MathWorld--A Wolfram Web Resource:
http://mathworld.wolfram.com/CorrelationCoefficient.html
Weisstein, E. W. (2012b). Spearman Rank Correlation Coefficient. From
MathWorld--A Wolfram Web Resource:
http://mathworld.wolfram.com/SpearmanRankCorrelationCoeffic
ient.html
Weisstein, E. W. (2012a). Wilcoxon Signed Rank Test. From
MathWorld--A Wolfram Web Resource.:
http://mathworld.wolfram.com/WilcoxonSignedRankTest.html
Wells, H. G. (1908). First and Last Things; A Confession Of Faith And
Rule Of Life. New York: G. P. Putnam's Sons.
127
Werro, N. (2008). Fuzzy Classification of Online Customers. PhD.
Dissertation, University of Fribourg, Switzerland.
Witten, I. H., & Frank, E. (2005). Data Mining, Practical Machine
Learning Tools and Techniques (Second ed.). San Francisco:
Morgan Kaufmann Publishers.
Zadeh, L. A. (1965). Fuzzy Sets. Information and Control (8), pp. 338–
353.
Zadeh, L. A. (1975a). Calculus of Fuzzy Restrictions. In L. A. Zadeh, K.S. Fu, K. Tanaka, & M. Shimura (Eds.), Fuzzy sets and Their
Applications to Cognitive and Decision Processes. New York:
Academic Press.
Zadeh, L. A. (1975b). The Concept of a Linguistic Variable and its
Application to Approximate Reasoning. Information Sciences , I
(8), pp. 199–251.
Zadeh, L. A. (2008). Is There a Need for Fuzzy Logic. Information
Sciences , 178 (13).
Zimmermann, H. J. (1997). Fuzzy Data Analysis. In O. Kaynak, L. A.
Zadeh, B. Turksen, & I. J. Rudas (Eds.), Computational
Intelligence: Soft Computing and Fuzzy-Neuro Integration with
Applications. Berlin: Springer-Verlag.
Zimmermann, H. J. (2000). Practical Applications of Fuzzy
Technologies. Berlin: Springer-Verlag.
Zimmermann, H. J., & Zisno, P. (1980). Latent Connectives in Human
Decision Making. Fuzzy Sets and Systems , 37-51.
128
A. Appendix
A.1. Datasets used in the Experiments
Table 9
Data Sets with Categorical Target Variable, Transformed into a Binary Target
Name
Target
Filter
abalone
rings <= 9
No
adult
class = ' >50K'
No
anneal
class <> '3'
yes
balance
class = 'L'
No
bands
band_type = 'band' Yes
breastw
car
class = 'benign'
acceptable <>
'unacc'
cmc
contraceptive_met
hod <> '1'
No
credit
a16 = '+'
No
creditg
class = 'good'
No
No
No
22
Records Link (accessed 08.2010)
http://archive.ics.uci.edu/ml/d
2743 atasets/Abalone
http://archive.ics.uci.edu/ml/d
21623 atasets/Adult
http://archive.ics.uci.edu/ml/d
597 atasets/Annealing
http://archive.ics.uci.edu/ml/d
404 atasets/Balance+Scale
http://archive.ics.uci.edu/ml/d
237 atasets/Cylinder+Bands
http://archive.ics.uci.edu/ml/d
atasets/Breast+Cancer+Wisc
437 onsin+(Diagnostic)
http://archive.ics.uci.edu/ml/d
1145 atasets/Car+Evaluation
http://archive.ics.uci.edu/ml/d
atasets/Contraceptive+Metho
1002 d+Choice
http://archive.ics.uci.edu/ml/d
431 atasets/Credit+Approval
http://archive.ics.uci.edu/ml/d
atasets/Statlog+(German+Cr
679 edit+Data)
http://archive.ics.uci.edu/ml/d
atasets/Pima+Indians+Diabe
489 tes
http://archive.ics.uci.edu/ml/d
87 atasets/Acute+Inflammation
glass
class <>
'tested_positive'
inflammation =
'yes'
type in ( '''build
wind float''',
'''build wind nonfloat''', '''vehic
wind float''',
'''vehic wind nonfloat''')
heart
y= '2'
No
hepatitis
class = 'LIVE'
No
http://archive.ics.uci.edu/ml/d
157 atasets/Glass+Identification
http://archive.ics.uci.edu/ml/d
188 atasets/Statlog+(Heart)
http://archive.ics.uci.edu/ml/d
59 atasets/Hepatitis
horse
outcome = '1'
Yes
124 http://archive.ics.uci.edu/ml/d
diabetes
diagnosis
22
No
No
No
See Section A.1.1 and A.1.2
129
atasets/Horse+Colic
hypothyroid
class <> 'negative'
Yes
internetads
y=1
No
ionosphere
class = 'g'
No
iris
class = 'Iris-setosa' No
letter
No
lymph
class = 'F'
class =
'metastases'
segment
class = 'sky'
No
sonar
class = 'Mine'
No
spectf
y=1
Yes
vehicle
class = 'saab'
No
waveform
class <> 1
No
wine
class = 2
No
wisconsin
diagnosis = 'B'
Yes
yeast
cls = 'CYT'
Yes
130
No
http://archive.ics.uci.edu/ml/d
1800 atasets/Thyroid+Disease
http://archive.ics.uci.edu/ml/d
atasets/Internet+Advertisem
2153 ents
http://archive.ics.uci.edu/ml/d
230 atasets/Ionosphere
http://archive.ics.uci.edu/ml/d
99 atasets/Iris
http://archive.ics.uci.edu/ml/d
13221 atasets/Letter+Recognition
http://archive.ics.uci.edu/ml/d
99 atasets/Lymphography
http://archive.ics.uci.edu/ml/d
atasets/Statlog+(Image+Seg
1490 mentation)
http://archive.ics.uci.edu/ml/d
atasets/Connectionist+Bench
143 +(Sonar,+Mines+vs.+Rocks)
http://archive.ics.uci.edu/ml/d
177 atasets/SPECTF+Heart
http://archive.ics.uci.edu/ml/d
atasets/Statlog+(Vehicle+Sil
552 houettes)
http://archive.ics.uci.edu/ml/d
atasets/Waveform+Database
3342 +Generator+(Version+2)
http://archive.ics.uci.edu/ml/d
110 atasets/Wine
http://archive.ics.uci.edu/ml/d
atasets/Breast+Cancer+Wisc
391 onsin+(Diagnostic)
http://archive.ics.uci.edu/ml/d
973 atasets/Yeast
Table 10
Data Sets with Numerical Target Variable, Transformed into a Fuzzy Proposition µ↑
(see Formula 43 and Formula 44)
Name
Target
Filter
auto93
µ (price)
Yes
autohorse
No
baskball
µ (horsepower)
µ
(points_per_minut
e)
bodyfat
µ (bodyfat)
No
bolts
µ (t20bolt)
No
breasttumor
µ (tumor_size)
No
cholestrol
µ (chol)
No
cloud
µ-(te)
No
µ
(violentcrimesperp
op)
Yes
↑
↑
↑−
↑
↑
↑
↑
No
↑−
communities
No
elusage
µ (prp)
µ (average_electric
ity_usage)
fishcatch
µ (wheight)
No
forestfires
µ (area)
No
fruitfly
µ (longevity)
No
housing
µ (price)
No
lowbwt
µ (bwt)
No
cpu
23
↑
↑
↑
↑
↑
↑
↑
No
23
Records
Link (accessed 08.2010)
http://www.amstat.org/public
ations/jse/v1n1/datasets.lock.
138 html
http://archive.ics.uci.edu/ml/d
65 atasets/Automobile
http://lib.stat.cmu.edu/datase
62 ts/smoothmeth
http://lib.stat.cmu.edu/datase
163 ts/bodyfat
http://lib.stat.cmu.edu/datase
30 ts/bolts
http://archive.ics.uci.edu/ml/
machine-learning181 databases/breast-cancer/
http://archive.ics.uci.edu/ml/d
203 atasets/Heart+Disease
http://lib.stat.cmu.edu/datase
68 ts/cloud
http://archive.ics.uci.edu/ml/d
atasets/Communities+and+C
1363 rime
http://archive.ics.uci.edu/ml/
machine-learning135 databases/cpu-performance/
http://lib.stat.cmu.edu/datase
35 ts/smoothmeth
http://www.amstat.org/public
ations/jse/datasets/fishcatch.
dat
http://www.amstat.org/public
ations/jse/datasets/fishcatch.t
97 xt
http://archive.ics.uci.edu/ml/d
346 atasets/Forest+Fires
http://www.amstat.org/public
ations/jse/datasets/fruitfly.da
t
http://www.amstat.org/public
79 ations/jse/datasets/fruitfly.txt
http://archive.ics.uci.edu/ml/d
348 atasets/Housing
http://www.umass.edu/statda
ta/statdata/data/lowbwt.txt
124 http://www.umass.edu/statda
See Section A.1.1 and A.1.2
131
ta/statdata/data/lowbwt.dat
meta
µ (error_rate)
No
parkinsons
µ (motor_UPDRS)
Yes
pbc
µ (survival_time)
Yes
pharynx
µ (days_survived)
Yes
pollution
µ (mort)
No
quake
µ (richter)
No
sensory
No
servo
µ (score)
µ (servo_rise_time
)
sleep
µ (total_sleep)
No
slump
µ (slump_cm)
Yes
strike
µ (volume)
No
veteran
µ(survival)
No
wineqred
µ (quality)
No
wineqwhite
µ (quality)
No
132
↑
↑
↑
↑
↑
↑
↑
↑
↑
↑
↑
↑
↑
No
http://archive.ics.uci.edu/ml/d
348 atasets/Meta-data
http://archive.ics.uci.edu/ml/d
1172 atasets/Parkinsons
http://lib.stat.cmu.edu/datase
114 ts/pbc
http://www.umass.edu/statda
ta/statdata/data/pharynx.txt
http://www.umass.edu/statda
117 ta/statdata/data/pharynx.dat
http://lib.stat.cmu.edu/datase
39 ts/pollution
http://lib.stat.cmu.edu/datase
1463 ts/smoothmeth
http://lib.stat.cmu.edu/datase
391 ts/sensory
http://archive.ics.uci.edu/ml/d
108 atasets/Servo
http://lib.stat.cmu.edu/datase
41 ts/sleep
http://archive.ics.uci.edu/ml/d
atasets/Concrete+Slump+Tes
75 t
http://lib.stat.cmu.edu/datase
416 ts/strikes
http://lib.stat.cmu.edu/datase
88 ts/veteran
http://archive.ics.uci.edu/ml/d
1058 atasets/Wine+Quality
http://archive.ics.uci.edu/ml/d
3247 atasets/Wine+Quality
A.1.1.
Attribute Selection Filters
Data set
Selected attributes in prediction experiments
anneal
surface_quality, family, ferro, thick, formability, chrom, condition,
steel, non_ageing, phos, surface_finish, bf, enamelability, width,
blue_bright_varn_clean, strength, carbon, shape, cbond, bw_me,
exptl, lustre, bl
Type,
City_MPG,
Highway_MPG,
Air_Bags_standard,
Drive_train_type,
Number_of_cylinders,
Engine_size,
RPM,
Engine_revolutions_per_mile,
Manual_transmission_available,
Fuel_tank_capacity, Passenger_capacity, Length, Wheelbase,
Width, U_turn_space, Rear_seat_room, Luggage_capacity, Weight,
Domestic
grain_screened,
ink_color,
proof_on_ctd_ink,
blade_mfg,
cylinder_division , paper_type, ink_type, direct_steam, solvent_type,
type_on_cylinder , press_type, press , unit_number , cylinder_size,
paper_mill_location, plating_tank, proof_cut, viscosity, caliper,
ink_temperature, humifity, roughness, blade_pressure, varnish_pct,
press_speed, ink_pct, solvent_pct, ESA_Voltage, ESA_Amperage ,
wax,
hardener,
roller_durometer,
current_density_num
,
anode_space_ratio, chrome_content
pctilleg,
pctkids2par,
pctfam2par,
racepctwhite,
numilleg,
pctyoungkids2par,
pctteen2par,
racepctblack,
pctwinvinc,
pctwpubasst,
numunderpov,
pctpopunderpov,
femalepctdiv,
pctpersdensehous, totalpctdiv, pctpersownoccup, pctunemployed,
malepctdivorce, pcthousnophone, pctvacantboarded, pctnothsgrad,
medfaminc,
pcthousownocc,
housvacant,
pcthousless3br,
pctless9thgrade,
medincome,
blackpercap,
pctlarghousefam,
numinshelters, numstreet, percapinc, agepct16t24, population,
numburban, pctwofullplumb, pcthousoccup, lemaspctofficdrugun,
pctemploy,
mednumbr,
pctoccupmgmtprof,
pctimmigrec10,
numimmig, pctbsormore, medrentpcthousinc, pctoccupmanu,
pctimmigrec8,
pctwwage,
agepct12t29,
pctlarghouseoccup,
hisppercap, popdens, pctimmigrecent, pctimmigrec5
Age, surgery, mucous_membranes, to_number(pulse) as pulse,
capillary_refill_time,
to_number(packed_cell_volume)
as
packed_cell_volume,
peristalsis,
pain,
abdominal_distension,
temperature_of_extremities, to_number(rectal_temperature)
on_thyroxine, query_on_thyroxine, on_antithyroid_medication, sick,
pregnant, thyroid_surgery, I131_treatment, query_hypothyroid,
query_hyperthyroid, lithium, goitre, tumor, hypopituitary, psych,
to_number(replace(TSH, '?', null)) as TSH, to_number(replace(T3,
'?', null)) as T3, to_number(replace(TT4, '?', null)) as TT4,
to_number(replace(T4U, '?', null)) as T4U, to_number(replace(FTI,
'?', null)) as FTI, referral_source
auto93
bands
communities
horse
hypothyroid
parkinsons
subject_no, age, sex, test_time, Jitter_prc, Jitter_abs, Jitter_rap,
Jitter_PPQ5, Jitter_DDP, Shimmer, Shimmer_dB, Shimmer_APQ3,
Shimmer_APQ5, Shimmer_APQ11, Shimmer_DDA, NHR, HNR,
RPDE, DFA, PPE
133
pbc
pharynx
Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8, Z9, Z10, Z11, Z12, Z13, Z14, Z15, Z16,
Z17
Inst, sex, Treatment, Grade, Age, Condition, Site, T, N, Status
slump
Cement, Slag, Fly_ash, Water, SP, Coarse_Aggr, Fine_Agg
spectf
f12s, f22r, f10s, f9s, f16s, f3s, f5s, f8s, f5r, f19s, f2s, f13s, f15s, f20s
wisconsin
radius, texture, perimeter, area, smoothness, compactness,
concavity, concave_points, symmetry, fractal_dimension
yeast
mcg, gvh, alm, mit, erl, pox, vac, nuc
A.1.2.
Record Selection Filters
Data set
Selection criteria
horse
surgery <> '?' and mucous_membranes <> '?' and pulse <> '?' and
capillary_refill_time <> '?' and packed_cell_volume <> '?' and
peristalsis <> '?' and pain <> '?' and abdominal_distension <> '?'
and temperature_of_extremities <> '?' and rectal_temperature <> '?'
hypothyroid
TSH <> '?' and T3 <> '?' and TT4 <> '?' and T4U <> '?' and FTI <> '?'
134
A.2. Database Queries
A.2.1.
Database Queries for Experiments with
Sharp Target Variables
Query 1: View for evaluation of experiment base data
create table evaluation_basis3 as
select
dataset, method_tmp as aggregation, ifc,
holdout_no,
correlation,
auc,mae,rmse,
c_s, c_s_sig, c_k, c_k_sig
from
(
select
ifcl_file,dataset,case when ifc is null then 'noifc' else ifc
end as ifc,
correlation,
percent_rank() over(partition by dataset order by correlation)
as r,
--correlation as r,
auc,mae,rmse,
replace(
replace(regexp_replace(replace(ifcl_file,
'./metainduction/'||dataset||'/'||ifc||'/', ''), '[[:digit:]]+_',
''),'.ifcl')
,
'./metainduction/'||dataset||'/', '') as method_tmp,
holdout_no,
c_s, c_s_sig, c_k, c_k_sig
from
(
select
ifcl_file,dataset,correlation,auc,mae,rmse,
replace(replace(ifc_method_tmp,
'./metainduction/'||dataset||'/', ''), '/', '') as ifc,
holdout_no, c_s, c_s_sig, c_k, c_k_sig
from
135
(
SELECT
ifcl_file,correlation,auc,mae,rmse,
replace(replace(REGEXP_SUBSTR(ifcl_file,
'./metainduction/[[:alnum:]]+/'),
'./metainduction/', ''), '/', '') as dataset,
REGEXP_SUBSTR(ifcl_file,
'./metainduction/([[:alnum:]]+)/([[:alpha:]])+/')
as
ifc_method_tmp,
holdout_no, c_s, c_s_sig, c_k, c_k_sig
from evaluations_repeated
)
)
)
where method_tmp not like '%numeric%'
and method_tmp not like '%ct%'
create index ix1 on evaluation_basis3(dataset)
create index ix2 on evaluation_basis3(ifc)
Query 2: Average correlation per aggregation method
select aggregation, avg(correlation)
from evaluation_basis3
where ifc <> 'noifc'
group by aggregation
order by avg(correlation) desc
Query 3: Test of significance in difference between logistic regression,
linear regression and regression tree as aggregation methods
select stats_wsr_test(corr_logreg, corr_regtree, 'ONE_SIDED_SIG'),
stats_wsr_test(corr_logreg, corr_linreg, 'ONE_SIDED_SIG')
from
(
select
dataset,
136
sum(case when aggregation = 'logreg' then correlation else 0
end) /
sum(case when aggregation = 'logreg' then 1 else 0 end) as
corr_logreg,
sum(case when aggregation = 'regtree' then correlation else 0
end) /
sum(case when aggregation = 'regtree' then 1 else 0 end) as
corr_regtree,
sum(case when aggregation = 'linreg' then correlation else 0
end) /
sum(case when aggregation = 'linreg' then 1 else 0 end) as
corr_linreg
from evaluation_basis3
where ifc <> 'noifc'
group by dataset
)
Query 4: Average correlation per induction and aggregation method
select
ifc, aggregation, avg(correlation)
from evaluation_basis3
group by ifc, aggregation
order by avg(correlation) desc
Query 5: Test of significance of difference between logistic regression
using NLR-IAF, NLD-IAF and regression trees without IAF
select
stats_wsr_test(corr_logreg_nlr, corr_regtree_no_iaf,
'ONE_SIDED_SIG'),
stats_wsr_test(corr_logreg_nlr, corr_logreg_nld, 'ONE_SIDED_SIG')
from
(
select
dataset,
sum(case when ifc||'.'||aggregation = 'nlr.logreg' then
correlation
137
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nlr.logreg' then 1 else 0
end)
as corr_logreg_nlr,
sum(case when ifc||'.'||aggregation = 'nld.logreg' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nld.logreg' then 1 else 0
end)
as corr_logreg_nld,
sum(case when ifc||'.'||aggregation = 'noifc.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'noifc.regtree' then 1
else 0 end)
as corr_regtree_no_iaf
from evaluation_basis3
where dataset is not null
group by dataset )
A.2.2.
Database Queries for Experiments with
Numerical Target Variables
Query 6: View for evaluation of experiment base data
create table evaluation_basis_num as
select
dataset, method_tmp as aggregation, ifc, fuzzification,
holdout_no,
correlation,
auc,mae,rmse,
c_s, c_s_sig, c_k, c_k_sig
from
(
select
ifcl_file,dataset,
nvl(replace(substr(ifc, 1, regexp_instr(ifc, '._l|_p')),'_',
''), 'noifc') as ifc,
138
replace(substr(ifc, regexp_instr(ifc, '._l|_p')+1, length(ifc)),
'_', '') as fuzzification,
correlation,
percent_rank() over(partition by dataset order by correlation)
as r,
--correlation as r,
auc,mae,rmse,
replace(
replace(regexp_replace(replace(ifcl_file,
'./metainduction_num/'||dataset||'/'||ifc||'/', ''),
'[[:digit:]]+_', ''),'.ifcl')
,
'./metainduction_num/'||dataset||'/', '') as method_tmp,
holdout_no,
c_s, c_s_sig, c_k, c_k_sig
from
(
select
ifcl_file,dataset,correlation,auc,mae,rmse,
replace(replace(ifc_method_tmp,
'./metainduction_num/'||dataset||'/', ''), '/', '') as ifc,
holdout_no, c_s, c_s_sig, c_k, c_k_sig
from
(
SELECT
ifcl_file,correlation,auc,mae,rmse,
replace(replace(REGEXP_SUBSTR(ifcl_file,
'./metainduction_num/[[:alnum:]]+/'),
'./metainduction_num/', ''), '/', '') as dataset,
REGEXP_SUBSTR(ifcl_file,
'./metainduction_num/([[:alnum:]]+)/([[:print:]])+/')
as
ifc_method_tmp,
holdout_no, c_s, c_s_sig, c_k, c_k_sig
from evaluations_repeated_num
)
)
)
create index ix3 on evaluation_basis_num(dataset)
139
create index ix4 on evaluation_basis_num(ifc)
create index ix5 on evaluation_basis_num(fuzzification)
Query 7: Average correlation per aggregation method, given linear ITF
select aggregation, avg(correlation)
from evaluation_basis_num
where ifc <> 'noifc'
group by aggregation
order by avg(correlation) desc
Query 8: Test of significance in difference between regression tree,
linear regression and average
select
stats_wsr_test(corr_regtree, corr_avg, 'ONE_SIDED_SIG'),
stats_wsr_test(corr_regtree, corr_linreg, 'ONE_SIDED_SIG')
from
(
select
dataset,
sum(case when aggregation = 'avg' then correlation else 0
end) /
sum(case when aggregation = 'avg' then 1 else 0 end) as
corr_avg,
sum(case when aggregation = 'regtree' then correlation else 0
end) /
sum(case when aggregation = 'regtree' then 1 else 0 end) as
corr_regtree,
sum(case when aggregation = 'linreg' then correlation else 0
end) /
sum(case when aggregation = 'linreg' then 1 else 0 end) as
corr_linreg
from evaluation_basis_num
where ifc <> 'noifc'
group by dataset
)
140
Query 9: Average correlation per target fuzzification method
select
fuzzification, avg(correlation)
from evaluation_basis_num
where aggregation = 'regtree'
group by fuzzification
order by avg(correlation) desc
Query 10: Test of significance between target fuzzification methods
select
stats_wsr_test(corr_l, corr_p, 'ONE_SIDED_SIG')
from
(
select
dataset,
sum(case when fuzzification = 'l' then correlation else 0
end) /
sum(case when fuzzification = 'l' then 1 else 0 end) as
corr_l,
sum(case when fuzzification = 'p' then correlation else 0
end) /
sum(case when fuzzification = 'p' then 1 else 0 end) as
corr_p
from evaluation_basis_num
where aggregation = 'regtree'
and ifc <> 'noifc'
group by dataset
)
Query 11: Average correlation per induction and aggregation method
(given linear target fuzzification)
select
ifc, aggregation, avg(correlation)
from evaluation_basis_num
where (fuzzification = 'l' or fuzzification is null)
and aggregation <> 'logreg_p'
group by ifc, aggregation
141
order by avg(correlation) desc
Query 12: Test of significance of difference between regression trees
using NLD-IAF , NLDU-IAF and regression trees without IAF
select
stats_wsr_test(corr_regtree_nld, corr_regtree_no_iaf,
'ONE_SIDED_SIG'),
stats_wsr_test(corr_regtree_nld, corr_regtree_nldu,
'ONE_SIDED_SIG'),
stats_wsr_test(corr_regtree_nld, corr_regtree_nlr,
'ONE_SIDED_SIG')
from
(
select
dataset,
sum(case when ifc||'.'||aggregation = 'nld.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nld.regtree' then 1 else
0 end)
as corr_regtree_nld,
sum(case when ifc||'.'||aggregation = 'nldu.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nldu.regtree' then 1 else
0 end)
as corr_regtree_nldu,
sum(case when ifc||'.'||aggregation = 'npr.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'npr.regtree' then 1 else
0 end)
as corr_regtree_nlr,
sum(case when ifc||'.'||aggregation = 'noifc.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'noifc.regtree' then 1
else 0 end)
142
as corr_regtree_no_iaf
from evaluation_basis_num
where dataset is not null
group by dataset
)
A.2.3.
Database Queries for Both Types of
Experiments
Query 13: Table with data parameters for each dataset
-- Execute 00_load_all.bat first
create or replace view data_params as
select * from
( -- improvement
select dataset, (corr_logreg_nlrcorr_logreg_no_iaf)/corr_logreg_no_iaf as improvement_by_iaf
from
(
select
dataset,
sum(case when ifc||'.'||aggregation = 'nlr.logreg' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nlr.logreg' then 1
else 0 end)
as corr_logreg_nlr,
sum(case when ifc||'.'||aggregation = 'noifc.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'noifc.regtree' then 1
else 0 end)
as corr_regtree_no_iaf,
sum(case when ifc||'.'||aggregation = 'noifc.logreg' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'noifc.logreg' then 1
else 0 end)
as corr_logreg_no_iaf
143
from evaluation_basis3
where dataset is not null
group by dataset
)
union
select
dataset, (corr_regtree_nld corr_regtree_no_iaf)/corr_regtree_no_iaf as improvement_by_iaf
from
(
select
dataset,
sum(case when ifc||'.'||aggregation = 'nld.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'nld.regtree' then 1
else 0 end)
as corr_regtree_nld,
sum(case when ifc||'.'||aggregation = 'noifc.regtree' then
correlation
else 0 end)
/ sum(case when ifc||'.'||aggregation = 'noifc.regtree' then 1
else 0 end)
as corr_regtree_no_iaf
from evaluation_basis_num
where dataset is not null
group by dataset
)
)
join
( -- column count
select
replace(lower(table_name), '_tr', '') as dataset,
sum(case when data_type in ('CHAR', 'VARCHAR2') then 1 else 0
end) as ncatcols,
sum(case when data_type in ('NUMBER', 'FLOAT') then 1 else 0
end) as nnumcols,
144
sum(case when data_type in ('CHAR', 'VARCHAR2') then 1 else 0
end)/count(*) as pcatcols,
sum(case when data_type in ('NUMBER', 'FLOAT') then 1 else 0
end)/count(*) as pnumcols,
count(*) as ncols
from
sys.all_tab_columns
where table_name in
('ABALONE_TR', 'ADULT_TR', 'ANNEAL_TR', 'BALANCE_TR',
'BANDS_TR', 'BREASTW_TR', 'CAR_TR', 'CMC_TR', 'CREDIT_TR',
'CREDITG_TR', 'DIABETES_TR', 'DIAGNOSIS_TR', 'GLASS_TR',
'HEART_TR', 'HEPATITIS_TR', 'HORSE_TR', 'HYPOTHYROID_TR',
'INTERNETADS_TR', 'IONOSPHERE_TR', 'IRIS_TR', 'LETTER_TR',
'LYMPH_TR', 'SEGMENT_TR', 'SONAR_TR', 'SPECTF_TR', 'VEHICLE_TR',
'WAVEFORM_TR', 'WINE_TR', 'WISCONSIN_TR', 'YEAST_TR', 'AUTO93_TR',
'AUTOHORSE_TR', 'BASKBALL_TR', 'BODYFAT_TR', 'BOLTS_TR',
'BREASTTUMOR_TR', 'CHOLESTROL_TR', 'CLOUD_TR', 'COMMUNITIES_TR',
'CPU_TR', 'ELUSAGE_TR', 'FISHCATCH_TR', 'FORESTFIRES_TR',
'FRUITFLY_TR', 'HOUSING_TR', 'LOWBWT_TR', 'META_TR',
'PARKINSONS_TR', 'PBC_TR', 'PHARYNX_TR', 'POLLUTION_TR',
'QUAKE_TR', 'SENSORY_TR', 'SERVO_TR', 'SLEEP_TR', 'SLUMP_TR',
'STRIKE_TR', 'VETERAN_TR', 'WINEQRED_TR', 'WINEQWHITE_TR')
and column_name <> 'IS_TRAINING'
and column_name <> 'Y'
group by table_name
)
using (dataset)
join
( -- correlation of linear regression
select dataset, avg(correlation) as avg_corr_linreg from
evaluation_basis3
where ifc = 'noifc'
and aggregation = 'linreg'
group by dataset
union
select dataset, avg(correlation) from evaluation_basis_num
where ifc = 'noifc'
and aggregation = 'linreg'
145
group by dataset
)
using (dataset) join
( -- row count
select 'abalone' as dataset, 'c' as target_type, count(*) as
nrows from abalone_tr
union select 'abalone', 'c', count(*) from abalone_tr
union select 'adult', 'c', count(*) from adult_tr
union select 'anneal', 'c', count(*) from anneal_tr
union select 'balance', 'c', count(*) from balance_tr
union select 'bands', 'c', count(*) from bands_tr
union select 'breastw', 'c', count(*) from breastw_tr
union select 'car', 'c', count(*) from car_tr
union select 'cmc', 'c', count(*) from cmc_tr
union select 'credit', 'c', count(*) from credit_tr
union select 'creditg', 'c', count(*) from creditg_tr
union select 'diabetes', 'c', count(*) from diabetes_tr
union select 'diagnosis', 'c', count(*) from diagnosis_tr
union select 'glass', 'c', count(*) from glass_tr
union select 'heart', 'c', count(*) from heart_tr
union select 'hepatitis', 'c', count(*) from hepatitis_tr
union select 'horse', 'c', count(*) from horse_tr
union select 'hypothyroid', 'c', count(*) from hypothyroid_tr
union select 'internetads', 'c', count(*) from internetads_tr
union select 'ionosphere', 'c', count(*) from ionosphere_tr
union select 'iris', 'c', count(*) from iris_tr
union select 'letter', 'c', count(*) from letter_tr
union select 'lymph', 'c', count(*) from lymph_tr
union select 'segment', 'c', count(*) from segment_tr
union select 'sonar', 'c', count(*) from sonar_tr
union select 'spectf', 'c', count(*) from spectf_tr
union select 'vehicle', 'c', count(*) from vehicle_tr
union select 'waveform', 'c', count(*) from waveform_tr
union select 'wine', 'c', count(*) from wine_tr
union select 'wisconsin', 'c', count(*) from wisconsin_tr
union select 'yeast', 'c', count(*) from yeast_tr
union select 'auto93', 'n', count(*) from auto93_tr
union select 'autohorse', 'n', count(*) from autohorse_tr
146
union select 'baskball', 'n', count(*) from baskball_tr
union select 'bodyfat', 'n', count(*) from bodyfat_tr
union select 'bolts', 'n', count(*) from bolts_tr
union select 'breasttumor', 'n', count(*) from breasttumor_tr
union select 'cholestrol', 'n', count(*) from cholestrol_tr
union select 'cloud', 'n', count(*) from cloud_tr
union select 'communities', 'n', count(*) from communities_tr
union select 'cpu', 'n', count(*) from cpu_tr
union select 'elusage', 'n', count(*) from elusage_tr
union select 'fishcatch', 'n', count(*) from fishcatch_tr
union select 'forestfires', 'n', count(*) from forestfires_tr
union select 'fruitfly', 'n', count(*) from fruitfly_tr
union select 'housing', 'n', count(*) from housing_tr
union select 'lowbwt', 'n', count(*) from lowbwt_tr
union select 'meta', 'n', count(*) from meta_tr
union select 'parkinsons', 'n', count(*) from parkinsons_tr
union select 'pbc', 'n', count(*) from pbc_tr
union select 'pharynx', 'n', count(*) from pharynx_tr
union select 'pollution', 'n', count(*) from pollution_tr
union select 'quake', 'n', count(*) from quake_tr
union select 'sensory', 'n', count(*) from sensory_tr
union select 'servo', 'n', count(*) from servo_tr
union select 'sleep', 'n', count(*) from sleep_tr
union select 'slump', 'n', count(*) from slump_tr
union select 'strike', 'n', count(*) from strike_tr
union select 'veteran', 'n', count(*) from veteran_tr
union select 'wineqred', 'n', count(*) from wineqred_tr
union select 'wineqwhite', 'n', count(*) from wineqwhite_tr
)
using (dataset) Query 14: Correlation of parameters with the performance improvement
by IFC
select
target_type,
CORR_S(improvement_by_iaf, ncatcols) c_ncatcols,
CORR_S(improvement_by_iaf, ncatcols, 'TWO_SIDED_SIG') p_ncatcols,
CORR_S(improvement_by_iaf, nnumcols) c_nnumcols,
147
CORR_S(improvement_by_iaf, nnumcols, 'TWO_SIDED_SIG') p_nnumcols,
CORR_S(improvement_by_iaf, pcatcols) c_pcatcols,
CORR_S(improvement_by_iaf, pcatcols, 'TWO_SIDED_SIG') p_pcatcols,
CORR_S(improvement_by_iaf, pnumcols) c_pnumcols,
CORR_S(improvement_by_iaf, pnumcols, 'TWO_SIDED_SIG') p_pnumcols,
CORR_S(improvement_by_iaf, ncols) c_ncols,
CORR_S(improvement_by_iaf, ncols, 'TWO_SIDED_SIG') p_ncols,
CORR_S(improvement_by_iaf, avg_corr_linreg) c_avg_corr_linreg,
CORR_S(improvement_by_iaf, avg_corr_linreg, 'TWO_SIDED_SIG')
p_avg_corr_linreg,
CORR_S(improvement_by_iaf, nrows) c_nrows,
CORR_S(improvement_by_iaf, nrows, 'TWO_SIDED_SIG') p_nrows,
CORR_S(improvement_by_iaf, nrows/ncols) c_nrowscols,
CORR_S(improvement_by_iaf, nrows/ncols, 'TWO_SIDED_SIG')
p_nrowscols
from data_params
group by target_type
Query 15: Table for scatter plot visualization of association between
linearity of data and improvement by IAF
select dataset, target_type, avg_corr_linreg, improvement_by_iaf
from data_params order by target_type, dataset
148
A.3. Experimental Results Tables
Table 11
Average Prediction Correlations per Dataset/Aggregation Method for Binary Targets
Dataset
Avg. Corr. Log. Reg.
Avg. Corr. Reg. Tree
Avg. Corr. Lin. Reg.
abalone
0.638687626
0.651733317
0.636539418
adult
0.659169773
0.651350849
0.636197273
anneal
0.823367865
0.846689248
0.797382729
balance
0.920732292
0.791055945
0.840604813
bands
0.454571882
0.351967704
0.426261177
breastw
0.939085094
0.927238012
0.931453636
car
0.914349826
0.933901714
0.811581886
cmc
0.425228906
0.421732164
0.418979581
credit
0.76498662
0.757839955
0.767073216
creditg
0.457163162
0.426897568
0.437699783
diabetes
0.54510302
0.533295481
0.546568895
diagnosis
1
0.868887353
0.965967697
glass
0.794456151
0.774937317
0.802028743
heart
0.695053211
0.643374806
0.684389592
hepatitis
0.291646106
0.383819768
0.388031234
horse
0.572088107
0.529928212
0.555148535
hypothyroid
0.792125757
0.892843929
0.79338345
internetads
0.805987731
0.753679747
0.78294501
ionosphere
0.819557055
0.799538551
0.798060482
iris
1
0.999193202
0.991252264
letter
0.626368894
0.798441855
0.480644051
lymph
0.695768921
0.672304979
0.726334152
segment
0.99999997
0.997855955
0.992554443
sonar
0.558775153
0.554680801
0.578090696
spectf
0.501761759
0.447468223
0.463867849
vehicle
0.475892051
0.460322532
0.437534698
waveform
0.82107583
0.785686835
0.74716555
wine
0.918745813
0.861574492
0.914019581
wisconsin
0.908323537
0.872676408
0.88191778
yeast
0.421658925
0.424627552
0.423187415
149
Table 12
Average Prediction Correlations per Dataset and Aggregation Method for Numerical
Target Variables
Dataset
Avg. Corr. Avg.
Avg. Corr. Reg. Tree
Avg. Corr. Lin. Reg.
auto93
0.691201303
0.857445115
0.738746817
autohorse
0.892653343
0.926796658
0.930059659
baskball
0.536398611
0.638183226
0.539943795
bodyfat
0.744776961
0.980938537
0.977226072
bolts
0.79763048
0.894265558
0.905782285
breasttumor
0.231971819
0.380396489
0.258283305
cholestrol
0.176404133
0.345649097
0.163348437
cloud
0.8020314
0.891982414
0.865600118
communities
0.731280609
0.79667882
0.780258252
cpu
0.788972056
0.814152614
0.794450568
elusage
0.865618561
0.897333607
0.872277672
fishcatch
0.845272306
0.918434123
0.894956413
forestfires
0.018454772
0.167736369
0.014733488
fruitfly
-0.131668697
0.165425495
-0.067547186
housing
0.687542408
0.908550517
0.86615543
lowbwt
0.741678073
0.810166446
0.788121932
meta
0.27658433
0.448502931
0.353513754
parkinsons
0.326735958
0.881670195
0.605425938
pbc
0.475841958
0.67048775
0.487261393
pharynx
0.606093213
0.775843245
0.683939249
pollution
0.656915329
0.811606345
0.655832652
quake
0.057471089
0.081629626
0.058700741
sensory
0.290356213
0.400912946
0.337693928
servo
0.703894008
0.828999856
0.780728183
sleep
0.713524339
0.753362622
0.601491837
slump
0.422059811
0.668746482
0.500413959
strike
0.382845386
0.485543157
0.418259534
veteran
0.358292594
0.483294002
0.413181002
wineqred
0.544798097
0.615254318
0.595187199
wineqwhite
0.507972143
0.577186397
0.543279329
150
Table 13:
Average Prediction Correlations for IAF on Data with Numerical Target Variables using
Linear versus Percentile ITF
Dataset
Linear ITF
Percentile ITF
auto93
0.88105593
0.833834299
autohorse
0.946780795
0.90681252
baskball
0.635879956
0.640486496
bodyfat
0.984034462
0.977842612
bolts
0.921790457
0.866740659
breasttumor
0.385000153
0.375792824
cholestrol
0.365144244
0.32615395
cloud
0.926867976
0.857096852
communities
0.82334782
0.770009819
cpu
0.937630747
0.690674481
elusage
0.905249904
0.88941731
fishcatch
0.94873926
0.888128986
forestfires
0.210580299
0.124892439
fruitfly
0.157932749
0.172918241
housing
0.940547453
0.87655358
lowbwt
0.814805887
0.805527006
meta
0.577117175
0.319888686
parkinsons
0.880604806
0.882735584
pbc
0.666064677
0.674910823
pharynx
0.785763664
0.765922825
pollution
0.807044092
0.816168598
quake
0.078567569
0.084691683
sensory
0.399599899
0.402225994
servo
0.872101166
0.785898546
sleep
0.783933708
0.722791535
slump
0.708012949
0.629480015
strike
0.522569102
0.448517211
veteran
0.515124253
0.45146375
wineqred
0.619064343
0.611444293
wineqwhite
0.58210665
0.572266143
151
Table 14
Average Prediction Correlations per Dataset and Three Different Combinations of
Supervised Aggregation and IAF for Binary Target Variables Dataset
Avg. Corr. Log. Reg.
+ IAF NLR
Avg. Corr. Log. Reg.
+ IAF NLD
Avg. Corr. Reg. Tr
w/o IAF
abalone
0.633651826
0.636990551
0.64867844
adult
0.662989858
0.656921233
0.608645742
anneal
0.826555309
0.823932942
0.853821973
balance
0.923393427
0.924777928
0.79058825
bands
0.461862401
0.450021541
0.343296921
breastw
0.939701709
0.939464426
0.930009743
car
0.91660187
0.913644582
0.948927087
cmc
0.42385694
0.423400481
0.455736376
credit
0.768277547
0.763557368
0.751852807
creditg
0.464104961
0.469843758
0.393045324
diabetes
0.556749014
0.545362638
0.50644607
diagnosis
1
1
0.944533837
glass
0.802605598
0.804760434
0.824622632
heart
0.694785041
0.70493683
0.657426735
hepatitis
0.374975313
0.311577565
0.339608818
horse
0.569485166
0.55213211
0.541218353
hypothyroid
0.80477987
0.800429276
0.898883448
internetads
0.807436554
0.808116466
0.798458728
ionosphere
0.809199099
0.819279121
0.764848148
iris
1
1
0.996991022
letter
0.6349395
0.62902598
0.727029396
lymph
0.68823764
0.690663197
0.727999682
segment
0.999999995
0.999999953
0.997593306
sonar
0.559774797
0.557996241
0.496422658
spectf
0.505452116
0.510651243
0.473619771
vehicle
0.46898602
0.496373681
0.559857093
waveform
0.826959479
0.821165897
0.783760649
wine
0.912450784
0.923247357
0.90726124
wisconsin
0.912735058
0.907890135
0.873368639
yeast
0.420860794
0.420696658
0.412990274
152
Table 15
Average Prediction Correlations per Dataset and Three Different Combinations of
Supervised Aggregation and IAF for Numerical Target Variables Dataset
Avg. Corr. Reg. Tr.
+ IAF NLD
Avg. Corr. Reg. Tr.
+ IAF NLDU
Avg. Corr. Reg. Tr.
w/o IAF
auto93
0.851949884
0.854702482
0.783694291
autohorse
0.934671172
0.932528555
0.939116035
baskball
0.651189815
0.65121036
0.623578896
bodyfat
0.989455545
0.990353116
0.991221665
bolts
0.899445255
0.90182103
0.921987087
breasttumor
0.379942005
0.37994241
0.252316629
cholestrol
0.354768905
0.346985441
0.193892889
cloud
0.901964456
0.89860454
0.913436117
communities
0.798599026
0.796617813
0.791015749
cpu
0.843304898
0.842528113
0.97677025
elusage
0.905475647
0.905995831
0.881926102
fishcatch
0.932313534
0.933532791
0.985895292
forestfires
0.175728178
0.169928128
0.014991533
fruitfly
0.212102028
0.212108602
-0.064702557
housing
0.918270044
0.917557553
0.900012141
lowbwt
0.813006747
0.812914092
0.783555356
meta
0.462534319
0.454644049
0.368499477
parkinsons
0.887066413
0.881144444
0.941203949
pbc
0.66081537
0.660799138
0.534079402
pharynx
0.773127351
0.773127987
0.708101866
pollution
0.829522663
0.831729118
0.60923147
quake
0.082057392
0.08047578
0.073515546
sensory
0.404010864
0.404010206
0.386991775
servo
0.82787911
0.827815554
0.912721346
sleep
0.77696476
0.764053467
0.730795112
slump
0.690106281
0.689333045
0.493530846
strike
0.496745411
0.496578249
0.415429314
veteran
0.471382971
0.471232266
0.437471553
wineqred
0.611102154
0.615205884
0.580172859
wineqwhite
0.572707345
0.573805738
0.563459737
153
Table 16
Relationship between Target Linearity and Improvement of Logistic Regression by IAF
NLR for Binary Target Variables
Dataset
Target linearity
Improvement by IAF NLR
abalone
0.632486147
-0.015038624
adult
0.601697632
0.003616993
anneal
0.847487701
0.265333889
balance
0.839927779
0.010051891
bands
0.397808335
0.101084917
breastw
0.92253246
-0.001225938
car
0.814076541
-0.001610122
cmc
0.351769834
0.17744701
credit
0.733383933
0.063102998
creditg
0.425527893
0.040954514
diabetes
0.527451425
0.044424251
diagnosis
0.972462485
-5.2E-16
glass
0.817993073
-0.003667589
heart
0.6866888
0.030462835
hepatitis
0.381693585
0.098467765
horse
0.552106327
0.111811784
hypothyroid
0.584747606
0.080805492
internetads
0.784842676
0.00291351
ionosphere
0.681485184
0.15769256
iris
0.97214843
2.97338E-06
letter
0.45185967
0.10947387
lymph
0.723280478
0.033367981
segment
0.956800746
-4.89814E-09
sonar
0.512394902
0.043510527
spectf
0.452480291
-0.002580917
vehicle
0.57214217
-0.206318239
waveform
0.724956824
0.075158502
wine
0.910709707
154
-0.029160901
Table 17
Relationship between Target Linearity and Improvement of Regression Trees by IAF
NLD for Numerical Target Variables
Dataset
Target linearity
Improvement by IAF NLD
autohorse
0.946606415
-0.004733029
auto93
0.736992093
0.087094666
baskball
0.623578896
0.044278148
bodyfat
0.990358409
-0.001781761
bolts
0.85950705
-0.024449184
breasttumor
0.236969021
0.505814364
cholestrol
0.170467119
0.82971591
cloud
0.912861146
-0.0125588
communities
0.797109195
0.009586759
cpu
0.930542139
-0.136639452
elusage
0.845600615
0.026702402
fishcatch
0.963623075
-0.054348325
forestfires
0.073146275
10.72182832
fruitfly
-0.064702557
-4.278108926
housing
0.854820529
0.020286285
lowbwt
0.778262746
0.037586867
meta
0.383398892
0.25518311
parkinsons
0.459022397
-0.057519453
pbc
0.530953313
0.237297988
pharynx
0.699754036
0.091830693
pollution
0.679583809
0.361588664
quake
0.066085663
0.116191016
sensory
0.336532333
0.043977907
servo
0.836899785
-0.092955244
sleep
0.709322872
0.063177281
slump
0.436362228
0.398304251
strike
0.42286715
0.195739913
veteran
0.397186311
0.077516853
wineqred
0.583219599
0.053310483
wineqwhite
0.52352684
0.01641219
155
156
A.4. IFCL Syntax and Application
A.4.1.
Grammar Notation
The syntax will be described with the aid of meta-linguistic formulae in
the Backus-Naur Form (BNF) as proposed by Backus, et al. (1960):
<…>
::=
… | …
<character>
<string>
<integer>
<decimal>
<empty>
Meta-linguistic variables: nodes of the grammar tree
Definition of syntax element placeholders
Alternative choice of syntax elements
A symbol
Any sequence of symbols
A sequence of digits (0-9)
A sequence of digits (0-9) containing one decimal point
An empty set of symbols: no symbol at all
All symbols that are not inside of less than / greater than signs ( <…> )
except the symbols ::= and | are used literally in the IFCL language.
Different meta-linguistic variable declarations are separated by an
empty line. Comments can be made in IFCL files as a sequence of
symbols enclosed by the character #. In fact, comments are filtered out
before parsing, so they are not part of the language itself.
A.4.2.
IFCL File Structure
<IFCL file> ::=
<IFCL file node> <IFCL file leaf>
<IFCL file node> ::=
<IFCL file execution call> <IFCL file node>
| <empty>
<IFCL file leaf> ::=
<connect to database>
<IFCL leaf action sequence>
<IFCL leaf action sequence> ::=
<IFCL leaf action> <IFCL action sequence> | <empty>
<IFCL leaf action> ::=
| <drop database table>
| <execute sql>
| <load database>
157
| <induce membership function>
| <aggregate multiple variables>
| <data classification>
| <evaluate correlations>
A.4.3.
o
Executing IFCL Files in Batch Mode
Syntax
<IFCL file execution call> ::= @execifcl: file == <ifcl file
path>;
<ifcl file path> ::= <string>
o
Example
@execifcl: file == ./metainduction/abalone/load.ifcl;
A.4.4.
o
Connecting to the Database
Syntax
<connect to database> ::=
@connect:
hostname == <database host name>
SID
port
;
== <database service identifier> ;
== <database server port number> ;
username
== <user name> ;
password
== <password> ;
<database host name> ::= <string>
<database service identifier>::= <string>
<database server port number> ::= <integer>
<user name> ::= <string>
<password> ::= <string>
o
Example
@connect:
HostName
158
== localhost;
SID
== XE;
Port
== 1521;
UserName
== ME;
Password
== passwort;
A.4.5.
o
Droping a Database Table
Syntax
<drop database table> ::= @drop: table == <table name>;
<table name>::= <string>
o
Example
@drop: table == abalone_te_nlr_regtree
A.4.6.
o
Executing an SQL Script
Syntax
<execute sql> ::= @execsql: <sql>
<sql> ::=
file == <sql script file path> ;
| command == <sql statement> ;
<sql script file path> ::= <string>
<sql statement> ::= <string>
o
Example
@execsql: file == metainduction/abalone/01_sql/table_split.sql;
A.4.7.
o
Loading Data into The Database
Syntax
<load database> ::= <SQL*Loader> | <IFCL loader>
<SQL*Loader>::=
@sqlldr:
data == <path to data file> ;
control == <path to sqlldr control file> ;
skip == <number of rows to skip>;
159
errors == <number of errors to accept>;
<path to data file> ::= <string>
<path to sqlldr control file> ::= <string>
<number of rows to skip> ::= <integer>
<number of errors to accept> ::= <integer>
o
Example
@sqlldr:
data == metainduction/abalone/00_data/data.csv;
control == metainduction/abalone/00_data/sqlldr.ctl;
skip == 0;
errors == 10000;
o
Syntax
<IFCL loader> ::=
@load:
file ==
<path to data file> ;
table == <table to be created and loaded> ;
delimiter == <data delimiting character> ;
attributes == <attribute list> ;
types == <type list>;
<path to data file> ::= <string>
<data delimiting character> ::= <character>
<attribute list> ::= <attribute> <attribute list> | <empty>
<attribute> ::= <string>
<type list> ::= <type> <type list> | <empty>
<type> ::= n | c
160
o
Example
@load:
file == ./metainduction/balance/00_data/00_data.csv;
table == balance_load;
delimiter == ,;
attributes == left_weight, left_distance, right_weight,
right_distance, class;
types == n, n, n, n, c;
A.4.8.
o
Inducing a Membership Function
Syntax
<induce membership function> ::=
@inducemf:
table == <table for membership function induction> ;
target variable == <table column> ;
analytic variables == <column> | <column list> | <all columns>
induction == <induction type> ;
output file == <path to output file for sql classification
template> ;
<optional skipped columns>
<optional left out columns>
<table for membership function induction> ::= <string>
<column> ::= <string>
<all columns> ::= *
<column list> ::=
<column> <column list or empty>
<column list or empty> = <column>
<column list or empty> |
<empty>
<induction type> ::= l | nlr | nlru | nld | nldu | cp | npr | npru
| npd | npdu | iff | nc | mm
<path to output file for sql classification template> ::= <string>
161
<optional skipped columns> ::=
skip columns == <column list> ;
<optional left out columns> ::=
let columns == <column list> ;
o
Example
@inducemf:
table == abalone_tr;
target variable == y;
analytic variables == *;
induction == cp;
output file == ./metainduction/abalone/02_output/cp.sql.template;
A.4.9.
o
Classification of Data
Syntax
<data classification> ::=
@classify:
classified table == <table to be classified> ;
template file == <path to sql classification template file> ;
output table == <table for storing resulting classification> ;
<table to be classified> ::= <string>
<path to sql classification template file> ::= <string>
<table for storing resulting classification> ::= <string>
o
Example
@classify:
classified table == abalone_te;
template file == ./metainduction/regtree.sql.template;
output table == abalone_te_rt;
A.4.10. Aggregating Multiple Variables
o
Syntax
<aggregate multiple variables> ::=
@aggregatemv:
162
table == <table to be aggregated> ;
analytic variables == <column> | <column list> | <all columns> ;
aggregation operator == <aggretation operator>;
target variable == <table column>;
output file == <path to output file>;
column alias == <name of column containing aggregated membership
value>;
<table to be aggregated> ::= <string>
<aggretation operator> ::= linreg | logreg | regtree | min | max |
ap | as | avg
<name of column containing aggregated membership value> ::=
<string>
o
Example
@aggregatemv:
table == abalone_tr_s;
analytic variables == *;
aggregation operator == linreg;
target variable == y;
output file ==
./metainduction/abalone/02_output/linreg.sql.template;
column alias == mfc;
A.4.11. Evaluating Predictions
o
Syntax
<evaluate correlations> ::=
@evaluate:
table == <table containing resulting prediction> ;
<analytic variables> ::= <column> | <column list> | <all columns>
;
target variable == <table column>;
<optional result table>
<optional output file>
<optional result table> ::=
163
result table == <table for storing evaluation results> | <empty> ;
<table for storing evaluation results> ::= <string>
<optional output file> ::=
output file == <file for storing evaluation results> | <empty> ;
<file for storing evaluation results> ::= <string>
o
Example
@evaluate:
table == abalone_te_rt;
target variable == y;
analytic variable == mfc;
result table == evaluations;
output file == metainduction/anneal/attreval.csv;
A.4.12. Data Preparation
The following example IFCL code shows how to prepare datasets for
training and evaluation using the IFCL load action. It shows also how
to split training and test data randomly.
o
Example:
@connect:
HostName
== localhost;
SID
== XE;
Port
== 1521;
UserName
== ME;
Password
== passwort;
@load:
file == ./metainduction_num/sensory/00_data/00_data.csv;
table == sensory_load;
delimiter == ,;
attributes ==
Occasion,Judges,Interval,Sittings,Position,Squares,Rows_,Columns,
Halfplot,Trellis,Method,Score;
types == c,c,c,c,c,c,c,c,c,c,c,n;
null == ?;
164
@drop: table == sensory_split;
@execsql: command ==
create table sensory_split as
select
Occasion,Judges,Interval,Sittings,Position,Squares,Rows_,Columns,
Halfplot,Trellis,Method,Score as y,
# percent rank fuzzification #
percent_rank() over (order by score) as y_p,
# linear fuzzification #
(score - min(score) over()) / (max(score) over()-min(score)
over()) as y_l,
case when dbms_random.value <= 0.667 then 1 else 0 end as
is_training
# random split of data for training and test sets #
from sensory_load;
@drop: table == sensory_tr;
# create table with training set #
@execsql: command ==
create table sensory_tr as
select * from sensory_split where is_training = 1;
@drop: table == sensory_te;
# create table with test set #
@execsql: command ==
create table sensory_te as
select * from sensory_split where is_training = 0;
A.4.13. Attribute Selection
The following example IFCL code shows how to select relevant
attributes from a relation regarding a target attribute.
o
Example:
@connect:
HostName
== localhost;
165
SID
== XE;
Port
== 1521;
UserName
== ME;
Password
== passwort;
@inducemf: # membership function induction #
table == sensory_tr;
target variable == y;
analytic variables == *;
induction == nlr;
output file ==
./metainduction_num/sensory/02_output/nlr.sql.template;
@classify: # inductive attribute fuzzification #
classified table == sensory_tr;
template file ==
./metainduction_num/sensory/02_output/nlr.sql.template;
output table == sensory_tr_nlr;
@drop: table == attribute_selection;
@evaluate: # ranking of correlation of fuzzified attribute values
and target #
table == sensory_tr_nlr;
target variable == y;
analytic variables == *;
output file == metainduction_num/sensory/attreval.csv;
result table == attribute_selection;
166
A.5. Curriculum Vitae
Michael Alexander Kaufmann
from Recherswil SO
born on September 12th, 1978:
1993 – 1998:
1998 – 1999:
1999 – 2003:
2003 – 2005:
2005 – 2008:
2008 – 2011:
2011 – 2012:
2008 – 2012:
College of Economics, Bern-Neufeld
International Studies in Los Angeles, California
Bachelor of Science in Computer Science, Fribourg
Master of Science in Computer Science, Fribourg
Data Warehouse Analyst at PostFinance, Bern
Data Architect at Swiss Mobiliar Insurance, Bern
Consultant at Nationale Suisse Insurance, Basel
PhD in Computer Science, Fribourg
A.6. Key Terms and Definitions
IFC
Inductive fuzzy classification: Assigning individuals to fuzzy
sets for which membership function is generated from data
so that the membership degrees provide support for an
inductive inference.
ITF
Inductive target fuzzification: Transformation of a numerical
target variable into a membership degree to a fuzzy class.
MFI
Membership function induction: Derivation of an inductive
mapping from attribute values to membership degrees based
on available data.
IFCL
Inductive fuzzy classification language: Software prototype
implementing IFC.
NLR
Normalized likelihood ratio: A proposed formula for MFI.
NLD
Normalized likelihood difference: Another proposed formula
for MFI.
IAF
Inductive attribute fuzzification: Transformation of attribute
values into membership degrees using induced membership
functions.
Zadehan
Variable
A variable with a range of [0,1] representing truth values,
analogous to Boolean variables with a range of {0,1}.
Zadehan
Logic
A framework for reasoning with gradual truth values in the
interval between 0 and 1.
167