**Experimental Evaluation of Learning Algorithms :-**

**Evaluating**the performance of learning systems is important because :

- Learning systems are usually designed to predict the class of "future" unlabeled data points.

Typical choices for performance Evaluation

- Error

- Accuracy

- Precision/Recall

Typical choices for Sampling Methods :

- Train/Test Sets

- K-Fold Cross-validation

**Evaluating predictions**

Suppose we want to make a prediction of a value for a target feature on example x :

- y is the observed value of target feature on example x.

- Å¶ is the predicted value of target feature on example x.

- How is the error measured?

**Sample Error and True Error**

The sample error of hypothesis f with respect to target function c and data sample S is :

errors(f) = 1/n ⅀xÎµsD (f(x),C(x))

The true error (denoted error (f)) of hypothesis f with respect to target function c and distribution D, is the probability that h will misclassify an instance drawn at random according to D.

errors(f) = Pr xÎµD(f(x)≠C(x))

**Difficulties in evaluating hypotheses with limited data**

Bias in the estimate : The sample error is a poor estimate of true error

- ==> test the hypothesis on an independent test set

We divide the example into:

- Training examples that are used to train the learner

- Test examples that are used to evaluate the learner

Variance in the estimate : The smaller the test set, the greater the expected variance.

Validation Set

**K-fold cross validation**

**Trade-off**

In machine learning, there is always a trade-off between

- complex hypotheses that fit the training data well

- simpler hypotheses that may generalize better.

As the amount of training data increases, the generalization error decreases.

## No comments:

## Post a Comment