**Course Title:** MATLAB Programming: Applications in Engineering, Data Science, and Simulation **Section Title:** Machine Learning with MATLAB **Topic:** Evaluating machine learning models and performance metrics. **Introduction** Machine learning model evaluation is a critical step in the model development process. It helps you to assess the performance of your model, identify potential issues, and compare different models. In this topic, we will cover the key concepts, techniques, and metrics used to evaluate machine learning models in MATLAB. **Keys to model evaluation** Model evaluation involves assessing how well a model performs on unseen data. The key to model evaluation is to use a hold-out dataset, also known as a test set, which is not used during the training process. This helps to ensure that the model is not overfitting to the training data and can generalize well to new, unseen data. **Model performance metrics** Common model performance metrics used in machine learning include: * **Accuracy**: The proportion of correctly classified instances out of all instances in the test set. * **Precision**: The proportion of true positives (correctly classified instances) out of all positive predictions made by the model. * **Recall**: The proportion of true positives out of all actual positive instances in the test set. * **F1-score**: The harmonic mean of precision and recall. * **Mean Squared Error (MSE)**: The average squared difference between predicted and actual values. * **Mean Absolute Error (MAE)**: The average absolute difference between predicted and actual values. * **Coefficient of Determination (R-squared)**: Measures the proportion of the variance in the dependent variable that is predictable from the independent variable. In MATLAB, you can use the `confusionmat()` function to calculate accuracy, precision, recall, and F1-score for classification models, and the `mse()` or `mae()` functions to calculate mean squared error and mean absolute error for regression models. ```matlab % Accuracy, precision, recall, and F1-score [cm, acc, precision, recall, f1score] = confusionmat(labels, predictions); % Mean Squared Error (MSE) mse = mse(predictions, actual_values); % Mean Absolute Error (MAE) mae = mae(predictions, actual_values); ``` **Visualizing model performance** Visualizing model performance can help to identify potential issues and compare different models. Common plots used to visualize model performance include: * **Confusion matrix**: Shows the number of true positives, false positives, true negatives, and false negatives. * **ROC curve**: Plots the true positive rate against the false positive rate at different thresholds. * **Precision-recall curve**: Plots precision against recall at different thresholds. In MATLAB, you can use the `plotconfusion()` function to create a confusion matrix plot and the `plotroc()` function to create an ROC curve plot. ```matlab % Confusion matrix plot plotconfusion(labels, predictions); % ROC curve plot [X,Y,T,AUC] = perfcurve(labels, predictions, 'Class 1'); plot(X,Y); xlabel('False Positive Rate'); ylabel('True Positive Rate'); ``` **Cross-validation** Cross-validation is a technique used to evaluate model performance on unseen data while using all the available data for training and testing. Common types of cross-validation include: * **K-fold cross-validation**: Divides the data into k subsets and uses each subset as a test set in turn. * **Leave-one-out cross-validation**: Treats each instance as a test set in turn and trains the model on the remaining instances. In MATLAB, you can use the `crossval()` function to perform k-fold cross-validation. ```matlab % K-fold cross-validation k = 5; % number of folds [idx, sets] = crossval(k, labels); model_loss = zeros(size(labels, 1),1); for i = 1:k % Train and test the model using the current fold train_idx = sets ~= i; test_idx = sets == i; train_model = fitcsvm(X(train_idx, :), labels(train_idx), 'KernelFunction', 'polynomial'); predictions = predict(train_model, X(test_idx, :)); model_loss = merge(loss, idx, i); end ``` **Model selection** Model selection involves choosing the best model from a set of candidate models. Common techniques used for model selection include: * **Grid search**: Trains multiple models with different combinations of hyperparameters and evaluates their performance. * **Random search**: Trains multiple models with different combinations of hyperparameters chosen randomly and evaluates their performance. In MATLAB, you can use the `train()` function to perform grid search and random search. ```matlab % Grid search n = [1 2 3 4]; % Grid size for hyperparameter tuning rng('default'); % Set random seed opts = defaultOptions(X, y, 'optimization'); param_grid = createParamGrid(opts, 'Size', n); [best_idx, ~] = findBestParams(param_grid, labels, X, Y); % Random search num_search = 20; % Number of searches search_opts = optimoptions('fminsearch', ... 'MaxIter', num_search, 'PopInitRange', 0.001, ... 'StepTolerance', 1.0e-10); [rand_idx, ~] = fminsearch(@randFun, bounds, search_opts); ``` **Conclusion** Evaluating machine learning models and performance metrics is a critical step in the model development process. By using hold-out datasets, common model performance metrics, and visualizing model performance, you can identify potential issues and compare different models. Cross-validation and model selection are also important techniques used in model evaluation. With MATLAB, you can easily evaluate machine learning models using its built-in functions and techniques. **Recommended Reading** * [Model Evaluation in Machine Learning](https://www.mathworks.com/discovery/model-evaluation.html) by MathWorks * [Evaluation Metrics for Machine Learning Models](https://scikit-learn.org/stable/modules/model_evaluation.html) by scikit-learn * [K-Fold Cross-Validation in Machine Learning](https://www.section.io/engineering-education/k-fold-cross-validation/) by Section **Now that you have learned about evaluating machine learning models and performance metrics in MATLAB, we encourage you to try out the different techniques and functions discussed in this topic. Practice with your own datasets and models to gain hands-on experience. Upon completing this module, you may have questions or need further clarification on specific concepts. Please feel free to ask for help in the comments below.**