Machine Learning

Introduction

Machine learning algorithms is an umbrella term that covers a range of generic algorithms. There are other branches of machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Supervised learning is the most commonly used.

Supervised learning requires training a model with labeled training data (data with parameters and a solution). The supervised learning algorithm then generates a solution that is as close to the solution as possible. This solution can then be applied to any other problem of the same type.

Unsupervised learning is basically what data mining is: it attempts to look for hidden patterns in the data.

Multivariate Linear Regression

Multivariate linear regression is a 3 step algorithm that converts a training data set into a set of weights that most closely reproduces the answer in the training data set.

The 3 steps are:

1. For each parameter, set its weight to an arbitrary value, say 1.0
2. For every piece of data, see how far off the function using the given weights are off by. The average error amount is the cost of the function.
3. Repeat step 2 with different weights to minimize the cost (ie. the error)

Step 2 requires determining how 'wrong' the prediction is from the answer. This typically is done by:

Cost = (ans1 - pred1)^2 + (ans2 - pred2)^2 + ... + (ansn - predn)^2 / n

By squaring each difference, we penalize larger errors more.

Step 3 requires picking new weights that reduces the cost. This can be done by something called a batch gradient descent. A gradient descent is an iterative optimization algorithm that can be used to minimize the cost function and to find the best weights. However, the data that can be handled with this approach has to be linear. You can use other algorithms such as gradient boosting for non-linear data.

Hands On Overview

Some software that will be used are:

• NumPy - Python library for scientific computing
• scikit-learn - Python machine learning library
• pandas - (panel datas) virtual spreadsheet

Using NumPy

You can multiply an entire array with numpy like so. This is more efficient than iterating over each number since the library can make use of SIMD (single instruction, multiple data) instructions on the CPU. Using libraries such as this and by removing iterative loops is called vectorizing the code.

If you find yourself writing loops over an array, you're probably doing it wrong.

import numpy as np

sizes = np.array([
    1, 2, 3, 4, 5
])

sizes = sizes * 0.3

Gradient Boosting

A machine learning algorithm that uses an ensemble of decision trees to predict values. This method can handle complex patterns and data that linear models can't.

A decision tree is basically a simple 'if' on a particular value. Eg:

sq_ft > 1000?
 -> Yes: +$50,000 value
 -> No:  -$50,000 value

Decision trees have one decision point. We can chain multiple trees together in order to have more decision points.

To increase the complexity of a model using decision trees, we can either:

1. Create a tree that's hundreds of layers deep with thousands of branching paths or
2. Create many small trees that contribute a little to the final answer (called ensemble learning)

Glossary

Word	Definition
Overfitting	Where the model is too specific to the training data.

Other Topics

• Regularization
• cross-validation

See Also

• https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471
• https://www.lynda.com/Data-Science-tutorials/Build-simple-home-value-estimator/548594/598227-4.html