In this article, we continue with a practical description of and, in our follow-on article, we will do the same for , a subset of .
What is ?
Machine learning is one approach to — not the only approach, but currently one that is easily the most successful in enterprise applications and more. Machine learning approaches in are different from explicit, rules-based approaches, such as expert systems, in that they are designed to learn from the data. The algorithms at the heart of applications use data to generate and refine rules (as opposed to the programmer explicitly defining the rules). The computer then decides how to respond based on what it has learned from the data.
So, how does this work? At the highest and simplest level, all methods have two phases: training and inference.
In the first phase, the algorithm, or model, is trained to recognize features in a dataset, such as the characteristics that are common to housing prices, consumer purchases or pictures of common objects. If the model sees enough data that is consistent and well-labeled, it can find patterns and “learn” the features in the data that are consistent with the labels. For example, it could be used to understand which features predict home purchase prices most accurately, which purchase attempts are fraudulent, and which objects under a scanner are apples, oranges or bananas.
Of course, you need great data to train an accurate model. But how do you know your model is accurate? You validate your model against a subset of the data that is not used for training and score the accuracy. This is an iterative process, with successive rounds of training and validation.
For a simple application, you might train your model on 70 percent of the data in a dataset and validate it on the remaining 30 percent. With a more complex application, you might use 60 percent of the data for training, 20 percent for validation and 20 percent for final testing. These ratios aren’t hard-and-fast rules, and the appropriate ratios for each phase can vary based on the dataset. It’s the data scientist’s job to look at the dataset and determine the ratios that will work best — and to determine when the trained model is accurate enough to be deployed.
In this second step in the process, the rubber hits the road. You put the trained model to work with real-world data and let it infer answers based on that data. You then monitor the performance of your model over time. If it’s not meeting your accuracy goals, you might send it back to boot camp for additional training, often with additional/new data that has been collected.
Solving simple problems with linear regression
Machine learning has been around for ages in various forms. Linear regression, for example, is a statistical technique that is a very basic form of . Linear regression is used to show the relationship between variables, often expressed in terms of a slope and a Y-intercept in a chart. […]