How often does your algorithm learn? And other important questions to ask ML vendors
Vendors are quick to include in their sales pitch, but Principal Engineer and Chief Data Scientist recommend technology buyers ask three questions to help see through the hype to determine what they are actually getting.
Copyright by which-50.com
Fralick chairs McAfee’s Analytic Center of Excellence and is responsible for the cybersecurity company’s technical analytic strategy that integrates into McAfee consumer and enterprise products.
Prior to Intel’s divestiture of McAfee, she was Chief Data Scientist in Intel’s Internet of Things Group where she developed and analytics for over eight different markets.
And while broadly applying the term makes it easier to market and talk about, for Fralick who has 40 years expertise in the field, each type of model under the umbrella has different levels of complexity and intelligence behind them.
“As a data scientist and an engineer, I look at to be very specific mathematically. I look at to be very different mathematically from .”
Speaking with Which-50 between sessions at MPower in Las Vegas last week, Fralick said most technology vendors claiming to do are really doing “and maybe a little bit of .”
Fralick shared her top three questions to help technology buyers better understand what they are getting from a solution, as well as what makes for a good or a bad answer.
1. How often does your Machine Learning algorithm actually “learn”?
The first thing Fralick recommends asking is, how often does your algorithm actually “learn”?
Models change over time, meaning when new information comes in the model is updating itself to be more relevant or more accurate. The process is outlined in the image below.
A correct response will indicate, “The algorithm learns at the rate that was determined at the time of model development and is updated periodically and applied to new signals.”
Some models may be updated continuously, while other models, they are updated based on statistically significant changes to inputs to the model, Fralick said.
Buyers should be wary of vague answers like “routinely”. Good follow up questions include “How often?”, “What signals does your company use to update the model?” Or even the very basic “How do you know the model is working?”
2. How accurate is your Machine Learning model?
Nothing is ever 100 per cent accurate.
“Accuracy is the measure of error,” Fralick explained. “So if your model is not as accurate as it could be, it means it has more error and giving you incorrect decisions.”
Error is measured differently for different models, such as false positives, false negatives, true positives and true negatives. But there are other ways to calculate error which require taking a look at the math. Fralick provided an example of a correct answer:
“The specific model for this purpose has a 95 per cent accuracy. We also measure other forms of accuracy, such as Root Mean Square Error, Generalized R^2, and additional ROC metrics such as Recall, F1 Score, and Matthew’s Correlation Coefficient.” […]
Vendors are quick to include in their sales pitch, but Principal Engineer and Chief Data Scientist recommend technology buyers ask three questions to help see through the hype to determine what they are actually getting.
Copyright by which-50.com
Prior to Intel’s divestiture of McAfee, she was Chief Data Scientist in Intel’s Internet of Things Group where she developed and analytics for over eight different markets.
And while broadly applying the term makes it easier to market and talk about, for Fralick who has 40 years expertise in the field, each type of model under the umbrella has different levels of complexity and intelligence behind them.
“As a data scientist and an engineer, I look at to be very specific mathematically. I look at to be very different mathematically from .”
Speaking with Which-50 between sessions at MPower in Las Vegas last week, Fralick said most technology vendors claiming to do are really doing “and maybe a little bit of .”
Fralick shared her top three questions to help technology buyers better understand what they are getting from a solution, as well as what makes for a good or a bad answer.
1. How often does your Machine Learning algorithm actually “learn”?
The first thing Fralick recommends asking is, how often does your algorithm actually “learn”?
Models change over time, meaning when new information comes in the model is updating itself to be more relevant or more accurate. The process is outlined in the image below.
A correct response will indicate, “The algorithm learns at the rate that was determined at the time of model development and is updated periodically and applied to new signals.”
Some models may be updated continuously, while other models, they are updated based on statistically significant changes to inputs to the model, Fralick said.
Buyers should be wary of vague answers like “routinely”. Good follow up questions include “How often?”, “What signals does your company use to update the model?” Or even the very basic “How do you know the model is working?”
2. How accurate is your Machine Learning model?
Nothing is ever 100 per cent accurate.
“Accuracy is the measure of error,” Fralick explained. “So if your model is not as accurate as it could be, it means it has more error and giving you incorrect decisions.”
Error is measured differently for different models, such as false positives, false negatives, true positives and true negatives. But there are other ways to calculate error which require taking a look at the math. Fralick provided an example of a correct answer:
“The specific model for this purpose has a 95 per cent accuracy. We also measure other forms of accuracy, such as Root Mean Square Error, Generalized R^2, and additional ROC metrics such as Recall, F1 Score, and Matthew’s Correlation Coefficient.” […]
Read more – which-50.com
Share this: