Is machine learning just a fancy term for advanced heuristic algorithms?
In computer science, there are different classes of computational problems, which can find their parallels in different levels of our universe’s physical organization. First there are deterministic problems solvable in polynomial time like the computation of a subatomic particle’s velocity, which can be translated to a relatively simple mathematical equation. Then there are non-deterministic problems, when there are usually multiple components to choose from to achieve the optimal result, from the point of physics this would mean an energetically efficient solution, from the point of evolutionary biology we could use the term fitness. If the fitness or solution to such a problem can be evaluated deterministically in polynomial time, we call such problems to be non-deterministic polynomial ones, or simply NP.
However, the non-deterministic problems can be made even harder. Let us imagine there is a dependency between the components, such as different particles interacting with one another through a force such as electromagnetic or gravitational one. In this case, it does not only matter how many of each particle there are to achieve the energetically efficient or optimal solution, but also their order, to say the least. If we continue in this exercise, the complexity rises until it reaches exponential time, space and even decidability, which means that for a certain problem we cannot, using modern computers, find a direct algorithm, whether deterministic or non-deterministic one, that would end not in millions of years, but actually never. That is why we can use only those algorithms that approximate the solution.
The algorithms that are not designed to find a solution to a given task directly, but instead approximate it, are called heuristic algorithms. Such algorithms are usually used to find a solution to different classes of unrelated problems, like the evolutionary algorithms, which can be used to approximate almost anything despite the fact that they as well as most other heuristic algorithms are used to quickly find a solution to NP problems. In artificial intelligence, evolutionary algorithms, especially genetic programming, are commonly used. The plus and minus of evolutionary algorithms is that they are not dependent or usually taught using data, but the speed and accuracy depends entirely on their parameterization and the choice of the initial population of guessed solutions, which are further evolved to a final one with the highest or lowest fitness.
Machine learning, on the other hand, introduces algorithms that are formed using datasets, usually split into training and testing parts. The result of the training is a model, which can be represented by so-called decision trees or, just for illustration, by IF conditions for different characteristics of the input. We can extend genetic programming to learn such ability from the data as well by evaluating fitness in every generation using the training dataset, if both input and output is known (supervised learning). Hence the originally heuristic algorithm also becomes a machine learning algorithm, if we just extend the evaluation of the fitness using the training dataset. Both machine learning algorithms such as linear regression, support vector machines and so on and genetic programming are then used to find an output for a given input, thus making a prediction or performing some task. The output is then not an originally presumed, deterministic solution, but rather an approximation, a guess with a given precision. Does not that sound like a heuristic algorithm?
In machine learning, deep learning with its modeling of a human mind goes even further, but as well as the evolutionary algorithm it is just a modeling, a usage of a natural process, in this case the process of human thinking. And from the evolutionary perspective, our brains are also a heuristic, approximative and predictive tool to quickly find a solution mostly to social problems that are not even non-deterministic, but usually not even solvable in exponential space by modern computers, maybe not even decidable. From the mathematical perspective, our brains’ decisions would probably be assigned a precision of about 70 % in social matters, but for such undecidable problems we actually cannot calculate it. Neural networks in machine learning give us a similar hope as our brain does, that for undecidable or very hard problems, such as predicting the outcomes of chaotic systems including spreading of fires, change of climates and the nuclear fusion, they would come up with a solution, a prediction, that would be good enough for our purposes, needs, or even for simulations of chaotic systems, where there are many dependent variables and an extreme sensitivity to initial conditions uncapturable by standard mathematical methods. Thus the neural networks, as well as our brains, are also heuristic in their nature.
