What are the pros and cons of using deep learning vs. traditional ML methods?

Deep learning, a subfield of machine learning, trains neural networks inspired by the human brain to make decisions. These networks have multiple layers of interconnected neurons that process data. The "deep" in deep learning refers to many hidden layers between input and output. Each layer learns to extract features, allowing networks to discover patterns without manual feature engineering. Deep learning excels in image and speech recognition, NLP, and robotics, requiring large labeled datasets and significant computational power. Key architectures include CNNs, RNNs, and GANs. This method processes unstructured data and automates feature extraction, distinguishing it from traditional machine learning.

Traditional machine learning is not dead and it will benefit from the new AI Regulations (EU act 2024). The reasons are simple. First, there are still many use cases when you can get a great accuracy using logistic regression for example. On top of that, when the AI regulation hit the market, to use deep models you will need to spend more on AI compliance and ensure its safety. If you know already that a simple model can reach like 90% accuracy, would it make sense to justify the extra burden of AI compliance costs to use a generative AI model that reaches 95% accuracy? It is arguable. The rule of thumbs is: the larger the model, the more prone to errors or unexpected results, the more safety procedures the company should budget in.

Deep learning can also help us in identifying patterns that humans don't usually comprehend or have seen before. Especially when we cannot label the billions of data points due to the size of the training data. There has been some recent development where we can create synthetic data using these deep learning models which can help increase our training set with labeled data.

A simple example to explain what does 'learning' really mean using a simple line. To represent a line, we have this line formula: y = mx +b where m is slope and b is y-intercept. From this formula, we can come up with y given x,m and b. Now, let's say we do not have the line formula but have a set of (x,y) data points. Deep learning will learn from this data and come up with unknown (x,y) and these set of points will form a line. In the real world, most of things are not linear. Therefore, there is no mathematical formula. This is where DL thrives. It 'learns' from data regardless of shape or form.

Deep learning is uniquely able to learn from much larger volumes of raw data. Importantly, the patterns learnt can be transferred to downstream tasks, this is the basis behind recent advances like ChatGPT, stable diffusion, and DALL-E 2.

Interpretability is critical for understanding how a model is performing and finding opportunities to improve it, so the loss of interpretability as our models get complicated has been a real pain point for us. We need simple interpretable models on top of the fancy black box models, just to tell us what the models are doing.

Deep learning is susceptible to bias. Due to their application in automated use-cases, the potential harm of negative biases is much greater (e.g. https://www.wsj.com/articles/BL-DGB-42522)

One thing that can help people in this field is to keep an open mind about each approach and not tie yourself to one method. For example, even if deep learning models sound interesting, do not jump into using them for trivial tasks. The computing resources required to train and deploy a deep learning model may not be worth the improvement in accuracy you get from using a deep learning model instead of a traditional model. That is also what is widely understood in the industry. There's a saying that I believe fits here, "keep things simple and stupid".

Some Traditional ML models can do really well with non linear data even if you have large and complex datasets e.g. Random forests, SVM etc. So, this is not a decisive factor. Deep learning can work great with feature selection and using more important features the most and without any or minimal preprocessing, that is one of the things that goes to it’s advantage.

Traditional ML has been the standard for use in natural language processing as there are several models and libraries just to use for that. NLTK and spaCy are the ones that I have used. However, a recent development is to use deep learning transformer architecture for larger datasets. Both Google BERT (Bidirectional Encoders Representations from Transformers) and ChatGPT (Generated Pre-Trained Transformer). ChatGPT uses both traditional and deep learning models in its architecture. A method that is discussed below.

As a biostatistician, combining deep learning with traditional machine learning allows me to maximize data utilization. I may use traditional machine learning for feature extraction from medical data, then employ deep learning for intricate patterns detection or disease prediction. Deep learning can also be used to augment data in cases of rare diseases with limited datasets, and traditional methods could then evaluate the model. Creating data embeddings using deep learning, followed by traditional clustering techniques, can help identify patient subgroups. Effectively, deep learning can be a powerful tool within a broader machine learning system, improving insights and predictions in medical informatics.