We live in a bubble of hype. Since GPT-4 and Claude They entered the scene, and it seems the solution to any software problem is "just throw in a prompt." And yes, generative AI is revolutionary, creative, and capable of complex reasoning.
But it has a giant Achilles' heel: It is not deterministic.
If you're building a system that needs to label, classify, or filter data with 100% reliability, using generative AI is an architectural mistake. It's a half-hearted attempt that will cost you money and headaches.
Let's see why.
The difference between creating and deciding
To understand the problem, we need to get down to the nitty-gritty of how these models work.
- Generative AI (LLMs): They are probabilistic models. Their job is to predict the next event. token (word or part of it) that best fits into a sequence. They are designed for the variability.
- Deterministic AI (Classical Machine Learning): Models like Random Forest, Logistic Regression or specific BERT-type networks for classification. Their job is to draw a line in the sand and say, "This is A" or "This is B.".
When you use an LLM to classify (e.g., "Is this email a complaint or a compliment?"), you are forcing a creative machine to do binary work.
The problem of "Temperature" and hallucination
I'm sure this has happened to you. You ask an LLM to return a JSON object with a category. They do it correctly 95 times. But on the 96th time, they decide to get "creative.".
Instead of returning { "category": "support" }, it gives you back:
{ "category": "support", "note": "It seems the user is angry" }.
Boom! You parser It breaks down. Your automation fails.
Generative AI hallucinate Because it doesn't understand truth, it understands plausibility. If you lower the temperature to 0, you reduce the risk, but you don't eliminate the probabilistic nature of the model. There are no mathematical guarantees that the output is always structurally identical.
Latency and Cost: Using a cannon to kill flies
Imagine you have to classify 10,000 customer reviews a day.
- GenAI Option: You send the text to the OpenAI/Anthropic API. It takes about 500ms – 2s per request. You pay for each input and output token.
- Deterministic Option: You train a small model (or use a pre-trained one from Hugging Face). It takes 20ms locally. Zero marginal cost once deployed.
Using a model with 175 trillion parameters to say "Yes" or "No" is inefficient. It's like using a Ferrari to go buy bread at the corner store: you get there, yes, but you use a lot of gas and it's difficult to park.
When to say YES and when to say NO
To avoid any doubt, here's my golden rule for deciding which technology to use in your projects:
Use Generative AI when:
- You need resume long texts.
- You need extract complex or unstructured entities.
- You need transform the tone or format of the text.
- Creativity is a feature, not a bug.
Use Deterministic AI (or Rules) when:
- The output is a closed set of options (Category A, B or C).
- You need consistency from 100% (Critical Systems).
- You process large volumes of data in real time.
- Failure is not an option.
The hybrid solution: The pragmatic approach
You don't have to choose one or the other. The best systems I'm seeing right now use a hybrid architecture.
They use deterministic models (fast and cheap) for the initial screening or mass classification. Only when the deterministic model has "low confidence" (e.g., it doesn't know if it's A or B) is the request sent to an LLM to analyze the nuance with their superior "reasoning.".
Conclusion
Generative AI is an incredible tool, but it's not a jack-of-all-trades. If you need to classify, look for reliability and speed.
Don't let yourself be carried away by the hype. Sometimes, a good Logistic Regression or a well-fitted BERT model outperforms GPT-4 in specific tasks.
Engineering is about choosing the right tool, not the most modern one.