Machine learning teams can spend months developing more complex models. This is often seen as a solution to performance issues, but the root cause of failure lies in inconsistent or poorly defined labeling. As a result, resources are wasted at the production scale, becoming insufficient for new tasks, thereby degrading the label quality of new solutions.
Artem Kalyta, a senior data scientist and machine learning engineer, has long observed this pattern in ML production. He develops such solutions in financial fraud intelligence and cybersecurity, where even a minimal error can slow production or lead to irreversible financial consequences and loss of end-user trust. Artem explained why the quality of labels, rather than complex architecture, should be the foundation of effective ML systems.
Label Quality as a First-Order System Design Decision
Artem’s experience has shaped his approach to development, leading him to rethink engineering processes by eliminating root causes. He explains it this way:
“In the systems I’ve worked with, labels encode business logic and security policy. ML models implement the policy embedded in the labels. This is especially critical for fraud prevention and security systems: how “normal” and “suspicious” behavior are defined determines the subsequent operation of the entire system. You can create a very complex architecture, but it won’t make any difference.” It will still ultimately reproduce the values encoded in the target model.”
Indeed, inconsistent data labeling introduces conflicting signals into training sets. For example, features characteristic of legitimate behavior are increasingly perceived by the model as indicators of fraud. At the same time, real patterns of risky events are blurred by the inclusion of “safe” examples in the set.
This results in a hard performance ceiling: the ROC-AUC metric plateaus. The advantages demonstrated by the model in offline testing conditions do not translate to a real production environment. Attempts to improve quality by increasingly complex architecture only increase computational costs and latency and do not solve the problem.
“Relabeling data frequently has a bigger impact than refining model performance. Chargebacks, proxy fraud, and anomalies were grouped under a single label in a project’s fraud detection system. Our team separated them into distinct classes according to predefined criteria. As a result, the ROC-AUC rose by 12% while the architecture stayed the same.”
Lessons from Production Fraud Systems
Artem also worked on large-scale projects in which models made more than 100 million decisions per day, and individual components processed more than a billion events per day, with a p99 latency of less than 200 ms.
“I noticed that replacing generic labels with domain-specific definitions improved performance, something that architectural changes don’t always achieve. In some cases, hybrid models were able to outperform earlier systems without any architectural intervention by improving label definitions.”
Artem also highlighted: “A model architecture is only as effective as its label definitions allow. Treating heterogeneous risks as a single class is a systemic flaw, not a data issue. Therefore, my team and I abandoned generic labels and implemented domain-specific labeling schemes made for specific risk scenarios. As a result, the pipeline remained simpler, and performance improved.”
Trust, Safety, and the Cost of Getting Labels Wrong
Incorrect labeling in trust and security systems, Artem explains, is effectively a policy violation. All organizational decisions depend on how situations are labeled, and the models are then implemented. If the same pattern is labeled inconsistently, for example, both as actual fraud and as “friendly” (a dispute with a client, etc.), this creates an inconsistent chain that triggers a large number of false positives in the model. The visible result of such poor labeling is the blocking of legitimate users, who will never return.
“I encountered this problem while preventing fraud at iGaming companies. The initial labels reflected previous decisions rather than current regulatory requirements. Depending on the data source, identical user behavior could be classified as “fraud” or “acceptable risk”, producing noise that could not be removed by any architectural modifications. We implemented more than 300 domain-specific risk functions to evaluate disputed cases and revised the labeling criteria to meet regulatory requirements. The system was better adapted to regulatory restrictions, and the company was able to avoid millions in fraud losses.”
Artem is convinced that labeling in trust and security systems is a constantly evolving process. As threat models are updated, so should labels. Ideally, labeling should be treated as operational infrastructure and verified and updated with the same rigor as the models it supports.
Rethinking ML Maturity Over Model Layout
Finally, Artem Kalyta explains that in modern development, ML maturity should be determined not by the complexity of the architecture, but by the quality of data labeling. Labeling encodes business policy; it alone defines the logic of decision-making. If it’s inconsistent, the entire model, which is subject to errors in the target variables, collapses.
A major problem in the industry, Artem believes, is that scaling challenges are incorrectly attributed to algorithmic limitations, when in fact the problem remains weak label management. In the long term, only those ML systems that value labeling quality will win.
Rashan is a seasoned technology journalist and visionary leader serving as the Editor-in-Chief of DevX.com, a leading online publication focused on software development, programming languages, and emerging technologies. With his deep expertise in the tech industry and her passion for empowering developers, Rashan has transformed DevX.com into a vibrant hub of knowledge and innovation. Reach out to Rashan at [email protected]
