A language model is added to a product as another feature: an endpoint, a button, or an input field. A month into operation, it turns out the service does not answer every time, answers slowly, or returns garbage. Everything after that depends on the wrapper around the call.
That wrapper can be built so a failure never leaves the model itself. The sequence below is the one used in corporate backend systems that run AI continuously, and it comes from software engineer and systems architect Shokhrukh Kodirov, a judge at the ECDMA Global Awards 2024/2025.
The Model Should Not Sit in the Data Path
The first decision comes before any code. The model stores no data, returns none, and takes no part in serving it. It receives what it is given and returns a result.
The call goes out from the server only: the browser talks to its own application, the application to an agent, the agent to the service. The access key never reaches the browser.
A grep confirms it. The name of the AI subsystem must not appear in any of the classes that serve data, and a separate test enforces this requirement. A clean log counts as no proof here: it only means nobody has exercised the connection yet.
A Timeout and Two Classes of Failure
The timeout is set to 5 seconds. No answer arrives; the function returns null; the interface shows no suggestion. The user sees no error because no error occurred on the user’s side.
Failures split into two classes.
Codes 400, 404, 422, and 501 mean the request itself is malformed. A retry changes nothing, so the feature is switched off until the process restarts.
Codes 5xx and 429 mean a temporary problem on the service side. Those are never switched off: pause, then retry.
“Mixing these two cases means either wasting requests for nothing or switching off a working feature because of a second of network noise,” Kodirov says.
final class AiClient
{
private const TIMEOUT = 5;
private const COOLDOWN = 45;
/** Codes where a retry will not change the answer. */
private const FATAL = [400, 404, 405, 410, 422, 501];
private static bool $off = false;
private static int $retryAfter = 0;
public function assist(array $context): ?array
{
if (self::$off || time() < self::$retryAfter) {
return null;
}
try {
$res = Http::timeout(self::TIMEOUT)
->withToken(config(‘ai.key’))
->post(config(‘ai.endpoint’), $context);
} catch (ConnectionException) {
self::$retryAfter = time() + self::COOLDOWN;
return null;
}
if (in_array($res->status(), self::FATAL, true)) {
self::$off = true;
return null;
}
if (! $res->successful()) {
self::$retryAfter = time() + self::COOLDOWN;
return null;
}
return $res->json(‘directives’);
}
}
The wrapper throws no exceptions. The calling code receives null and behaves as though no suggestion had been planned.
Why a Length Filter Gives No Protection
A data leak gets found later than other defects, because the product keeps working normally.
A common move is to cap the length of outbound strings. In one corporate record-keeping system serving about 300 employees, the cap was set at 64 characters and treated as a safeguard.
“A person’s name, date of birth, card number and diagnosis fit comfortably into 64 characters. That is under fifty. A length limit is not privacy,” Kodirov says.
Reliable filters work on the type of the value. Numbers, booleans, and identifiers pass outward. A string is allowed only for a key drawn from a closed enumeration with an explicit list of values, and the key name has to match a strict mask.
Nested structures are discarded whole: nesting is how form content slips inside one harmless-looking key.
Sanitization runs twice, independently: on the client before sending and on the server upon receipt. The server does not trust the client, even its own.
The identifier mask requires the D modifier, and this mistake keeps turning up in other people’s code. Without it, PCRE matches $ before a trailing newline as well, so the string “ms\n” passes validation as a correct identifier.
A Test That Guarantees Nothing
A passing test suite on its own is no assurance that errors will stay out of sight later.
In that same system, 115 tests passed while material that should never have left the installation was going out verbatim. The test meant to catch the leak contained this line:
if ($key === ‘target’) { continue; }
It explicitly skipped the one field able to carry content. The green suite confirmed exactly what the test author had agreed in advance to leave unchecked.
Two techniques go into the working process after that.
Mutation checking: remove the fix; the test has to turn red; put the fix back. A test that stays green in both cases protects nothing.
A hostile probe: a script sends nested objects, long prose, a real name, a card number, keys such as Patient-Name and __proto__, a string with a newline on the end, and then shows what was actually left. In the case described, three leaks turned up that way. No green test had shown any of them.
Queues, Budgets and Retries
An AI subsystem rarely stands alone in a system. Background jobs, file uploads, and event broadcasts usually run alongside it, and four mistakes recur in these areas.
- Idempotency rests on a set. A counter fails here: two successful PUTs of the same part add two, the counter reaches the expected total, and assembly quietly puts together a corrupted file. With a set keyed by part number, a repeat simply overwrites the entry, and readiness requires a range with no gaps.
- A task budget comes from other people’s ceilings. The task stops at the fortieth second because the supervisor kills the process at the sixtieth, and redelivery occurs at the ninetieth. A task that is still alive at the ninetieth second is sent to a second worker while the first one is still writing.
- A dedicated queue with no guaranteed consumer is worse than a shared one. A queue nobody reads fails quietly: tasks pile up, the attempt counter stays at zero, nothing reaches the log, and the interface shows “queued” forever.
- Expendable work has to be separated from important work. With the real-time server unavailable, each event broadcast hangs for about two seconds, and three attempts eat roughly six seconds of worker time. In one such incident, 264 tasks were added to the failed table. In a shared pool, a dead real-time server would have slowed report writing, degrading important work for the sake of expendable work.
What Changes Over the Next Eighteen Months
A year ago, AI was introduced into projects as a feature. Now it is designed as a subsystem, with its own failure boundary, privacy boundary, and licensing model. By the same rules used to build a payment layer or a file layer. In practice, that comes down to three requirements absent from specifications a year ago. The AI layer has no right to cause a failure in the main feature, and a test locks the requirement in.
Structure and behavior go outward; content stays inside the installation. Decision logic and prompts live on the service; the code that ships to the client holds neither. Kodirov, founder and chief technology officer of Vegoogin LTD, expects connecting a model to become a routine operation over the next eighteen months. The work will move elsewhere: building the structure around it that allows the model to be unavailable, be wrong, or be swapped for another one without a single change in the application code.
Фотограф: Jakub Zerdzicki:Pexels
