Start a project
ServicesWorkPartnersAboutBlog Contact Start a project

Procurement ·

How to choose an AI development company, and when not to hire one

Published by a vendor, so read it adversarially. Every criterion here can be used against us, and three of them we currently fail — those are named at the end.

This guide is published by an AI development company, which is a reason to discount it. Read it as you would a competitor's brief: every criterion below can be used against us, and three of them we currently fail — those are named at the end. What follows is what separates a firm that builds production systems from a firm that resells an API with a project manager attached: acceptance criteria that can fail, who owns the prompts and the evaluation set when you leave, which obligations follow your data rather than your vendor, and the answers that end a conversation.

First decide whether this needs an agency at all

The cheapest AI project is the one you never commission. Three alternatives beat an agency more often than agencies admit: an off-the-shelf product, one contractor, and a permanent hire.

Buy the product when the problem has a common shape. Meeting notes, deflection over a public help centre, search across a tidy document set, invoice extraction in a standard format — each is somebody's entire business. A build wins only when the process is genuinely yours: a workflow no product models, an integration surface nobody serves, or data you may not send anywhere.

Hire one contractor when the work is one discipline and one deliverable — a prototype, a retrieval spike, an evaluation harness. Firms earn their margin when the work spans data, application and infrastructure at once and somebody must be accountable where those disagree. Hire internally when the capability has to outlive the engagement, and write the handover into the contract at the start.

Then there is the case where nothing should be built yet. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, naming poor data quality, inadequate risk controls, escalating costs and unclear business value (Gartner). Three of those four are settled before a vendor is chosen. If nobody can say what the system replaces, what a wrong answer costs or where the data lives, no vendor selection repairs it. Our own AI consulting scoping calls are free and a fair number end with "do not build this", which is cheaper to hear in week one than in month seven.

Telling an engineering firm from a reseller with an API key

Both have the same website. The difference appears within ten minutes of asking about failure, because one has been on the wrong end of it and the other has only ever demonstrated the happy path.

Reselling is not disgraceful. Plenty of useful software is a thin layer over somebody else's model. The failure is paying a reseller's price and expecting an engineering firm's behaviour at two in the morning.

Ask what happens when the model is wrong. The engineering answer describes a trace — the question, the retrieved documents with scores, the assembled context, the output, the model and prompt versions — then how a specific complaint is reproduced and which component the failure is attributed to. The reseller's answer is about improving the prompt, and it arrives instantly, because it is the only answer available to somebody with no record of what happened.

Then ask to see an evaluation harness — the harness, not the results, anonymised, from any past project. A firm without one will offer to build it for you as a chargeable discovery, which is a complete answer to your question.

  • "How do you reproduce a wrong answer?" Good: a retained trace per request, components named. Evasive: prompt engineering, and a newer model.
  • "Show me an evaluation harness." Good: an anonymised repository and an argument about thresholds. Evasive: a proposal to build one at your expense.
  • "What is idempotent in this design?" Good: a specific list and the incident that produced it. Evasive: a definition of idempotency.
  • "What went wrong in production most recently?" Good: a date, a cause, a code change and mild embarrassment. Evasive: a client who kept changing their mind.

Acceptance criteria, and why "we will fine-tune a model" is not a deliverable

A deliverable can be observed to exist or not exist. "Fine-tune a model", "implement RAG" and "build an agent" name activities, and an activity completes by definition — nobody can fail to have fine-tuned.

A real acceptance criterion names five things: the test set, the metric, the threshold, who adjudicates, and what happens if it is missed. Agree all five before work starts, against data the vendor did not choose. Criteria written afterwards describe what the build happened to achieve, which is why they always pass.

This is why a fixed accuracy figure promised before anyone has seen your data is a red flag rather than confidence. Accuracy on a retrieval or extraction task is a property of your corpus and your question distribution far more than of the vendor's skill, so a firm quoting 95% before reading your documents is quoting somebody else's problem — and once your corpus turns out to hold three document generations, two languages and a scanning backlog, that number becomes a dispute rather than a target. Measure a baseline first, then set thresholds against it and against what a wrong answer costs you (how to evaluate a RAG system).

Report per category, never as one average, and score refusal as its own criterion. A blended figure hides the single stratum somebody in your business actually cares about.

ACCEPTANCE - phase 2, supplier invoice extraction

Test set     220 invoices sampled by the client from Q1 production,
             unseen by the vendor, adjudicated by the client's AP team.
             Held by the client. The vendor receives scores, not the set.

Fields       invoice_number, supplier_gstin, invoice_date,
             taxable_value, tax_amount, total

Metric       exact-field accuracy, reported per field and per supplier,
             never averaged into a single number

Threshold    >= 0.97  invoice_number, supplier_gstin, total
             >= 0.93  remaining fields
             no field below 0.90 for any of the top 20 suppliers by volume

Refusal      a document that is not an invoice is rejected, not guessed;
             measured on 40 negative cases supplied by the client

Exceptions   any field under threshold routes to the review queue with the
             page image and the extractor's own confidence attached

Evidence     the scoring script, read by both parties, run by the client on
             the client's infrastructure, output archived with the result

If not met   one remediation cycle of N working days at the vendor's cost;
             if still not met the phase closes, phase 3 does not begin,
             and the phase 3 fee is not payable
An acceptance clause in the shape that can actually fail. The thresholds are placeholders — yours come from a measured baseline and from what a wrong field costs you, not from a vendor's brochure.

The three lines a weak vendor negotiates hardest

Watch which lines attract resistance; the resistance is more informative than the proposal.

Test set custody: a vendor who wants the set in their own repository wants to tune against it, and a set tuned against stops measuring. Per-field reporting: averaging is how a collapsed category disappears behind nine healthy ones. The consequence clause: a criterion with no stated consequence is a wish — it need not be punitive, but one that costs nothing to miss will be missed.

Ownership: code, prompts, evaluation sets, weights and what the vendor learns about you

Source code ownership is the clause every contract covers and the least valuable item on the list. The parts that decide whether you can leave usually go unmentioned.

Ask for each of the five artefacts below by name. A contract that assigns "the software" and says nothing about the rest leaves the working part ambiguous, and ambiguity resolves in favour of whoever holds the repository. Our own answers are on the FAQ.

Source code, and the account it runs in

Assignment on payment, not a licence to use, and possession from the first week rather than a handover at the end that a dispute can delay. Then ask whose cloud account it runs in: code you own, deployed where you cannot log in, is not a system you control.

Prompts, tool definitions and retrieval configuration

This is the product: prompts, system messages, tool definitions, chunking parameters, retrieval and reranking settings, guardrail rules. They hold more accumulated judgement than the application code, and they are what a departing vendor characterises as reusable methodology. "The vendor retains its pre-existing tools and libraries; everything configured, tuned or written for this system is the client's" is the clause you want.

The evaluation set

The golden set is the most expensive artefact in the project, because it is paid for in your domain experts' time, and the one most often left on the vendor's laptop. It is also what makes changing vendors survivable.

Fine-tuned weights and adapters

If anything is trained, establish who holds the weights or adapter and whether you can export it — some hosted fine-tuning does not let the artefact leave, a fact to discover before training rather than during an exit. Training on personal data also creates an obligation hard to satisfy later, because a model that has learned from a person's data is not straightforwardly erasable (the DPDP Act and AI).

What the vendor accumulates about your business

An engagement also produces knowledge: your process, your exception taxonomy, your labelled data. Ask two questions. Does our data, or anything derived from it, improve anything you sell to someone else. Does our workflow become a template you resell in our sector. Both have acceptable answers; neither has an acceptable silence.

Data handling, when the buyer or the data is Indian

The obligation does not transfer with the work. Under the DPDP Act you remain responsible for processing carried out on your behalf, which makes vendor selection a compliance decision rather than a procurement one.

Section 8(1) of the Digital Personal Data Protection Act, 2023 makes a Data Fiduciary responsible for complying with the Act "in respect of any processing undertaken by it or on its behalf by a Data Processor", section 8(2) permits engaging a processor "only under a valid contract", and section 8(7) extends erasure to the processor (MeitY). A vendor's reassurance is therefore not a defence: the contract is the mechanism, and the diligence is yours.

So the questions are concrete, and "we are DPDP compliant" answers none of them: a compliance claim describes a management posture, and the questions below ask for a region and a retention period.

Sector rules bite earlier than the general law. For an RBI-regulated entity, the Master Direction on Outsourcing of Information Technology Services (10 April 2023, effective 1 October 2023) requires audit, monitoring and inspection rights for the entity and the RBI, prior approval before the service provider uses sub-contractors, and audits reaching those sub-contractors (Khaitan & Co). What may go into a hosted model's prompt is in RBI data residency and calling an external LLM. And every prompt is a copy, so ask where redaction happens and what is logged before it.

  • Every sub-processor named with its region — model provider, observability, transcription, annotation
  • Retention stated separately for vendor and model provider, with the setting that enforces it
  • A described path by which a deletion reaches logs, vector stores, caches, backups and anything trained
  • Audit and inspection rights that name your regulator, if you have one

Pricing shapes, and the failure each one invites

Three shapes cover almost every engagement: fixed scope, ongoing retainer, embedded pod. Each is defensible and each fails characteristically, so the useful question is which failure you are equipped to manage.

Fixed scope fails by making discovery expensive. AI work discovers things by nature — the data is messier than the sample, the real question distribution is not what the business described — and every discovery becomes a change request, so the incentive on both sides bends towards not mentioning them. Counter it with short phases and the discovery placed deliberately in phase one: a measured baseline, a written finding, and the right to stop without buying phase two.

A retainer fails by decaying into maintenance: the first quarter builds and the fourth is a support contract at a build rate, unreviewed because nothing has gone wrong. An embedded pod fails by dissolving accountability — once the vendor's engineers sit inside your standups, nobody owns an acceptance criterion. Counter the first by naming an outcome for each period and asking out loud whether it still exists, and the second by keeping one deliverable a quarter that is accepted or rejected as a unit by somebody outside the pod. The three shapes as we run them are on engagement models, and what actually moves the number is on AI development cost in India.

Whichever shape you choose, ask what the vendor does when their estimate is wrong. The answer tells you whether the cost of being wrong lands on the party who produced it, on you, or on the quality of the work — and the third is the one nobody names, because it never appears on an invoice.

Who actually writes the code

Ask for the names, roles and locations of the people who will do the work, and ask directly whether any part of it is subcontracted. The question is ordinary; the reaction to it is diagnostic.

The common pattern is sold by the people in the room and delivered by a partner firm you never meet. That is bad undisclosed, because then your acceptance criteria, your security review and your data agreement were negotiated with a party that is not doing the processing, and you learn this during an incident. For a regulated buyer it is worse than awkward: the RBI direction above requires prior approval before a service provider uses sub-contractors and requires audits to reach them.

Four questions. Who wrote the last three systems resembling this one, and what did each of them do. Will those people be on this engagement, and for what share of their time. What is the notice period if the lead engineer leaves, and what is written down that lets somebody else continue. Is any part subcontracted, to whom, in which country, under what agreement.

ZenMagix does not subcontract; the people who scope the work build it. That claim costs nothing to make on a website, which is exactly why you should treat it as you would any other vendor's version — ask for the names, and ask to speak to one of them without an account manager on the call.

What happens after launch, which is where the surprises live

The system you accept is not the system you will be running in eighteen months. Three things move underneath it whether or not anybody is paid to watch: your data, the model, and the price per query.

Data drift is the quietest. The corpus changes, a policy is superseded, and an answer correct in March is wrong in September while every dashboard stays green. Ask who re-runs the evaluation set after launch, on what schedule and against which corpus snapshot. If the first signal will be a complaint, you bought a system with no instrumentation.

Model deprecation is scheduled rather than hypothetical. Microsoft gives at least 60 days' notice before retiring a generally available Azure OpenAI model, states that afterwards inference requests return 410 Gone and existing deployments stop working, and describes a roughly 18-month lifecycle from general availability to retirement, with provisioned deployments not auto-upgraded (Microsoft Learn). Anthropic also commits to at least 60 days, after which requests to a retired model fail, and notes that Amazon Bedrock and Google Cloud set their own schedules (Anthropic). So plan for a migration roughly annually and ask who pays. If the vendor will handle it, ask what handling means when the replacement scores differently on your test set — the fourth reason to hold that set yourself.

Cost per query moves in both directions and is rarely watched. Provider prices change, and so does the system: a reranker here, a second pass there, a wider retrieval window, and the unit economics that justified the build no longer hold. Instrument tokens, cost per query and p95 latency from the first week, beside the quality numbers (running unattended).

  • Who re-runs the evaluation set after launch, how often, and what raises an alert
  • What a model migration costs, who pays, and how the replacement is re-accepted against your test set
  • An exit runbook: credentials, data export, the evaluation set, and whether the environment rebuilds from the repository alone

The answers that should end the conversation

Some answers are not merely weak. They tell you the firm has not run one of these systems, or has decided you will not check. Any single one is enough to stop.

Read the list uncharitably, because the failure mode of vendor selection is charity. Each item substitutes a claim about the vendor for a fact about your system: certificates instead of regions, team size instead of names, a guarantee instead of a measurement.

  • "We guarantee 95% accuracy." Before seeing your data that figure came from somebody else's corpus or from nowhere.
  • "We use the latest models." Model choice is the last variable that matters and the first thing a reseller reaches for.
  • "The prompts and configuration remain our IP." In a bespoke build that licenses back to you the only part that is genuinely yours.
  • "We will build the evaluation set after go-live." The set is how anyone knows go-live is safe. Afterwards it is an autopsy.
  • "We are certified", offered as the answer to where the data goes. A certificate names a management system; the question named a region.
  • "We would fix that with a better prompt", about a wrong answer nobody can reproduce. Without a trace that is a guess, and the guesses are billed to you.
  • "We have 200 engineers." Then name the four who would be on this, and what each shipped last.
  • A refusal to let you speak to a reference without the vendor on the call.
  • A fixed price for a scope containing the word "etc", which prices an argument rather than a system.
  • A full estimate returned within an hour of a two-paragraph brief. Somebody has priced a template, and the template is what gets delivered.

The criteria we would currently fail

A buyer's guide that only lists tests its author passes is a sales document with headings. Three checks above are ones ZenMagix does not satisfy today, so here they are with the answer we would give.

Named case studies with numbers. This is the most useful single thing a buyer can ask for and we cannot supply it today: our work page describes the classes of problem we are brought in on rather than named clients, because the named versions are not cleared for publication. That is a true explanation and still a worse answer than a case study. Ask everyone, us included, for a named client, the measured position before and after, and a person you may telephone.

Named authors. These articles are published under the company rather than a byline, on the reasoning that the organisation stands behind the claim and an invented byline would be the easiest thing on the page to fake. The cost is real: you cannot check an individual's track record.

Third-party reviews. We have no verified review profile anywhere. Reviews and independent listings are among the few signals a vendor cannot write for itself, so their absence should count against a vendor rather than read as neutral. Bain's analysis of roughly 500 million AI citations found that 89% of unbranded prompts — those naming no company — are fulfilled by third-party sources rather than the companies' own sites, dispersion by topic running from 76% to 99% (Bain & Company). If an assistant produced your shortlist, most of what it quoted was not written by the firms on it.

That is the test worth applying to any buyer's guide, this one included: find the criteria the author would fail and see whether they are in the document. If a guide's criteria describe its publisher exactly, you are reading a specification written backwards from a company.

Sources

Every regulatory or measured claim above is attributed here. Where we could not find a source we trust, the sentence says so instead of guessing.

Questions people ask about this

How do I check that a vendor has actually built this before?

Ask for an artefact rather than a claim: an anonymised evaluation harness, a redacted acceptance clause, or a production incident with its date, its cause and the change that followed. Firms that have done the work produce one within a day. Firms that have not offer a case study deck, which is the same claim in a larger font.

Should an accuracy target be written into the contract?

Yes, but not at signature. Put the method in at signature — the test set, who holds it, the metric, per-category reporting, who adjudicates, and the consequence of missing — and set the numbers once the first phase has measured a baseline on your data. A figure agreed before anyone has seen the corpus is one both parties will later argue was unreasonable.

Who should own the evaluation set?

You should, and you should physically hold it. It is paid for in your domain experts' time, it is the only instrument that compares a new vendor against the incumbent, and a set living in the vendor's repository gets tuned against until it stops measuring anything. Hand over scores, and keep a second set nobody has run against.

Is a cheaper hourly rate actually cheaper?

Only if the rework rate is comparable, and the rework rate is what an hourly figure hides. The costs that decide an AI project are rarely build hours: they are the discovery nobody scoped, the migration when a model retires, the exception handling assumed away, and the cost per query once the system was made more accurate. Compare total cost over two years including one model migration.

Related

Where this connects

AI Engineering

AI Consulting & Readiness Assessment

A paid diagnostic that tells you what is worth building and what is not.
AI Engineering

AI PoC & MVP Development

A working AI product in weeks, scoped to prove one thing to investors or users.
AI Engineering

Custom Software Development

Line-of-business systems built to your process, owned outright by you.

Read next: How to evaluate a RAG system before you trust it in production · What the DPDP Act actually requires of an AI system

More notes on the blog index, and the practical questions about cost, IP and data residency are on the FAQ.

Run this list at us before you run it at anyone else

Bring the questions above to a scoping call. It is free, it is done by the people who would build the thing, and a fair number of them end with the recommendation not to build it — a cheaper conclusion to reach in week one than in month seven.

Start a conversation See our work