Most SI vendor shortlists get decided on the wrong evidence: the slickest demo, the most confident pitch, or whoever replied first. Three to five proposals arrive in different formats, each answering a slightly different question, and the choice comes down to gut feel.
A scorecard fixes that. It makes every vendor answer the same brief, makes everyone on your side score the same criteria, and puts the deal-breakers in writing before anyone is charmed. What follows is the one we would use on the buying side. It is vendor-neutral and works for studios, agencies and large consultancies alike.
Check that you need a vendor at all
A scorecard compares vendors. It does not tell you whether the project should exist. If the problem could be solved by configuring an existing SaaS product or a plain workflow tool, a custom SI build adds risk without adding much value. Run the idea through our build-vs-buy framework for when SI is the wrong answer before you write the brief. A good vendor will raise the same question in the first call, and whether they do is worth a note on the scorecard.
How do I choose an SI development company?
Send the same written brief and sample data to three to five vendors, ask each the same questions, and score their answers on a weighted scorecard. Weigh most heavily how they frame the problem, how they test quality before launch, how they handle your data, and whether you will own the code, prompts, evaluation data and model artefacts. Then score team continuity, delivery process, references, post-launch support and exit terms. Disqualify any vendor that refuses an NDA, full ownership or a reference call.
How to run the comparison
The method matters as much as the criteria. If each vendor hears a different version of the problem, you end up scoring their sales skills rather than their engineering.
- Write one brief and send it unchanged to every vendor. Describe the business problem, the users, the systems involved, the data you hold and what a good outcome looks like. Leave the solution open, because how they fill that gap is part of what you are scoring.
- Share the same sample material with all of them, such as a set of real questions or documents with personal details removed, and only after an NDA is signed.
- Ask the same written questions, at least one per criterion below, and ask for written answers. Written answers are easier to compare and harder to bluff.
- Hold one working session with each vendor's lead engineer, not only their sales team, using the same agenda each time.
- Have two or three people on your side score independently, then meet to compare. A large gap between scorers usually points to a question nobody asked clearly, so ask it again.
- Call references only for the top two, once the scores are in, so that one warm reference call does not colour the whole comparison.
If nobody on your side has bought SI work before, it can help to have the brief and the weights reviewed by someone with SI consulting experience who is not bidding for the build.
The scorecard
Score each criterion from 0 to 5, where 0 means no answer or a worrying one and 5 means a specific, evidenced answer that fits your situation. Multiply each score by its weight, then add them up. The weights below are a starting point: weight 3 where a weak answer can sink the project, weight 2 where the gap matters but can be closed in the contract or during the work. Adjust them to your own risks before the proposals arrive, not after you have a favourite.
Weight 3: the criteria that sink projects
- Problem framing. Did they restate your problem in their own words, question parts of the brief, and say what they would not build? A 5 names the riskiest assumption in your project and how they would test it early. A 1 jumps straight to a model name and an architecture diagram.
- Evaluation practice. How will they know the system is good enough to launch? A 5 describes an evaluation set built from your real questions with your domain experts, scored automatically on every change, with written thresholds that block a release, and can show an anonymised report from past work. A 1 offers another demo. Our piece on eval-first SI describes what good looks like here.
- Data and security handling. Ask where your data is stored and processed, who can see it, whether it is used for anything else and how it is deleted. Ask how they test for the risks in the OWASP Top 10 for LLM Applications, such as prompt injection, sensitive information disclosure and excessive agency; a team that does this will have something like a prompt injection testing checklist. If the system will handle personal data of people in the EU, the GDPR requires a written contract with any processor, so ask for their data processing agreement. A 5 hands over written policies. A 1 offers reassurance.
- Ownership of code, prompts and models. The deliverable is more than source code: it includes prompts, evaluation datasets, retrieval indexes and the pipelines that build them, infrastructure code, and any fine-tuned weights or adapters. Under US copyright law a transfer of ownership needs a signed written instrument, so check that the assignment names all of these, and that model provider, cloud and repository accounts sit in your organisation's name from the start. A 5 offers this in their standard terms. A 1 wants everything hosted on their own accounts.
Weight 2: the criteria that decide the working relationship
- Team seniority and continuity. Who exactly will do the work, how experienced are they, and will the people in the pitch still be on the project at launch? Ask what happens when a key engineer leaves. A 5 names the team and agrees to name key people in the contract. A 1 says the team will be assigned after signing.
- Delivery process and communication. How is work planned, shown and accepted? Where are decisions recorded, and who joins your calls? A 5 shows you a real status update and release note with client details removed. A 1 says they are agile.
- References. Can they put you on a call with a client from a similar project, ideally one whose system is still running? A 5 offers references without being chased. A 1 points only to logos and case studies.
- Post-launch support. SI systems change after launch even when nobody touches the code: providers retire models (OpenAI, for example, publishes a deprecations page with shutdown dates), your documents change, and user questions drift from the ones you tested. Ask who monitors quality and cost, how a model change is tested and what support covers. A 5 re-runs the evaluation suite before any model switch, as a matter of routine.
- Exit terms. What do you receive if the contract ends, in what form, and how is the vendor's access removed? Ask for a handover plan that includes at least one release your own team deploys alone. A 5 has written this down before you asked.
With these weights the maximum is 110 points: four criteria at weight 3 and five at weight 2, each scored out of 5. Treat the total as a guide, not a verdict. Two vendors a few points apart are effectively tied, and a tie is settled by the reference calls and the working session, not by the arithmetic. Look at the shape as well as the sum: a vendor that scores high overall but low on evaluation or ownership is a bigger risk than the total suggests.
Red flags that disqualify outright
Some answers should end the conversation whatever the total. Write these down before the proposals arrive, so nobody argues them away later.
- They will not sign an NDA before seeing your data, or ask for production data before any discussion of security.
- They refuse to assign ownership of prompts, evaluation data or fine-tuned artefacts, or insist that core accounts stay in their name.
- They cannot describe how they test quality beyond a demo, or quote accuracy figures before they have seen your data.
- They cannot produce a single client willing to speak to you.
- The people in the pitch will not be the people doing the work, and they will not say who will.
- They claim a certification such as SOC 2 or ISO 27001 but will not share the report or certificate, or its scope does not cover the team doing your work.
- There is no exit clause, or leaving would mean losing access to your own system.
Score the evidence, not the demo.
What this scorecard leaves out
It is deliberately general. If you are buying a retrieval system, add the RAG-specific questions in what to ask a RAG development company before you sign, such as how retrieval and answer quality are scored separately and how document permissions are enforced. If you are comparing teams in Pakistan, the checks in outsourcing SI development to Pakistan cover time-zone overlap, cross-border data transfers and checking the legal entity. Both fit inside the criteria above without changing the method.
It also says nothing about the commercial model. Whether you buy a fixed scope or engineering time changes how much of the uncertainty each side carries. Settle it before you sign, not after the first change request.
Where Octalcode fits
Octalcode is one of the vendors you might put through this scorecard: a studio in Lahore, founded in 2022, with a US presence in Austin, that builds LLM applications, SI agents and RAG systems. We would expect to be scored like anyone else, red flags included.
If you want help writing the brief or setting the weights before you approach anyone, that is part of our SI consulting work. You keep the scorecard either way.