How to Choose an AI Agent Platform: A Workflow-Based Scorecard

A platform demo shows a clean answer. The purchase decision is the assignment behind it. Most evaluations start with feature grids and product tours, which hides the question that actually determines success: can this platform complete the specific work you give it, end to end, under your constraints?

Start with the assignment, not the demo. Define the finished outcome, then map the inputs, permissions, data ownership, connected tools, channels, human handoff, evidence, and billing unit that the assignment requires. Apply hard gates before any scoring, because a gate failure cannot be offset by points. Only among passing candidates do you run a weighted scorecard, forecast cost at your real volume, and pilot the weakest link.

TL;DR: Buy for the assignment, not the demo. First define the completed job: required output, inputs and permissions, data owner, connected tools, channels, human handoff, evidence, and billing unit. Apply hard gates (compliance, channel, action permission, auditability) before scoring. Then use a weighted scorecard with arithmetic to compare survivors, forecast cost at your real volume, and pilot one high-value intent for two weeks before contract.

Start with the completed assignment, not the demo

Define the assignment as the finished outcome a customer should experience. "Answer shipping questions" is an assignment. "Confirm return eligibility and initiate a refund against order records, then confirm the refund in the billing system" is a different, harder assignment. The second one needs permissions, live data, and action confirmation. The first only needs a trained knowledge base.

Sort your top five request types into three output classes:

• Answer-only: the agent returns accurate information from approved sources. Completion is a correct answer.
• Action: the agent changes state, such as an order, subscription, or account record. Completion requires the change to succeed and be confirmed by the responsible system. An API 200/202 response alone is not confirmation; read the operation state or the authoritative record. On timeout, inspect the operation before retrying, and use idempotency keys to prevent duplicate actions.
• Assisted: the agent prepares context, gathers information, and verifies eligibility, then a person approves or completes a judgment-sensitive step. Completion is defined by whoever owns the final decision.

For each class, write down the definition of done. If you cannot state what done means for your most frequent request types, you cannot evaluate a platform fairly. This is the first step of any buying process, and it does not require vendor information. For a concise background on what agents are and what they are not, see the buying checklist.

Map six input areas before opening the feature grid

Before you look at a single vendor, write down six facts about the assignment:

1. Outputs. What interaction, answer, or state change must happen?
2. Inputs and permissions. Which systems hold the needed data? What scope does the agent need, such as read-only order lookup versus refund execution? OWASP describes excessive agency as a risk arising from excessive functionality, permissions, or autonomy; these are separate things to evaluate.
3. Data owner. Who controls the source documents, customer data, and conversation logs? Where may they be stored? May they be used for model training?
4. Tools and recovery. Which third-party systems must the agent call? What happens when an API fails, a call times out, or a record is missing? Does the platform confirm the result of an action instead of assuming it worked?
5. Channels. Where do your customers actually reach you: web chat, email, WhatsApp, Instagram, phone, or a helpdesk inbox?
6. Human queue. Who receives escalations? Which inbox or team owns them? How does the handoff carry context, and who sees that it happened?

The same platform can be right for one assignment and wrong for another. A helpdesk suite with deep ticketing is a strong fit when your team already lives in that inbox. A standalone agent with native store actions may fit an ecommerce catalog better. The honest comparison starts from your matrix, not from the vendor list.

Hard gates cannot be offset by score

A hard gate is a binary requirement. If a candidate fails a gate, it leaves the process regardless of how many points it earns elsewhere. Score only the candidates that pass every gate you define.

Typical gates include:

• Compliance and data residency. Separate your legal obligations, contractual requirements, and assurance evidence. HIPAA applicability, GDPR responsibilities, and a SOC 2 report answer different questions. Confirm the contracted service, data flows, report scope, and controls with the relevant privacy and security reviewers. A website badge does not establish that coverage.
• Channel coverage. If your customers use a channel the platform does not support, the deployment fails before it starts. Verify the exact channels you need, including social, email, and voice where relevant.
• Identity verification for account actions. If the assignment touches customer records or performs account actions, the platform must support an authenticated customer session or a server-validated identity flow, then enforce object-level authorization on every request. A typed email, order number, or account ID is a lookup key, not identity proof. The chatbot must never collect passwords, admin tokens, or one-time codes; a guest lookup can send a secure link to the stored contact address without exposing records in chat.
• Required permissions and actions. If the assignment requires executing a refund or updating a record, the platform must have an authenticated integration with that system and confirmed result handling. A chat that only explains the policy does not complete an action request.
• Auditability and export. You must be able to view conversation logs, action confirmations, and failures. If you cannot export the data, you cannot audit quality or switch vendors later.
• Spend control. For usage-based meters, confirm that alerts, caps, and overage rates exist and that you can set a hard limit. If the vendor cannot state the unit or the rate, treat that as a gate failure for budgeting purposes.

These are candidate gates. Adjust them to your regulatory, security, and operational reality. What matters is the mechanism: a failed gate ends the evaluation. No weighted score can rescue it.

Build the weighted scorecard

Among candidates that pass every gate, use a weighted scorecard. Assign weights to the criteria that matter for your assignment. Score each candidate from 0 to 4 on each criterion, where 4 means the platform fully meets the requirement and 0 means it does not.

The formula is simple arithmetic:

weighted_score = Σ (weight_i × score_i) / Σ weight_i

A sample weight set for a customer support assignment:

For illustration, here is how two hypothetical candidates might score. These numbers are examples, not vendor results.

Candidate A leads on completion and auditability; Candidate B scores higher on cost control and exit. The current weights favor A overall. If your business cannot tolerate unpredictable spend or values flexibility, reweight those criteria and rerun the arithmetic. The scorecard is a decision aid, not a verdict. It tells you where to focus your pilot and which vendor to prefer when the weights reflect your priorities.

A related method and template live in the scorecard guide, and a separate ROI framework is available in the AI agent ROI calculator.

Worked example: subscription account support

Apply the method to a realistic scenario. The following example is hypothetical, not a customer result or industry benchmark. A SaaS company with about 2,000 active accounts and one support person receives roughly 1,200 support contacts per month. The team wants an agent to handle subscription account work.

Define done for the top intents:

• Password reset and login help: answer-only. Done when the customer receives accurate steps from approved documentation, or a secure reset link is sent to the stored contact address.
• Subscription plan change: action. Done when the customer has authenticated through a secure session, the plan change succeeds, and the billing record confirms it.
• Billing dispute question: assisted. The agent gathers order and payment context, verifies eligibility, then hands off to a person for judgment.
• Cancellation with retention: assisted. The agent asks the cancellation reason and completes the cancellation only when policy allows; otherwise it routes to a person. An AI that only explains the cancellation policy does not complete the action request.

Now apply the gates. The agent must connect to the billing system with permission to change plans, must authenticate the customer before touching account records, must log actions, must operate on the channels customers actually use, and must have a spend cap. Any candidate that cannot change a plan with confirmation and authenticated identity is out, regardless of how good its answers are.

For the scorecard, use the weights above. The example shows how a candidate with strong answers can lose to one with stronger integration, handoff, and cost control. Run your own scores with your own evidence.

Next, forecast the meter. Use your volume and scope assumptions, and label them clearly. Suppose 75% of the 1,200 contacts are in scope, so 900 are eligible. If the agent completes 60% of eligible requests, that is 540 resolutions per month. At an assumed $0.90 per fully automated resolution for illustration, the variable component is about $486 per month before the base plan. This rate is hypothetical; get a configuration quote from each vendor in its own billing unit. Do not compare one vendor's resolution price to another vendor's message credits; they are not the same unit.

Read the meter before you sign

Every platform bills differently, and the unit changes what the contract costs. Documented examples include:

• Seats. Zendesk plans are listed per agent per month, with Support Team at $19, Suite Team at $55, and Suite Professional at $115 when billed annually. Freshdesk plans are likewise per agent per month: Growth at $19, Pro at $55, and Enterprise at $89 billed annually. Seat-based pricing scales with team size, not automation volume.
• Message credits. Chatbase lists Hobby at $32 per month billed annually with 700 message credits, Basic at $120 with 4,000, and Business at $400 with 15,000, with auto-recharge at $40 per 1,000 credits. One credit is not one guaranteed resolved conversation.
• Model-weighted credits. YourGPT sells credit pools of 10 million, 30 million, and 100 million on its Essential, Professional, and Advanced annual plans, and consumption depends on the model. Its documentation lists GPT-4o-mini at 0.5x credits and GPT-4o at 5x, so the same question costs differently by model.
• Successful outcomes or resolutions. Intercom Fin is priced per outcome, with an outcome counted when a customer confirms resolution, does not ask for more help, or Fin completes a configured procedure including a handoff. Fin can also run on an existing helpdesk with no seat costs and a 50-outcome monthly minimum. Gorgias bills AI Agent per resolved interaction, with AI Agent on every plan; the visible pricing page is interactive, so request a configuration quote for exact per-interaction rates. Under current billing, a ticket fully resolved by AI can incur both a helpdesk ticket fee and an automation fee, while accounts created before May 28, 2025 may remain on legacy billing that charges only one fee.
• Sessions. Freshdesk includes the first 500 Freddy AI sessions per plan and charges $49 per additional 100 sessions.
• Billable conversations. Tidio counts a conversation as billable only when a human agent replies, and its Lyro AI product is sold separately, starting at $32.50 per month with 50 AI conversations on the standalone path.

Before signing, ask the vendor for your allowance, overage rate, and any minimum commitment in your volume. Cap overage before launch. If the vendor cannot state the unit, it is not ready to quote.

Check the exit before entry

Procurement is as much about leaving as joining. Confirm in writing: what you can export (conversations, logs, and action records), how long the contract runs, what notice is required to cancel, and whether your data is used to train models. If the platform owns the only copy of your conversation history and you cannot export it, you have built a dependency, not a deployment. A platform that treats your data as yours is easier to leave, and that discipline keeps vendors honest.

Pilot the weakest link

Run a two-week pilot on one high-value intent. Pick the request type with the highest volume or the highest risk. Measure four outcomes separately:

• Resolved correctly, with the action confirmed by the responsible system.
• Escalated to a human with full context.
• Failed because an action or API returned an error.
• Wrong answer that appeared resolved, found by sampling conversations.

Keep failures in the denominator. If an eligible request is routed to a person or fails a lookup, it is not a success. Review a sample of conversations rather than trusting the dashboard. The point of the pilot is not the headline resolution rate; it is the failure distribution. That distribution tells you whether to invest in knowledge, permissions, or handoff design before full rollout.

When the platform is the wrong category

Some assignments belong to a different category of tool. A phone line is better served by a voice platform than by a chat widget. A research team that needs cited sources from the web should look at research agents, not support desks. A sales list is not a support queue. Compare platforms that can complete the same assignment; a single overall ranking that mixes voice, research, and helpdesk tools is not a useful decision aid. Related category guides include AI voice agents, research AI agents, and ecommerce AI support.

FAQ

Why use a weighted scorecard instead of a feature comparison? A feature grid treats all capabilities as equally important and rewards whoever lists the most boxes. A weighted scorecard ties capability to your specific assignment, your channels, and your cost constraints. It also surfaces where a vendor's strength actually matters to you, and it forces you to state priorities before a sales demo shapes them.

What is a hard gate, and how is it different from a score? A hard gate is a binary requirement that cannot be offset by points. Examples include a required channel, a required integration with refund permission, a data residency requirement, or the ability to export logs. A candidate that fails a gate is excluded. Scoring only ranks the candidates that pass every gate.

Can I evaluate a vendor without a hands-on test? You can narrow the field from public documentation, pricing pages, and integration lists. But you cannot verify action confirmation, handoff quality, or failure recovery without testing. Use documentation to filter and to build the scorecard, then pilot the surviving candidates on real traffic before committing to a long contract.

Can I compare message credits across vendors? No. Credits mean different things. Chatbase message credits are not the same unit as YourGPT model-weighted credits, Zendesk automated resolutions, Intercom outcomes, Freshdesk sessions, or Tidio billable conversations. Compare only after converting each vendor's unit to your expected volume in that vendor's own meter.

How long should the pilot last? Two weeks on one intent is a reasonable start, long enough to see real traffic and failure patterns and short enough to contain cost. Extend it if the intent is seasonal or low volume. Judge success by the failure distribution and the cost per verified resolution, not by the headline resolution percentage alone.

Do small teams need a scorecard? Yes, but a smaller one. A solo operator still faces the same gate risks: missing channel, missing permission, unpredictable bills, and no export path. The scorecard protects small teams from paying for features they will not use while missing the one integration the work actually requires.

Conclusion

The decision principle is simple: buy for the completed assignment, not the polished demo; gate before you score; and pilot the weakest link. The platform that wins a demo is not the platform that finishes the work. Start from the finished job, define its inputs and permissions, apply hard gates, score the survivors with arithmetic, forecast cost in the vendor's own unit, and verify on real traffic. That method separates a procurement decision from a sales pitch and gives you a defensible answer you can explain to anyone on the team.