Choosing a retainer agency is a bigger decision than hiring for a one-off project. With a project, a bad fit costs you one engagement. With a retainer, a bad fit costs you months of wasted budget and missed opportunities. The relationship needs to work on a technical, communication, and strategic level.
Here are the 10 questions that separate great retainer agencies from mediocre ones. Ask all of them before signing anything.
1. Are You a Certified Shopify Partner?
This is the minimum bar. Shopify Partner status means the agency has built and maintains active Shopify stores, has access to development resources and beta features, and meets Shopify’s quality standards.
What to look for:
- Active Shopify Partner badge
- Shopify Plus Partner status (if you’re on Plus)
- Portfolio of live Shopify stores
- Experience with your specific store type (DTC, B2B, subscription, multi-market)
Red flag: An agency that works across 15 platforms and treats Shopify as one of many. You want specialists, not generalists.
2. What Are Your Response Time SLAs?
A retainer without defined response times isn’t a retainer. It’s a discount on hourly billing with a monthly commitment. SLAs (Service Level Agreements) should be specific and measurable.
What to look for:
- Emergency response: 2-4 hours
- High priority: Same business day
- Standard requests: 24-48 hours
- Clear escalation path for critical issues
- Weekend/after-hours availability for emergencies
Red flag: “We’ll get to it as soon as we can” is not an SLA. If the agency won’t commit to specific times in writing, they won’t prioritize your work.
Follow-up question: What happens if you miss an SLA? Good agencies have penalty clauses or credit systems. It keeps them accountable.
3. Who Will Actually Work on My Store?
Many agencies sell you on their senior team during the sales process, then hand you off to junior developers after you sign.
What to look for:
- Named team members who will work on your account
- Their specific Shopify experience and portfolio
- Whether the team is in-house or outsourced
- How many other clients they support simultaneously
- Backup coverage when your primary developer is unavailable
Red flag: “Our team will be assigned based on availability.” This means you’ll get whoever is free, which usually means the least busy (least experienced) person.
4. How Do You Handle Communication?
Communication problems kill retainer relationships faster than technical issues. You need to know how and when you’ll interact with your team.
What to look for:
- Dedicated Slack channel or communication platform
- Regular sync meetings (weekly or bi-weekly)
- Clear request submission process (not just random Slack messages)
- Single point of contact (account manager or project manager)
- Documentation of all decisions and changes
Red flag: Email-only communication with no regular cadence. Email threads get buried, context gets lost, and response times balloon.
The ideal setup: A shared Slack channel for quick questions, a task management tool for formal requests, and a weekly 30-minute sync to review progress and priorities.
5. What Does Your Reporting Look Like?
You’re paying a fixed monthly fee. You should know exactly what you’re getting for it.
What to look for:
- Monthly reports delivered proactively (not on request)
- Hours used and how they were allocated
- Performance metrics (page speed, uptime, conversion rate)
- Work completed with links to specific changes
- Recommendations for next month
- Risks or issues identified
Red flag: No regular reporting, or reports that only list hours worked without connecting work to business outcomes. Hours without context are meaningless.
Ask to see a sample report from an existing client (anonymized). The quality of reporting tells you a lot about how organized and accountable the agency is.
6. Can You Show Me Case Studies or Client References?
Past performance is the best predictor of future results. Any agency can make promises. Good agencies can prove they deliver.
What to look for:
- Case studies showing specific retainer outcomes (not just project launches)
- Measurable results: speed improvements, conversion rate gains, revenue impact
- Long-term client relationships (2+ years shows sustained value)
- References you can actually contact
Red flag: Only project-based case studies with no retainer-specific examples. Building a store and maintaining one are different skills. Make sure they can demonstrate both.
Questions for references:
- How responsive is the team when something breaks?
- Do they proactively suggest improvements?
- Have they ever missed an SLA?
- Would you recommend them for a retainer?
7. How Flexible Is Your Retainer Structure?
Your needs will change month to month. A rigid retainer that only covers specific task types wastes your investment.
What to look for:
- Hours usable across different work types (dev, CRO, strategy, bug fixes)
- Ability to shift priorities without renegotiating scope
- Option to add project work outside the retainer when needed
- Reasonable hour rollover policy
- Easy tier upgrades or downgrades
Red flag: “Bug fixes come from your retainer hours, but new features require a separate project agreement.” This means every significant improvement costs extra on top of your monthly fee.
The best retainers let you use your hours however you need: 100% bug fixes one month, 100% new features the next, or any mix in between.
8. What’s Your Tech Stack Expertise?
Shopify development covers a wide range of technologies. Make sure the agency can handle whatever your store needs.
What to look for:
- Liquid theme development (mandatory)
- JavaScript frameworks (React for Hydrogen, Alpine.js for themes)
- Custom app development (Node.js, Ruby, Python)
- Shopify APIs (Admin, Storefront, Checkout Extensions)
- Performance optimization (Core Web Vitals, Lighthouse)
- Integration experience (ERPs, CRMs, payment gateways)
For Plus merchants, also verify:
- Checkout extensibility experience
- Script development
- Flow automation expertise
- Multi-market and B2B setup
Red flag: An agency that only does Liquid theme work but claims to offer full-service retainers. When you need a custom app or integration, they’ll either outsource it (adding cost and quality risk) or tell you it’s out of scope.
9. What Are Your Contract and Exit Terms?
Never sign a retainer with punitive exit terms. If the agency delivers value, you’ll stay. If they don’t, you should be able to leave.
What to look for:
- Month-to-month contracts (or short terms with easy exit)
- 30-day cancellation notice at most
- No early termination fees
- Clear ownership of all code and assets you’ve paid for
- Transition support when ending the engagement
- Complete code handover and documentation
Red flag: 12-month minimum commitments with early termination fees. An agency that locks you in is an agency that doesn’t trust its own value delivery.
Critical clause: Make sure your contract explicitly states that all custom code, themes, and assets created under the retainer belong to you. Some agencies retain IP rights to work they build, which creates leverage if you try to leave.
10. Do They Understand Your Business, Not Just Your Code?
The best retainer agencies operate as partners, not vendors. They understand your business goals, competitive landscape, and growth plans, not just your Liquid templates.
What to look for:
- Discovery questions about your business during the sales process
- Strategic input on how technical decisions impact revenue
- Awareness of your industry and competitive landscape
- Willingness to push back on requests that won’t serve your goals
- Proactive suggestions aligned with your business objectives
Red flag: An agency that only asks about your tech stack and never about your customers, revenue goals, or competitive position. Technical execution without business context produces technically sound but commercially irrelevant work.
Test this in the sales process: Share a business challenge (not a technical request) and see how they respond. Do they jump to a technical solution, or do they ask clarifying questions about the business impact first?
Evaluation Scorecard
Rate each agency on these 10 criteria from 1-5. Any agency scoring below 35 out of 50 has significant gaps.
| Question | Weight | Score (1-5) |
|---|---|---|
| Shopify Partner status | High | __ |
| Response time SLAs | High | __ |
| Team transparency | High | __ |
| Communication structure | Medium | __ |
| Reporting quality | Medium | __ |
| Case studies/references | High | __ |
| Retainer flexibility | Medium | __ |
| Tech stack expertise | High | __ |
| Contract/exit terms | High | __ |
| Business understanding | Medium | __ |
| Total | __/50 |
Our Approach
At Capaxe Labs, we’re transparent about how we stack up against these criteria. We’re a certified Shopify Partner with same-day SLA response times, dedicated named teams, weekly syncs, monthly performance reports, flexible hour allocation, month-to-month contracts, and full code ownership.
We’d rather you evaluate us rigorously and choose us with confidence than sign up without knowing what you’re getting.
Book a consultation to see how we answer all 10 questions for your specific needs. Or view our retainer plans to see pricing and what’s included.
