Scaling a Research Panel from 800 to 48,000 in Three Years

Context

When I joined the AWS Fintech R&D research function in October 2022, the active research panel was approximately 800 internal users. The function supported research across several product teams covering payment processing, billing, and financial reporting workflows. The panel was sustainable at that scale through informal recruitment — researchers reaching out to known points of contact, asking for participants, working through individual relationships. It worked, but it didn't scale, and the function's roadmap required research support across a much larger product surface.

The strategic ask, when I came in, was to build a panel that could support 22+ product teams concurrently. That meant going from approximately 800 to a target north of 30,000 within 24 months, eventually reaching 48,000+ by the end of three years. The constraint set made this non-trivial: PCI-DSS compliance for any participant data touched by financial workflow studies; AWS internal research-compliance review for every recruitment, screener, and study protocol; and the cultural reality that participation in research is voluntary, can't be incentivized in the way external panels are, and competes with users' real product work.

Method

The approach we built had four components, and the sequence mattered.

The recruitment screener architecture came first. Most panels die not because recruitment is too hard, but because the wrong people get into the panel and pollute findings until the panel is functionally useless. We built a tiered screener that captured role, function, primary product surface, tenure, frequency of relevant workflow execution, and willingness-to-participate signals across a granular set of categories. The screener was long enough to be unpleasant on first encounter — about 30 questions — but it produced segmentation that made the panel directly usable for evaluative research without re-screening every study.

Internal compliance approvals were the second component, and they took longer than the technical build. AWS has a research compliance review process that examines methodology, data handling, recipient consent, and storage architecture. For a regulated-industry function, the review is rigorous and the approvers are not easily convinced. We invested in a pre-approved methodology library — interview guides, survey templates, longitudinal protocols — that could be drawn from for new studies without restarting the compliance review from scratch. Building the library took about 90 days of legal and methodology back-and-forth. After that, study setup time dropped from weeks to days.

The governance structure was the third component. A 48,000-person panel left unmanaged becomes a list, not a panel. We established intake rules (which studies could draw from which segments), capacity limits per participant (no participant invited to more than two studies per quarter), opt-out tracking, and a quarterly hygiene review that pruned non-respondents and re-engaged dormant participants. The governance structure required minimal headcount — fundamentally a single ResearchOps role — but high discipline.

The operations playbook was the fourth, and most under-discussed. The playbook documented the entire end-to-end flow: how to file an intake request, how to scope a study against panel availability, how to draft a screener using the pre-approved library, how to launch recruitment, how to handle no-shows and reschedules, how to close out a study with appropriate documentation, and how to feed findings back into the repository. The playbook was the artifact that made the panel scale beyond a single researcher's bandwidth — it allowed associate researchers and embedded researchers across product teams to use the panel without being trained one-on-one.

Insight

The lesson that emerged most clearly across the three years is that panel scale isn't a recruitment problem; it's a governance problem with a recruitment surface. Most teams trying to scale research panels focus on recruitment volume — running more outreach, leveraging more channels, making the screener shorter to increase completion. None of that addresses the durability question. The panels that survive their first year of use are the panels with governance structures that ration participation, prune the panel proactively, and treat the screener as segmentation infrastructure rather than a friction step.

The second lesson, which surprised me, is that the pre-approved methodology library was the single highest-leverage artifact in the system. The technical work — the recruitment screener, the dashboard tooling, the data infrastructure — was important, but it could have been built in many forms. The methodology library was specific. It made every subsequent study fast to launch and consistent in quality, and it became the artifact that researchers across the organization actually wanted to use. The marginal cost of a 23rd or 35th study fell dramatically once the library reached a critical mass of templates.

Impact

The panel reached 48,000+ active participants by early 2026. Time-to-launch for evaluative studies dropped from approximately 3 weeks to 4 business days. The function supported research across 22+ product teams concurrently with a small ResearchOps team. The compliance review process was streamlined enough that AWS internal compliance no longer treated each study as a fresh approval — they reviewed the methodology library annually and approved studies that drew from it on a fast-track basis.

The panel also produced unexpected secondary outputs. Because participants opted in across multiple categories of work, we could run cross-cutting longitudinal studies — diary studies on workflow patterns, repeated quantitative surveys on adoption signals — that wouldn't have been possible in a smaller panel. Several of these became leading indicators for product roadmap decisions across the organization.

Reflection — what I'd do differently, what generalizes

The single thing I'd do differently is invest in the methodology library earlier and more aggressively. We treated it as a parallel workstream when it should have been the lead workstream. The recruitment infrastructure follows from the methodology library, not the other way around — knowing what kinds of studies you'll run shapes what kind of segmentation the panel needs.

What generalizes from this work to other organizations: the four-component architecture (screener, compliance, governance, playbook) translates well to any regulated-industry research function building panel capacity. I've used variants of this architecture in subsequent advisory work in healthcare technology and clinical software. The compliance specifics differ — HIPAA in healthcare, FDA HFE in medical devices, SOX or PCI-DSS in finance — but the structural pattern holds.

What does not generalize: the headcount requirements and the time horizons. A function that's resourced with one ResearchOps role can scale a panel to ~50,000 over three years if the parent organization has the user population to draw from. A function with broader resourcing can move faster. A function constrained to part-time ResearchOps support will stall around 5,000–8,000 active participants and will spend most of its energy on operations rather than insight production. The architecture assumes a baseline operational investment that's worth being honest about up front.

Most regulated-industry research functions are sitting somewhere between 1,000 and 8,000 active panel participants and have stopped growing. The reasons are almost always governance, not recruitment. The first diagnostic question is always: how often does the function refuse a research request because the panel can't support it? If the answer is "rarely," the panel isn't being asked enough hard questions. If the answer is "often," the governance is working — and the next move is methodology library investment, not headcount expansion.

Other case studies →

More on building research operations inside enterprise organizations, mixed-methods coverage at scale, and the patterns that travel between industries.