siva Let’s talk
Case Study / Product Strategy · Enterprise SaaS Program Testing

Program Testing: designing confidence into publishing

Program Test Run lets Journey Orchestrator admins dry-run complex, multi-step programs before they go live. I co-authored the PRD with the PM and designed the full experience, reframing the brief from “build a testing feature” to the real question: how do you make publishing feel safe for the first time? It was developed as part of the broader Journey Orchestrator transformation.

My role
Product Designer, sole designer. Co-authored the PRD with the PM, designed every screen and error state, and shipped an interactive prototype ahead of the engineering POC.
Collaboration
  • Product Management (scope & PRD)
  • Engineering (feasibility & dry-run engine)
  • QA (13 test dimensions)
  • Community (demand validation)
At a glance
  • Gainsight · GE-200868
  • ~4 months end-to-end
  • 50 tickets shipped
  • Released Nov 2025 · behind feature toggle
Programs / Q3 Renewal Outreach Test Mode
Test mode active · Read-only · All users see this view until Exit Test Mode.
Entry trigger · 312 participantsSource
Send emailv A / v B
Evaluate · 2 branchesLogic
Test configuration
Path A outcomePositive
Path B outcomeNegative
Participants20 max
Daily runs left94 / 100
Run Now

Test mode, a fully isolated, read-only view with a persistent blue banner and a configuration pane for path outcomes and run limits.

The problem, from the customer's mouth

Admins could only find errors after publishing

The PRD was grounded in direct customer feedback, not internal assumption. Two pain signals drove the brief.

// Signal 01

Data mismatch

Admins had no confidence participant data was accurate before running. Wrong field values, null tokens, unsubscribed contacts, none surfaced before publish. Programs silently dropped participants or sent broken emails to real customers.

Ref: a customer screen recording showing unexpected participant drops post-publish.
// Signal 02

Program flow complexity

Multi-step programs with conditional branches were impossible to mentally simulate. The only way to validate routing was to publish and watch, then stop the program and clone it to fix mistakes.

Admins were stopping programs and creating multiple clones to resolve mistakes.

The challenge wasn't ‘build a testing feature.’ It was: how do you make publishing feel safe for the first time?

The problem in the customer's own words, publishing was the only way to find out what broke.
The central decision

Original vision vs. what shipped

The PRD proposed a far broader scope than the MVP. Trimming it was a deliberate product decision made against the release window: each cut kept the simulation fast, safe, and easy to interpret, and the rest was documented for a later phase.

Shipped in MVP

  • Test mode, a separate, read-only view
  • Test configuration, path outcomes (positive / negative / random)
  • Participant filtering + manual selection (up to 20)
  • Dry-run engine, full simulation, no live messages
  • Email preview with version indication + participant count
  • Slack, API & Evaluate previews with token resolution
  • Snapshot results, participant counts per node
  • Multi-user safety, program locked in test mode
  • Daily run limit (100/day) with warning
  • Pre-run field validation & manual-only sync

Consciously descoped

  • Individual step-by-step navigation (per-participant Next button)
  • Admin approval gates on Evaluate branches
  • Test-send emails to internal IDs from preview
  • Participant data cards highlighting empty / unsubscribed values
  • Export of test-run activity / simulation log
  • Summary emails to admins on execution status
  • Set custom condition values in the test panel
  • Previous-step value overrides for Evaluate testing
  • Participant approval UI between branches
  • Percentage-based random outcomes

Each descoped item represented design complexity that would have compromised the MVP promise. The approval-gate feature, for example, was explicitly marked TBD as a competitive feature requiring further user research before commitment.

The scoping decision was documented and deliberate, with the reasoning captured in the PRD.
Design process

Designing the test experience

  1. Customer research & PRD authoring

    Reviewed customer screen recordings of participant drops and community threads describing cloning workarounds, then co-authored the PRD: problem framing, solution workflow, user stories, acceptance criteria, and the MVP-vs-future split, all before a single screen.

  2. Flow mapping on Miro

    Mapped the full state space with PM, program states, test-mode transitions, selection logic, dry-run paths, result states. This caught a key ambiguity early: what if another admin opens the program mid-test? Answer: a multi-user lock with a clear read-only state.

  3. Prototyping before the engineering POC

    Built an interactive prototype on Vercel before engineering's backend spike, validating the core flow with PM. This let both tracks run in parallel, engineering spiked the dry-run engine while design iterated on the interaction model.

  4. Figma mocks through build

    Maintained a living spec across every screen and error state, adjusting in real time as engineering surfaced backend constraints like participant-count caps.

  5. Scope negotiation under constraint

    The Next-button simulation, approval gates, and test-send email were deferred mid-build. Each removal required a design response, updating the PRD, communicating the change in rollout notes, and ensuring the remaining feature still answered the core trust question.

  6. QA partnership on every state

    Worked with QA across 13 test dimensions and defined UI states for dry-run failure, empty pool, multi-user lock, field-validation warnings, and exit confirmation, none left to engineering.

Design and engineering ran in parallel because the prototype de-risked the interaction model first.
Key design decisions

Key decisions that shaped the test experience

01

A separate view instead of an overlay

Challenge: an overlay or side-by-side split created ambiguity about whether the admin was ‘in’ the live program or the test.

Resolution: test mode opens in a completely separate view, the editor is gone, editing is disabled, a persistent blue banner anchors context, and the only exit is explicit.

Separate view + banner = zero context confusion
02

Testing with real participants

Challenge: custom condition values and test-sends to internal IDs were appealing, but they shifted the simulation away from reality.

Resolution: the dry run uses actual synced participants and real field values, with token resolution through the same pipeline as live execution. What admins preview is exactly what customers would receive.

Real data = real confidence
03

Snapshot first, step-through later

Challenge: the PRD's most detailed feature was an individual Next-button simulation, the fullest expression of the ‘simulate’ vision.

Resolution: deferred. Simulating all selected participants at once and showing a snapshot answered the highest-priority question, does the branching logic work? faster and with less cognitive load. Step-through is a clear Phase 2 addition for debugging.

Validate before you debug
04

Every error state is a designed state

Challenge: the feature has a dozen-plus failure modes, dry-run failed, field not resolved, empty pool, daily limit, multi-user lock, re-sync needed.

Resolution: each state was explicitly designed with specific copy, icon, and primary action. QA received a designed response for every scenario, zero ambiguous error strings at handoff.

8+ error states designed before QA began
05

Removing auto-sync

Challenge: adding a source auto-synced test participants, confusing admins about where participants came from or whether they were real.

Resolution: auto-sync was removed entirely; participants now enter only through explicit manual sync. ‘Sync All’ became ‘Sync’ to reinforce intent. This was a behaviour change affecting all programs, requiring cross-functional alignment, not just a test-mode tweak.

Manual-only sync = full admin control
The Test Run Configuration screen, opened in a dedicated Test Mode view with its own badge, choosing a Positive or Negative test-user path.
Decision 01 in production, Test Mode opens as its own dedicated view (note the badge), not an overlay, so there's never any doubt about live vs. test.
Screens designed

The shipped test flow

The draft program under test: Audience, Welcome Email, Survey, a 10-day wait, Not Responded / Survey Responded branches, Evaluate, and Education.
The program under test, a branching flow that's impossible to validate by reading alone.
Test Run Configuration step 1: choose a test-user path, Positive (happy path) or Negative (handles bumps), with Positive selected.
Step 1, choose the path: send test users down the happy path, or stress the error branches.
Step 2: the participant picker table with synced participants and companies, and a confirmation that 30 participants were added.
Step 2, pick real participants: the dry run uses actual synced data, not synthetic stand-ins.
The dry run in progress, the flow dimmed with a reassuring banner that participants are gearing up, no live messages sent.
The dry run executes against the real pipeline, with zero live messages or CRM writes.
The snapshot results with participant counts on every node: Audience 30, Welcome Email 30, Survey 30, CTA 6, Evaluate 24, Catch All 4.
The snapshot, counts on every node answer the core question: does the branching logic work?
The Select Participant dropdown listing individuals (John Doe, Liam Carter, Emma Thompson and more) to replay any single participant's path.
Drill into any single participant to replay exactly the path they took.
The Test Run Finished confirmation over the results, inviting the admin to finalize the setup.
Test run finished, confidence to publish, earned before anything goes live.
Outcome & impact

Publishing, finally de-risked

0
Tickets shipped
All categories
0
QA dimensions
incl. X-Org & Redshift
0
Live side effects
Fully isolated dry run
0+
Community requests closed
By this feature
  • Admins can validate programs before any customer is affected, token resolution, branch routing and variant assignment, with no live message or CRM record
  • The vision-vs-MVP decision paid off: a focused dry run + snapshot shipped on time, with step-through sequenced as a clear Phase 2
  • Removing auto-sync improved data quality across all JO programs, resolving a long-standing source of participant confusion
  • 8+ explicitly designed error states meant zero QA ambiguity at handoff
  • The Phase 1 snapshot experience establishes the mental model Phase 2 will build on
Leadership reflection

Where I led, and where the team delivered

The PRD came out of the design process, problem framing, solution workflow, and the MVP-vs-future split were design contributions before a screen was touched. The prototype was running before engineering had a backend POC. The hardest decision was letting go of the feature the PRD spent the most words on, the step-by-step simulator, in favour of a faster, more focused dry run that answered the user's core question better. That conviction, under technical constraint, is the strategy.

Interested in working together?

I co-authored the PRD and designed every state down to the error cases. I am comfortable carrying an ambiguous problem all the way to something shippable and trustworthy.

Get in touch